Crawl Open Source Projects

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.

novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、充值订阅、新闻发布等功能。

Open Source Web Crawler for Java

This repo is archived. Thanks for wooyun! 乌云公开漏洞、知识库爬虫和搜索 crawl and search for wooyun.org public bug(vulnerability) and drops

英语字典 英语词库 字典词库 四级单词 六级单词 考研单词 雅思 托福 SAT GMAT TOEFL GRE

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Gospider - Fast web spider written in Go

Dungeon Crawl: Stone Soup official repository

https://spatie.be/docs/crawler

Create and generate sitemaps with ease

news-please - an integrated web crawler and information extractor for news that just works

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

A web-based roguelike written in Dart.

Polite, slim and concurrent web crawler.

Gryffin is a large scale web security scanning platform.

Flexible Node.js AI-assisted crawler library

Analyze the security of any domain by finding all the information possible. Made in python.

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

XSS spider - 66/66 wavsep XSS detected

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

A zero-configuration static pre-renderer for React apps

Anemone web-spider framework

Minimal, unopinionated static site generator powered by webpack

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.

浏览过的精彩逆向文章汇总，值得一看

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星途WeGoing、猜画小歌、知乎答题王、腾讯中国象棋、跳一跳、题多多黄金版）

A configurable and extensible PHP web spider

A scalable frontier for web crawlers

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

360/0Kee-Team/crawlergo动态爬虫结合长亭XRAY扫描器的被动扫描功能

Linux file search, instant results as you type

Run a high-fidelity browser-based web archiving crawler in a single Docker container

JS破解逆向，破解JS反爬虫加密参数，已破解极验滑块w（2022.2.19），QQ音乐sign（2022.2.13），拼多多anti_content，boss直聘zp_token，知乎x-zse-96，酷狗kg_mid/dfid，唯品会mars_cid，中国裁判文书网（2020-06-30更新），淘宝密码，天安保险登录，b站登录，房天下登录，WPS登录，微博登录，有道翻译，网易登录，微信公众号登录，空中网登录，今目标登录，学生信息管理系统登录，共赢金融登录，重庆科技资源共享平台登录，网易云音乐下载，一键解析视频链接，财联社登录。

A high performance web crawler / scraper in Elixir.

ChinesePersonRelationGraph, person relationship extraction based on nlp methods.中文人物关系知识图谱项目,内容包括中文人物关系图谱构建,基于知识库的数据回标,基于远程监督与bootstrapping方法的人物关系抽取,基于知识图谱的知识问答等应用。

Use NLP to predict stock price movement associated with news

Crawl BookCorpus

Make a ZIM file from any Web site and surf offline!

crawl and scrape web pages in rust

This project crawls bug bounty platform scopes (like Hackerone/Bugcrowd/Intigriti/etc) hourly and dumps them into the bounty-targets-data repo

Static website generator that lets you use HTTP servers and frameworks you already know

Exploring word2vec embeddings as a graph of nearest neighbors

A Facebook crawler

A Burp Suite Extension that try to find all sub-domain, similar-domain and related-domain of an organization automatically! 基于流量自动收集整个企业或组织的子域名、相似域名、相关域名的burp插件

一个自动获取游戏远程ip，并自动写成SSTAP/NETCH规则文件的脚本

The A11y Machine is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports.

Crawler Microservice for the YaCy Grid

HTML to Markdown converter and crawler.

Advanced python library to scrap Twitter (tweets, users) from unofficial API

Blockchain Transactions Investigation Tool

CLI tool to crawl a website and check HTTP status codes

（更新）数据接口，淘宝(带精确预售量、精确月销量)，拼多多，小红书，微信公众号，大众点评，快手，京东，饿了么，B站，知乎，微博，Bigo，TEMU，得物、贝壳，shopee，百度指数，等数据接口；大模型训练预料

Scrapy Book Code

Common Crawl support library to access 2008-2012 crawl archives (ARC files)

Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses.