Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Crawler Open Source Projects

Open source projects categorized as Crawler
Categories > Data Processing > Crawler
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit a year agopypi Scrapy} Downloads
Easyspider ⭐ 36,416
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化
most recent commit 7 months ago
Lux ⭐ 29,828
👾 Fast and simple video download library and CLI tool written in Go
dependent packages 8total releases 40most recent commit 7 days ago
Colly ⭐ 23,382
Elegant Scraper and Crawler Framework for Golang
dependent packages 328total releases 22most recent commit a year ago
Proxy_pool ⭐ 19,442
Python ProxyPool for web spider
most recent commit 2 years ago
Pyspider ⭐ 15,943
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit 2 years agopypi pyspider} Downloads
Examples Of Web Crawlers ⭐ 13,142
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
most recent commit 2 years ago
Crawlee ⭐ 12,871
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit a year agonpm crawlee} Downloads
Tushare ⭐ 12,165
TuShare is a utility for crawling historical data of China stocks
most recent commit 2 years ago
Webmagic ⭐ 11,080
A scalable web crawler framework for Java.
dependent packages 22total releases 25most recent commit a year ago
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit a year ago
Photon ⭐ 10,244
Incredibly fast crawler designed for OSINT.
total releases 18most recent commit 2 years ago
Python ⭐ 9,097
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
most recent commit 2 years ago
Avbook ⭐ 8,777
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 2 years ago
Spider Flow ⭐ 8,075
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 2 years ago
Katana ⭐ 7,995
A next-generation crawling and spidering framework.
dependent packages 1total releases 8most recent commit a year ago
Wechatsogou ⭐ 5,967
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit 2 years ago
Scrapy Redis ⭐ 5,596
Redis-based components for Scrapy.
dependent packages 21total releases 18most recent commit a year agopypi scrapy-redis} Downloads
Ferret ⭐ 5,540
Declarative web scraping
dependent packages 5total releases 56most recent commit 2 years ago
Haipproxy ⭐ 5,384
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit 3 years ago
Autoscraper ⭐ 5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit 2 years agopypi autoscraper} Downloads
Proxypool ⭐ 5,154
An Efficient ProxyPool with Getter, Tester and Server
most recent commit 2 years ago
Douyin_tiktok_download_api ⭐ 4,844
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T
total releases 21most recent commit 2 years ago
The 10 Most Depended On Crawler Open Source Projects
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit a year agopypi Scrapy} Downloads
Node Rate Limiter ⭐ 1,444
A generic rate limiter for node.js. Useful for API clients, web crawling, or other tasks that need to be throttled
dependent packages 329total releases 16latest release May 19, 2021most recent commit 2 years agonpm limiter} Downloads
Colly ⭐ 23,382
Elegant Scraper and Crawler Framework for Golang
dependent packages 328total releases 22latest release March 08, 2022most recent commit a year ago
Linkinator ⭐ 955
🐿 Scurry around your site and find all those broken links.
dependent packages 313total releases 106latest release November 22, 2023most recent commit a year agonpm linkinator} Downloads
Static Site Generator Webpack Plugin ⭐ 1,538
Minimal, unopinionated static site generator powered by webpack
dependent packages 214total releases 21latest release November 19, 2018most recent commit 7 years agonpm static-site-generator-webpack-plugin} Downloads
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit a year ago
Puppeteer Sharp ⭐ 3,626
Headless Chrome .NET API
dependent packages 120total releases 78latest release December 05, 2023most recent commit 3 months agonuget PuppeteerSharp} Downloads
Crawler Detect ⭐ 1,842
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
dependent packages 110total releases 156latest release July 21, 2023most recent commit 2 years agopackagist jaybizzle/crawler-detect} Downloads
Zhihu Api ⭐ 267
Unofficial API for zhihu.
dependent packages 103total releases 39latest release July 16, 2017most recent commit 8 years agonpm zhihu-api} Downloads
Querylist ⭐ 2,598
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 73total releases 21latest release July 05, 2021most recent commit 3 years agopackagist jaeger/querylist} Downloads
The 10 Latest Releases In Crawler Open Source Projects
Taobao_bra_crawler ⭐ 189
a taobao web crawler just for fun.
latest release December 26, 2023most recent commit 7 years ago
Spider ⭐ 426
The fastest web crawler written in Rust. Maintained by @a11ywatch.
dependent packages 5total releases 368latest release December 11, 2023most recent commit a year agocargo spider} Downloads
Comiccrawler ⭐ 251
An image crawler written in Python.
total releases 175latest release December 10, 2023most recent commit a year ago
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 9 years ago
Crawley ⭐ 208
The unix-way web crawler
total releases 60latest release December 07, 2023most recent commit 2 years ago
Google News Scraper ⭐ 144
Lightweight scraper for Google News
dependent packages 2total releases 12latest release December 06, 2023most recent commit 2 years agonpm google-news-scraper} Downloads
Puppeteer Sharp ⭐ 3,626
Headless Chrome .NET API
dependent packages 120total releases 78latest release December 05, 2023most recent commit 3 months agonuget PuppeteerSharp} Downloads
Xalpha ⭐ 1,851
基金投资管理回测引擎
dependent packages 1total releases 53latest release December 04, 2023most recent commit 2 years agopypi xalpha} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Kafka
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2025 Awesome Open Source.  All rights reserved.