Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Crawler Open Source Projects

Open source projects categorized as Crawler
Categories > Data Processing > Crawler
Scrapy ⭐ 47,417
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87most recent commit 5 days agopypi Scrapy} Downloads
Lux ⭐ 21,172
👾 Fast and simple video download library and CLI tool written in Go
dependent packages 5total releases 31most recent commit 4 days ago
Colly ⭐ 19,582
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22most recent commit 23 days ago
Proxy_pool ⭐ 18,022
Python爬虫代理IP池(proxy pool)
most recent commit 2 days ago
Pyspider ⭐ 15,819
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit 2 months agopypi pyspider} Downloads
Newspaper ⭐ 12,678
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18most recent commit a month agopypi newspaper3k} Downloads
Tushare ⭐ 12,165
TuShare is a utility for crawling historical data of China stocks
most recent commit 15 days ago
Examples Of Web Crawlers ⭐ 11,050
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
most recent commit a year ago
Webmagic ⭐ 10,770
A scalable web crawler framework for Java.
dependent packages 15total releases 21most recent commit 2 months ago
Easyspider ⭐ 10,062
A visual no-code/code-free web crawler/spider易采集：一个可视化爬虫软件，可以无代码图形化的设计和执行爬虫任务
most recent commit 2 days ago
Crawlab ⭐ 9,868
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 15most recent commit a day ago
Photon ⭐ 9,272
Incredibly fast crawler designed for OSINT.
total releases 18most recent commit 5 months ago
Avbook ⭐ 8,777
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 3 months ago
Crawlee ⭐ 8,419
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246most recent commit 21 hours agonpm @crawlee/types} Downloads
Python ⭐ 8,296
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
most recent commit a month ago
Spider Flow ⭐ 7,775
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 3 months ago
Infospider ⭐ 6,649
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透
most recent commit 15 days ago
Katana ⭐ 6,588
A next-generation crawling and spidering framework.
most recent commit a day ago
Node Crawler ⭐ 6,446
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30most recent commit 3 months agonpm crawler} Downloads
Awesome Web Scraping ⭐ 5,777
List of libraries, tools and APIs for web scraping and data processing.
most recent commit a month ago
Awesome Crawler ⭐ 5,548
A collection of awesome web crawler,spider in different languages
most recent commit a month ago
Wechatsogou ⭐ 5,495
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit 2 months ago
Ferret ⭐ 5,386
Declarative web scraping
dependent packages 3total releases 34most recent commit 2 days ago
The 10 Most Depended On Crawler Open Source Projects
Scrapy ⭐ 47,417
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87latest release March 01, 2022most recent commit 5 days agopypi Scrapy} Downloads
Colly ⭐ 19,582
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22latest release March 08, 2022most recent commit 23 days ago
Node Crawler ⭐ 6,446
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30latest release November 23, 2020most recent commit 3 months agonpm crawler} Downloads
Crawler Detect ⭐ 1,780
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
dependent packages 103total releases 151latest release March 15, 2022most recent commit 4 days agopackagist jaybizzle/crawler-detect} Downloads
Querylist ⭐ 2,568
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 71total releases 21latest release July 05, 2021most recent commit 10 months agopackagist jaeger/querylist} Downloads
Rod ⭐ 3,858
A Devtools driver for web automation and scraping
dependent packages 58total releases 385latest release September 23, 2022most recent commit 2 days ago
Newspaper ⭐ 12,678
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18latest release September 28, 2018most recent commit a month agopypi newspaper3k} Downloads
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 32total releases 23latest release May 30, 2012most recent commit 3 years agorubygems anemone} Downloads
Crawler ⭐ 2,349
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
dependent packages 27total releases 96latest release May 30, 2022most recent commit 5 days agopackagist spatie/crawler} Downloads
Laravel Sitemap ⭐ 1,950
Create and generate sitemaps with ease
dependent packages 27total releases 57latest release September 22, 2022most recent commit 11 days agopackagist spatie/laravel-sitemap} Downloads
The 10 Latest Releases In Crawler Open Source Projects
Crawlee ⭐ 8,419
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246latest release September 23, 2022most recent commit 21 hours agonpm @crawlee/types} Downloads
Rod ⭐ 3,858
A Devtools driver for web automation and scraping
dependent packages 58total releases 385latest release September 23, 2022most recent commit 2 days ago
Laravel Sitemap ⭐ 1,950
Create and generate sitemaps with ease
dependent packages 27total releases 57latest release September 22, 2022most recent commit 11 days agopackagist spatie/laravel-sitemap} Downloads
Google Play Scraper ⭐ 1,962
Node.js scraper to get data from Google Play
dependent packages 22total releases 84latest release August 23, 2022most recent commit 22 days agonpm google-play-scraper} Downloads
Ferret ⭐ 5,386
Declarative web scraping
dependent packages 3total releases 34latest release July 30, 2022most recent commit 2 days ago
Heritrix3 ⭐ 2,469
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 2total releases 9latest release July 27, 2022most recent commit a month ago
Gau ⭐ 2,664
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
dependent packages 1total releases 29latest release July 23, 2022most recent commit 3 months ago
Crawlergo ⭐ 2,468
A powerful browser crawler for web vulnerability scanners
total releases 1latest release July 07, 2022most recent commit 5 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Classification
Data Science
Scraper
Transaction
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.