Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Crawler Open Source Projects

Open source projects categorized as Crawler
Categories > Data Processing > Crawler
Scrapy ⭐ 47,307
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87most recent commit 18 hours agopypi Scrapy} Downloads
Lux ⭐ 21,085
👾 Fast and simple video download library and CLI tool written in Go
dependent packages 5total releases 31most recent commit 19 days ago
Colly ⭐ 19,582
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22most recent commit 10 days ago
Proxy_pool ⭐ 17,880
Python爬虫代理IP池(proxy pool)
most recent commit 5 days ago
Pyspider ⭐ 15,819
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit a month agopypi pyspider} Downloads
Newspaper ⭐ 12,678
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18most recent commit a month agopypi newspaper3k} Downloads
Tushare ⭐ 12,165
TuShare is a utility for crawling historical data of China stocks
most recent commit 3 days ago
Xsstrike ⭐ 11,555
Most advanced XSS scanner.
most recent commit 7 days ago
Examples Of Web Crawlers ⭐ 11,050
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
most recent commit a year ago
Webmagic ⭐ 10,770
A scalable web crawler framework for Java.
dependent packages 15total releases 21most recent commit 2 months ago
Crawlab ⭐ 9,819
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 15most recent commit 8 days ago
Photon ⭐ 9,272
Incredibly fast crawler designed for OSINT.
total releases 18most recent commit 5 months ago
Avbook ⭐ 8,777
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 3 months ago
Crawlee ⭐ 8,349
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246most recent commit a day agonpm @crawlee/types} Downloads
Python ⭐ 8,296
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
most recent commit 22 days ago
Spider Flow ⭐ 7,775
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 2 months ago
Easyspider ⭐ 7,281
A visual no-code/code-free web crawler/spider一个可视化爬虫软件，可以无代码图形化设计和执行的爬虫任务
most recent commit 12 hours ago
Infospider ⭐ 6,649
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透
most recent commit 3 days ago
Katana ⭐ 6,485
A next-generation crawling and spidering framework.
most recent commit 5 days ago
Node Crawler ⭐ 6,437
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30most recent commit 3 months agonpm crawler} Downloads
Awesome Web Scraping ⭐ 5,777
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 23 days ago
Awesome Crawler ⭐ 5,548
A collection of awesome web crawler,spider in different languages
most recent commit 23 days ago
Wechatsogou ⭐ 5,495
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit a month ago
The 10 Most Depended On Crawler Open Source Projects
Scrapy ⭐ 47,307
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87latest release March 01, 2022most recent commit 18 hours agopypi Scrapy} Downloads
Colly ⭐ 19,582
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22latest release March 08, 2022most recent commit 10 days ago
Node Crawler ⭐ 6,437
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30latest release November 23, 2020most recent commit 3 months agonpm crawler} Downloads
Crawler Detect ⭐ 1,751
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
dependent packages 103total releases 151latest release March 15, 2022most recent commit 2 months agopackagist jaybizzle/crawler-detect} Downloads
Querylist ⭐ 2,568
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 71total releases 21latest release July 05, 2021most recent commit 10 months agopackagist jaeger/querylist} Downloads
Rod ⭐ 3,810
A Devtools driver for web automation and scraping
dependent packages 58total releases 385latest release September 23, 2022most recent commit 20 hours ago
Newspaper ⭐ 12,678
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18latest release September 28, 2018most recent commit a month agopypi newspaper3k} Downloads
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 32total releases 23latest release May 30, 2012most recent commit 3 years agorubygems anemone} Downloads
Crawler ⭐ 2,278
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
dependent packages 27total releases 96latest release May 30, 2022most recent commit 4 months agopackagist spatie/crawler} Downloads
Laravel Sitemap ⭐ 1,929
Create and generate sitemaps with ease
dependent packages 27total releases 57latest release September 22, 2022most recent commit a month agopackagist spatie/laravel-sitemap} Downloads
The 10 Latest Releases In Crawler Open Source Projects
Crawlee ⭐ 8,349
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246latest release September 23, 2022most recent commit a day agonpm @crawlee/types} Downloads
Rod ⭐ 3,810
A Devtools driver for web automation and scraping
dependent packages 58total releases 385latest release September 23, 2022most recent commit 20 hours ago
Laravel Sitemap ⭐ 1,929
Create and generate sitemaps with ease
dependent packages 27total releases 57latest release September 22, 2022most recent commit a month agopackagist spatie/laravel-sitemap} Downloads
Google Play Scraper ⭐ 1,962
Node.js scraper to get data from Google Play
dependent packages 22total releases 84latest release August 23, 2022most recent commit 9 days agonpm google-play-scraper} Downloads
Ferret ⭐ 5,358
Declarative web scraping
dependent packages 3total releases 34latest release July 30, 2022most recent commit a month ago
Heritrix3 ⭐ 2,469
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 2total releases 9latest release July 27, 2022most recent commit 17 days ago
Gau ⭐ 2,664
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
dependent packages 1total releases 29latest release July 23, 2022most recent commit 3 months ago
Crawlergo ⭐ 2,448
A powerful browser crawler for web vulnerability scanners
total releases 1latest release July 07, 2022most recent commit 5 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Classification
Scraper
Data Science
Transaction
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.