Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Spider Scrapy Open Source Projects

Open source projects categorized as Spider Scrapy
Learn_python3_spider ⭐ 14,425
python爬虫教程系列、从0到1学习python爬虫，包括浏览器抓包，手机APP抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifu 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等
total releases 2most recent commit 4 months ago
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 4 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 5 months ago
Haipproxy ⭐ 5,384
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit a year ago
Ecommercecrawlers ⭐ 3,724
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼
most recent commit a year ago
Weibospider ⭐ 3,294
持续维护的新浪微博采集工具🚀🚀🚀
most recent commit 4 months ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 7 years ago
Gerapy ⭐ 3,144
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 49most recent commit 3 months ago
Scrapydweb ⭐ 2,839
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
total releases 18most recent commit 6 months ago
Scrapyd ⭐ 2,766
A service daemon to run Scrapy spiders
dependent packages 15total releases 11most recent commit 3 months agopypi scrapyd} Downloads
Spiderkeeper ⭐ 2,685
admin ui for scrapy/open source scrapinghub
total releases 6most recent commit a year ago
Python3 Spider ⭐ 2,582
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
most recent commit 6 months ago
Scrapy Examples ⭐ 2,550
Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
most recent commit 6 years ago
Feapder ⭐ 2,333
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、
dependent packages 3total releases 173most recent commit 4 months agopypi feapder} Downloads
Image Downloader ⭐ 2,029
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
most recent commit 3 months ago
Quotesbot ⭐ 1,178
This is a sample Scrapy project for educational purposes
most recent commit a year ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit 6 months agopypi scutils} Downloads
Reptile ⭐ 1,081
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
most recent commit 3 years ago
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
total releases 65most recent commit 2 years ago
Jspider ⭐ 1,006
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
most recent commit 2 years ago
Querido Diario ⭐ 944
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
most recent commit 3 months ago
Kimuraframework ⭐ 874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
dependent packages 2total releases 10most recent commit 2 years agorubygems kimurai} Downloads
Funpyspidersearchengine ⭐ 862
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
most recent commit 2 years ago
The 10 Most Depended On Spider Scrapy Open Source Projects
Scrapyd ⭐ 2,766
A service daemon to run Scrapy spiders
dependent packages 15total releases 11latest release September 25, 2023most recent commit 3 months agopypi scrapyd} Downloads
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 4 months agopypi icrawler} Downloads
Scrapy Wayback Machine ⭐ 70
A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
dependent packages 4total releases 3latest release February 15, 2021most recent commit 3 years agopypi scrapy-wayback-machine} Downloads
Flake8 Scrapy ⭐ 18
A Flake8 plugin to catch common issues on Scrapy spiders
dependent packages 4total releases 1latest release October 10, 2018most recent commit 2 years agopypi flake8-scrapy} Downloads
Dature ⭐ 39
新浪博客批量备份工具
dependent packages 2total releases 29latest release March 31, 2023most recent commit a year agonpm dature} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
Antch ⭐ 177
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
dependent packages 1total releases 1latest release December 25, 2017most recent commit 4 years ago
Scrapy Puppeteer ⭐ 45
Library that helps use puppeteer in scrapy.
dependent packages 1total releases 10latest release September 11, 2023most recent commit 3 months agopypi scrapy-puppeteer-client} Downloads
The 10 Latest Releases In Spider Scrapy Open Source Projects
Aio Scrapy ⭐ 43
Implement scrapy with asyncio
total releases 33latest release October 20, 2023most recent commit 3 months ago
Scrapyd ⭐ 2,766
A service daemon to run Scrapy spiders
dependent packages 15total releases 11latest release September 25, 2023most recent commit 3 months agopypi scrapyd} Downloads
Goscrapy ⭐ 34
GoScrapy: Harnessing Go's power for efficient web scraping, inspired by Python's Scrapy framework.
total releases 26latest release September 20, 2023most recent commit 3 months ago
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7latest release September 20, 2023most recent commit 6 months ago
Scrapy Puppeteer ⭐ 45
Library that helps use puppeteer in scrapy.
dependent packages 1total releases 10latest release September 11, 2023most recent commit 3 months agopypi scrapy-puppeteer-client} Downloads
Spidermon ⭐ 486
Scrapy Extension for monitoring spiders execution.
total releases 18latest release September 01, 2023most recent commit 5 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.