Scrapy Mongodb

MongoDB pipeline for Scrapy. This module supports both MongoDB in standalone setups and replica sets. scrapy-mongodb will insert the items to MongoDB as soon as your spider finds data to extract.
Alternatives To Scrapy Mongodb
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Learn_python3_spider14,425
3 months ago2August 07, 201929mitPython
python爬虫教程系列、从0到1学习python爬虫,包括浏览器抓包,手机APP抓包,如 fiddler、mitmproxy,各种爬虫涉及的模块的使用,如:requests、beautifulSoup、selenium、appium、scrapy等,以及IP代理,验证码识别,Mysql,MongoDB数据库的python使用,多线程多进程爬虫的使用,css 爬虫加密逆向破解,JS爬虫逆向,分布式爬虫,爬虫项目实战实例等
Crawlab10,521
3 months ago1March 03, 201958bsd-3-clauseGo
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Awesome Crawler5,859
4 months ago27mit
A collection of awesome web crawler,spider in different languages
Haipproxy5,372
1a year ago7June 18, 201844mitPython
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Ecommercecrawlers3,724
a year ago43mitPython
实战🐍多种网站、电商数据爬虫🕷。包含🕸:淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
Weibospider3,294
3 months ago7mitPython
持续维护的新浪微博采集工具🚀🚀🚀
Distribute_crawler3,176
7 years ago26Python
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Gerapy3,144
83 months ago49July 19, 202360mitPython
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Scrapydweb2,839
35 months ago18August 31, 202356gpl-3.0Python
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
Scrapyd2,766187152 months ago11September 25, 202331bsd-3-clausePython
A service daemon to run Scrapy spiders
Alternatives To Scrapy Mongodb
Select To Compare


Alternative Project Comparisons
Popular Spider Projects
Popular Scrapy Projects
Popular Data Processing Categories

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Database
Mongodb
Spider
Scrapy