Awesome Open Source

Programming Languages

Search results for mongodb scrapy

80 search results found

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Jd_spider ⭐ 728

Two dumb distributed crawlers

Python Spider ⭐ 680

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红

爬取百度百科中文页面，抽取三元组信息，构建中文知识图谱

爬虫实例：微博、b站、csdn、淘宝、今日头条、知乎、豆瓣、知乎APP、大众点评

Scrapy Mongodb ⭐ 327

MongoDB pipeline for Scrapy. This module supports both MongoDB in standalone setups and replica sets. scrapy-mongodb will insert the items to MongoDB as soon as your spider finds data to extract.

Findtrip ⭐ 324

机票爬虫（去哪儿和携程网）。flight tickets multiple webspider.(scrapy + selenium + phantomjs + mongodb)

Data Engineering Projects ⭐ 322

Personal Data Engineering Projects

pigat ( Passive Intelligence Gathering Aggregation Tool ) 被动信息收集聚合工具

Zi5book ⭐ 183

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两

Scrapy_demo ⭐ 150

all kinds of scrapy demo

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题；HTTP协议介绍； Requests使用；解析器Xpath介绍； MongoDB与MySQL；多线程爬虫； Scrapy介绍；Scrapy-redis介绍；使用docker部署；使用nomad管理docker集群；使用EFK查询docker日志

Openscraper ⭐ 80

An open source webapp for scraping: towards a public service for webscraping

Distributed Multi User Scrapy System With A Web Ui ⭐ 71

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Daguerrespider ⭐ 69

50张配图，超细教程！一步一步的教你用Scrapy爬取草榴网站的图片，并下载到本地。欢迎star

Fishfishjump ⭐ 57

Fish Fish Jump is a solution in the python that simply and basic for search engines. 🐟 🐟 🐟

Devsearch ⭐ 45

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Scrapy Flask Imdb Python ⭐ 44

Python project scraping imdb and web application implemented using Flask.

MongoDB extensions for Scrapy

Sinahousecrawler ⭐ 42

基于scrapy,scrapy-redis实现的一个分布式网络爬虫,爬取了新浪房产的楼盘信息及户型图

Scrapy Admin ⭐ 40

A django admin site for scrapy

Buscaimoveis Scraper ⭐ 32

Projeto que coleta anúncios de imóveis a venda em grandes plataformas como OLX, Zap Imóveis, etc

Crawlerflow ⭐ 30

Web Crawlers orchestration framework that lets you create datasets from multiple web sources using yaml configurations.

Epicscrapy1024 ⭐ 25

BOOM💥BOOM💥BOOM💥!! Python3 + Scrapy + MongoDB . 5 million data and 10 gigabyte torrent file per day !!! 💥 The world's largest Chinese BBS.

Raindrop Spider ⭐ 24

A simple distribute spider based on scrapy framework.

Autohome ⭐ 23

Using Scrapy to crawl Autohome, storage into MonogDB, simple analysis and NLP coming soon

Poky Engine ⭐ 21

A simple search engine in python using Tornado, Scrapy, Redis and MongoDB

Scholarscape ⭐ 21

Scrapyuniversal ⭐ 20

基于Scrapy的通用爬虫框架

Restaurant Finder Featurereviews ⭐ 19

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

Detectorist Scraper ⭐ 19

A scrapy spider to extract post, thread, and user information from a vBulletin forum to a MongoDB database.

Movie Scrapy ⭐ 19

时光网电影数据和海报爬虫

Youkunews ⭐ 17

一个基于 Scrapy 的优酷资讯视频爬虫

News Intelligent Classification Wechat Mini Program ⭐ 17

基于 Scrapy 的新闻智能分类微信小程序，是一个文本分类相关的应用，目的是打造出一个可以对新闻进行智能分类的微信小程 + Scrapy + MongoDB + scikit-learn + Flask + 微信小程序，涉及爬虫、文本分类、Web 开发和微信小程序。

Zhihu_data ⭐ 16

Scrapy Pipelines ⭐ 16

A collection of pipelines for Scrapy

The world's leading data crawler platform!

Pinduoduo_spiders ⭐ 15

Scrapy框架,抓取商品信息(已爬70w+数据)

Newspaper Crawler ⭐ 15

Scrapy based crawler which crawls newspaper.

Bili_rank ⭐ 15

Meituanspider ⭐ 13

美团爬虫，基于scrapy_redis

Scrapyproject ⭐ 13

Scrapy项目（mysql+mongodb豆瓣top250电影）

Scrapy Rotated Proxy ⭐ 13

A scrapy middleware to use rotated proxy ip list.

Olx_scraper ⭐ 13

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Scrapy Mongodb Queue ⭐ 13

Use scrapy with mongodb to store the request queues (FIFO or LIFO)

Scrapyd Mongodb ⭐ 12

Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management

Hupu_data ⭐ 12

Web_full_stack_application ⭐ 12

show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Moroccanhousing Etl ⭐ 11

Moroccan housing data pipeline using scrapy, mongodb , zyte and digitalocean cloud

Cors Api ⭐ 11

An Unofficial API for CORS.

Estatematch ⭐ 11

EstateMatch is a mobile application that uses a web-scraper to find Real Estate, sort it using AI, and display it to the user based on their preferences using a Tinder type design, making it easier to search and find your ideal property match.

Jianshuspider ⭐ 11

Use Node.js,HighChart,BootStrap,Mongo,Cucumber with Gulp to scrapy information from Jianshu.

Scrapy Mongodb Pipeline ⭐ 9

MongoDB pipeline for Scrapy. It allows to update existing entries (set new values or add elements to array) when item values are spread over multiple pages

Jobcrawler ⭐ 9

Scrapy Project For Crawling Job Information on 51Job. 基于Scrapy+Python3的51Job招聘信息爬虫

Management Platform For Python Spider Project

一个 python scrapy 爬虫 utility，定制任何我想抓取的web infomation！

Nse Stock Scraper ⭐ 8

This is Web Scraper utilizing Scrapy Framework, MongoDB and AfricasTalking to get stock prices for companies listed on the Nairobi Stock Exchange. This project will store ticker name and price as well notify via SMS once properly setup via AfricasTalking.

Disqus Crawler ⭐ 8

Crawl DISQUS comments from a blog into a local MongoDB database

Crepricespider ⭐ 8

中国房价行情平台creprice.cn爬虫，基于scrapy、redis、mongodb进行开发

Scrapy Httpcache ⭐ 8

A scrapy middleware to save http cache to MongoDB

Bangumispider ⭐ 8

对Bangumi.tv进行爬虫

Githubcrawler ⭐ 8

分布式Github爬虫

Django Adminlte Scrapy Mongodb ⭐ 7

machine learning on predicting your first job

Aminer Spider ⭐ 7

Scrapy Pipeline Mongodb ⭐ 7

A pipeline of saving item into MongoDB

Scrapy_myanimelist ⭐ 7

Crawl anime, reviews and profiles from myAnimeList.net

Pico Nova ⭐ 7

Python scraper / crawler for various torrent sites

Waseda Syllabus Scraper ⭐ 6

A web scraper for scraping the syllabus search database at Waseda University.

Scrapy Zufang ⭐ 6

分布式租房爬虫系统

Scrapy_redis_sxs ⭐ 6

Hkexnews_scrapy ⭐ 5

使用 Scrapy 拿滬港通及深港通持股紀錄

Scrapy_spider ⭐ 5

基于Scrapy-Redis框架与Mongodb的分布式爬虫-elasticsearch搜索引擎打造

I don't like house hunting. I'd rather write a tool instead.

Ds001 Scraping To Analysis Extra Store ⭐ 5

✨ The current project is a basic process pipeline for extraction, transformation, loading, analysis and presentation. All of this was done using appropriate web scraping, data analysis/presentation and database tools.

Webscraping_workshop ⭐ 5

Materials for my workshop "Web Scraping with Scrapy and MongoDB running on Docker" at PyLadies, Berlin, March 31, 2019.

Web_crawler_0608 ⭐ 5

Stockspider ⭐ 5

this is a spider for stocks using MongoDB+scrapy+redis

Coursewebcrawler ⭐ 5

web crawler for courses on scrapy

Job Search Bot ⭐ 5

A Scrapy-based Python web crawler to notify users on a daily basis with up-to-date job postings.

Related Searches

Javascript Mongodb (19,125)

Express Mongodb (7,958)

Reactjs Mongodb (5,012)

Mongodb Mongoose (3,697)

Python Mongodb (2,998)

Mongodb Mongo (2,816)

Typescript Mongodb (2,411)

Python Scrapy (2,369)

Docker Mongodb (2,207)

Java Mongodb (2,017)

1-80 of 80 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.