Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Scrapy Open Source Projects

Open source projects categorized as Scrapy
Scrapy Puppeteer ⭐ 103
Scrapy + Puppeteer
total releases 1most recent commit 3 years ago
Aox_proxy_pool ⭐ 103
本项目是为了解决在抓取代理ip后, 代理ip失效快, 不稳定的问题 以及代理ip使用不方便等问题。
most recent commit 5 years ago
Docs ⭐ 102
《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志
most recent commit 5 years ago
Hive ⭐ 101
lots of spider (很多爬虫）
most recent commit 5 years ago
Jkcrawler ⭐ 100
使用 Scrapy 写成的 JK 爬虫，图片源自哔哩哔哩、Tumblr、Instagram，以及微博、Twitter
most recent commit 3 years ago
Sequentialeventextration ⭐ 99
Sequential Event Experiment based on Travel note crawled from XieCheng，基于50W携程出行游记的采集与顺承事件图谱构建．
most recent commit 5 years ago
Scrapy Monitor ⭐ 99
scrapy-monitor，实现爬虫可视化，监控实时状态
most recent commit 7 years ago
Verticlesearchengine ⭐ 98
Academic Search Engine using Scrapy, MongoDB, Lucene/Solr, Tika, Struts2, Jquery, Bootstrap, D3, CAS
most recent commit 11 years ago
Copybook ⭐ 97
用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站
most recent commit 4 years ago
Scraper ⭐ 96
Firmware scraper
most recent commit a year ago
Wswp ⭐ 95
Code for the second edition Web Scraping with Python book by Packt Publications
most recent commit 4 years ago
Capturer ⭐ 94
capture pictures from website like sina, lofter, huaban and so on
most recent commit 2 years ago
Terpene Profile Parser For Cannabis Strains ⭐ 93
Parser and database to index the terpene profile of different strains of Cannabis from online databases
most recent commit a year ago
Scrapy Gui ⭐ 93
A simple, Qt-Webengine powered web browser with built in functionality for basic scrapy webscraping support.
total releases 9most recent commit a year ago
Scrapyscript ⭐ 92
Run a Scrapy spider programmatically from a script or a Celery task - no project required.
total releases 10most recent commit 2 years ago
Java Curl ⭐ 92
Pure java CURL implementation
total releases 5most recent commit 3 years ago
Easypoi ⭐ 91
简单、免费、高效的百度地图poi采集和分析工具。
most recent commit a year ago
Pagser ⭐ 91
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
dependent packages 5total releases 3most recent commit 7 months ago
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 4 years ago
Weibo Album Crawler ⭐ 90
新浪微博相册大图多线程爬虫。
most recent commit 2 years ago
Android Apps Crawler ⭐ 88
An extensible crawler for downloading Android applications in third-party markets.
most recent commit 8 years ago
Funda Scraper ⭐ 88
Scraper of the Dutch real estate website www.funda.nl, implemented in Python with Scrapy
most recent commit 7 years ago
Python_demo ⭐ 88
一些简单有趣的Python小Demo
most recent commit 5 years ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.