Awesome Open Source

Programming Languages

Search results for spider scrapy

307 search results found

Learn_python3_spider ⭐ 14,425

python爬虫教程系列、从0到1学习python爬虫，包括浏览器抓包，手机APP抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifu 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Haipproxy ⭐ 5,384

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Weibospider ⭐ 3,294

持续维护的新浪微博采集工具🚀🚀🚀

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Python3 Spider ⭐ 3,064

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Scrapyd ⭐ 2,766

A service daemon to run Scrapy spiders

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Image Downloader ⭐ 2,029

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Reptile ⭐ 1,081

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼腾讯视频豆瓣研招网微博笔趣阁小说百度热点 B站 CSDN 网易云阅读阿里文学百度股票今日头条微信公众号网易云音乐拉勾有道 unsplash 实习僧汽车之家英雄联盟盒子大众点评链家 LPL赛程台风梦幻西游、阴阳师藏宝阁天气牛客网百度文库睡前故事知乎 Wish

Django Dynamic Scraper ⭐ 1,069

Creating Scrapy scrapers via the Django admin interface

Jspider ⭐ 1,006

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

Kimuraframework ⭐ 874

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Zhihu_spider ⭐ 855

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

Icrawler ⭐ 792

A multi-thread crawler framework with many builtin image crawlers provided.

Spider_python ⭐ 732

Tweetscraper ⭐ 698

TweetScraper is a simple crawler/spider for Twitter Search without using API

Python Spider ⭐ 680

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红

Linkedin ⭐ 602

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Python Fxxk Spider ⭐ 571

收集各种免费的 Python 爬虫项目

Alltheplaces ⭐ 502

A set of spiders and scrapers to extract location information from places that post their location on the internet.

Spiderman ⭐ 498

基于 scrapy-redis 的通用分布式爬虫框架

Scrapy Rotating Proxies ⭐ 474

use multiple proxies with Scrapy

Awesome Scrapy ⭐ 450

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

Spider Admin Pro ⭐ 438

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看和爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

爬取百度百科中文页面，抽取三元组信息，构建中文知识图谱

Fbcrawl ⭐ 415

A Facebook crawler

Newscrawl ⭐ 402

狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑

Scrapybook ⭐ 378

Scrapy Book Code

Ants Go ⭐ 368

open source, distributed, restful crawler engine in golang

爬虫实例：微博、b站、csdn、淘宝、今日头条、知乎、豆瓣、知乎APP、大众点评

Scrapy Mongodb ⭐ 327

MongoDB pipeline for Scrapy. This module supports both MongoDB in standalone setups and replica sets. scrapy-mongodb will insert the items to MongoDB as soon as your spider finds data to extract.

🎊 Design and implement of lightweight crawler framework.

Httpproxymiddleware ⭐ 318

A middleware for scrapy. Used to change HTTP proxy from time to time.

Tieba_spider ⭐ 298

百度贴吧爬虫(基于scrapy和mysql)

Spider_world ⭐ 297

🕷spider world with me

Hotel Review Analysis ⭐ 254

Sentiment analysis and aspect classification for hotel reviews using machine learning models with MonkeyLearn.

Happy Spiders ⭐ 247

🔧 🔩 🔨 收集整理了爬虫相关的工具、模拟登陆技术、代理IP、scrapy模板代码等内容。

Scrapy Jsonrpc ⭐ 238

Scrapy extension to control spiders using JSON-RPC

Scrapy Deltafetch ⭐ 232

Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

Weapp Girls ⭐ 224

wechat app of girls scrapy spider via Node.js

Wayback Machine Scraper ⭐ 219

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Finance_news_analysis ⭐ 206

金融新闻数据挖掘分析

News_spider ⭐ 203

新闻抓取（微信、微博、头条...）

Crawlab Lite ⭐ 184

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Zi5book ⭐ 183

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Jobspiders ⭐ 171

scrapy框架爬取51job(scrapy.Spider)，智联招聘(扒接口)，拉勾网(Crawl

Qqmusicspider ⭐ 168

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手

Goribot ⭐ 162

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Fp Server ⭐ 154

Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池

Scrapy_demo ⭐ 150

all kinds of scrapy demo

Django Covid19 ⭐ 150

实时接口获取中国各个城市、省份、国家的新型冠状肺炎（新冠肺炎 / 2019-nCoV / Covid-19）。疫情数据以及整体统计详情，新增美国各州统计、每日疫情数据 API。爬虫实时追踪新冠疫情变化，数据来自丁香园和 covidtracking.com。数据大屏示例：http://ncov.leafcoder.cn/ 项目文档：http://ncov.leafcoder.cn/docs/

Scrapy_guru ⭐ 146

Everybody can be scrapy guru

Weibosearch ⭐ 144

A distributed Sina Weibo Search spider base on Scrapy and Redis.

Scrapyredisbloomfilter ⭐ 144

Scrapy Redis Bloom Filter

Scrapy Training ⭐ 141

Scrapy Training companion code

Deep Deep ⭐ 130

Adaptive crawler which uses Reinforcement Learning methods

Unmaintained 🐳 ☕ 🕷️ Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege

Linkedinscraper ⭐ 112

Scrapes public information off of LinkedIn

Autologin ⭐ 106

A project to attempt to automatically login to a website given a single seed

Scrapyd Django Template ⭐ 103

Basic setup to run ScrapyD + Django and save it in Django Models. You can be up and running in just a few minutes

lots of spider (很多爬虫）

Jkcrawler ⭐ 100

使用 Scrapy 写成的 JK 爬虫，图片源自哔哩哔哩、Tumblr、Instagram，以及微博、Twitter

Copybook ⭐ 97

用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站

Scrapyscript ⭐ 92

Run a Scrapy spider programmatically from a script or a Celery task - no project required.

Scrapy_ipproxypool ⭐ 86

免费 IP 代理池。Scrapy 爬虫框架插件

Blockchainspider ⭐ 83

A toolkit for blockchain data collection

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

Python Spider ⭐ 81

python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Openscraper ⭐ 80

An open source webapp for scraping: towards a public service for webscraping

Weibospider ⭐ 79

微博爬虫，一个基于Scrapy框架的轻量微博爬虫，Sina Weibo Spider

Awesome Python Primer ⭐ 78

自学入门 Python 优质中文资源索引，包含书籍 / 文档 / 视频，适用于爬虫 / Web / 数据分析 / 机器学习方向

Couch Crawler ⭐ 77

A search engine built on top of couchdb-lucene

Dictionary_crawler ⭐ 76

This is a python code based on Scrapy package to crawl famous online dictionaries like Oxford, Longman, Cambridge, Webster, and Collins to make a dataset

Memex Program Index ⭐ 76

A list of memex-related tools and their repository URLs

Get itbooks from ebooks's website for free,such as allitebooks,digilibraries,etc

Inventus ⭐ 74

Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.

Distributed Multi User Scrapy System With A Web Ui ⭐ 71

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Daguerrespider ⭐ 69

50张配图，超细教程！一步一步的教你用Scrapy爬取草榴网站的图片，并下载到本地。欢迎star

Tumblrspider ⭐ 67

使用scrapy编写的python爬虫

Scrapy S3pipeline ⭐ 66

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

Scrapy Spider Example ⭐ 62

Scrapy spider example for Scrapy Tutorial Series

Web Iota ⭐ 60

Iota is a web scraper which can find all of the images and links/suburls on a webpage

Awsomespider ⭐ 58

Python爬虫小项目汇总（招聘信息/电影信息/股票信息/天气信息/贴吧信息/图片信息/视频信息..

Webwalker ⭐ 55

分类下子项目信息抓取

Godataaccess ⭐ 54

🪲Data access framework in native Golang(Golang实现的类Scrapy框架)

Scrapy python crawler/spider with post/get login (handles CSRF), variable level of recursions and optionally save to disk

A scrapy project to extract the text and metadata of articles from news websites

Scrapy Boilerplate ⭐ 49

Small set of utilities to simplify writing Scrapy spiders.

Scrapy Pyppeteer ⭐ 48

Use pyppeteer from a Scrapy spider

Github Trending ⭐ 47

GitHub trending repositories and developers APIs for real time, powered by crawlers | 通过爬虫获取 GitHub 热门项目和开发者的实时 API

Scrapy Do ⭐ 46

A daemon for scheduling Scrapy spiders

Scrapybook 2nd Edition ⭐ 45

Scrapy Book 2nd Edition Code http://scrapybook.com/

Devsearch ⭐ 45

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Related Searches

Python Scrapy (2,438)

Python Spider (2,155)

Crawler Scrapy (994)

Crawler Spider (709)

Scraper Scrapy (575)

Javascript Spider (442)

1-100 of 307 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2025 Awesome Open Source. All rights reserved.