Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Crawler Scrapy Open Source Projects

Open source projects categorized as Python Crawler Scrapy
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit 3 months agopypi Scrapy} Downloads
Wechatsogou ⭐ 5,753
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit 5 months ago
Ecommercecrawlers ⭐ 3,724
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼
most recent commit a year ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 7 years ago
Gerapy ⭐ 3,144
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 49most recent commit 3 months ago
Python3 Spider ⭐ 2,582
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
most recent commit 5 months ago
Feapder ⭐ 2,333
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、
dependent packages 3total releases 173most recent commit 3 months agopypi feapder} Downloads
Python Crawler ⭐ 1,576
从头开始 系统化的 学习如何写Python爬虫。 Python版本 3.6
most recent commit 2 years ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit 5 months agopypi scutils} Downloads
Querido Diario ⭐ 944
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
most recent commit 3 months ago
Scrapy Selenium ⭐ 842
Scrapy middleware to handle javascript pages using selenium
dependent packages 2total releases 6most recent commit 6 months agopypi scrapy-selenium} Downloads
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7most recent commit 5 months ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42most recent commit 3 months agopypi icrawler} Downloads
Tweetscraper ⭐ 698
TweetScraper is a simple crawler/spider for Twitter Search without using API
total releases 8most recent commit 3 years ago
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
most recent commit 3 years ago
Python Fxxk Spider ⭐ 571
收集各种免费的 Python 爬虫项目
most recent commit 4 months ago
Spidermon ⭐ 486
Scrapy Extension for monitoring spiders execution.
total releases 18most recent commit 5 months ago
Personrelationknowledgegraph ⭐ 480
ChinesePersonRelationGraph, person relationship extraction based on nlp methods.中文人物关系知识图谱项目,内容包括中文人物关系图谱构建,基于知识库的数据回标,基于远
most recent commit 5 years ago
Vault ⭐ 477
swiss army knife for hackers
most recent commit 9 months ago
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
total releases 10most recent commit 5 years ago
Awesome Scrapy ⭐ 450
A curated list of awesome packages, articles, and other cool resources from the Scrapy community.
most recent commit a year ago
Fbcrawl ⭐ 415
A Facebook crawler
most recent commit 4 years ago
Scrapybook ⭐ 378
Scrapy Book Code
most recent commit 6 years ago
The 8 Most Depended On Python Crawler Scrapy Open Source Projects
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Random_user_agent ⭐ 79
A package to get list of user agents based on filters such as operating system, software name etc..
dependent packages 30total releases 9latest release December 26, 2018most recent commit 10 months agopypi random-user-agent} Downloads
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 3 months agopypi icrawler} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 5 months agopypi scutils} Downloads
Scrapy Selenium ⭐ 842
Scrapy middleware to handle javascript pages using selenium
dependent packages 2total releases 6latest release January 24, 2019most recent commit 6 months agopypi scrapy-selenium} Downloads
Scrapy Crawl Once ⭐ 50
Scrapy middleware which allows to crawl only new content
dependent packages 1total releases 2latest release March 03, 2017most recent commit 4 years agopypi scrapy-crawl-once} Downloads
Scrapy Zyte Api ⭐ 30
Zyte API integration for Scrapy
dependent packages 1total releases 23latest release October 19, 2023most recent commit 3 months agopypi scrapy-zyte-api} Downloads
The 10 Latest Releases In Python Crawler Scrapy Open Source Projects
Aio Scrapy ⭐ 43
Implement scrapy with asyncio
total releases 33latest release October 20, 2023most recent commit 3 months ago
Scrapy Zyte Api ⭐ 30
Zyte API integration for Scrapy
dependent packages 1total releases 23latest release October 19, 2023most recent commit 3 months agopypi scrapy-zyte-api} Downloads
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7latest release September 20, 2023most recent commit 5 months ago
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Spidermon ⭐ 486
Scrapy Extension for monitoring spiders execution.
total releases 18latest release September 01, 2023most recent commit 5 months ago
Scrapingant Client Python ⭐ 26
ScrapingAnt API client for Python.
total releases 16latest release August 09, 2023most recent commit 8 months ago
Gerapy ⭐ 3,144
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 49latest release July 19, 2023most recent commit 3 months ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 3 months agopypi icrawler} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.