Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Crawler Scrapy Open Source Projects

Open source projects categorized as Crawler Scrapy
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit 3 months agopypi Scrapy} Downloads
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 3 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 5 months ago
Wechatsogou ⭐ 5,777
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit 5 months ago
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18most recent commit 5 months agopypi scrapy-redis} Downloads
Haipproxy ⭐ 5,384
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit a year ago
Ecommercecrawlers ⭐ 3,724
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼
most recent commit a year ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 7 years ago
Gerapy ⭐ 3,144
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 49most recent commit 3 months ago
Python3 Spider ⭐ 2,582
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
most recent commit 6 months ago
Feapder ⭐ 2,333
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、
dependent packages 3total releases 173most recent commit 4 months agopypi feapder} Downloads
Scrapely ⭐ 1,668
A pure-python HTML screen-scraping library
dependent packages 2total releases 13most recent commit 4 years agopypi scrapely} Downloads
Python Crawler ⭐ 1,576
从头开始 系统化的 学习如何写Python爬虫。 Python版本 3.6
most recent commit 2 years ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit 6 months agopypi scutils} Downloads
Querido Diario ⭐ 944
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
most recent commit 3 months ago
Kimuraframework ⭐ 874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
dependent packages 2total releases 10most recent commit 2 years agorubygems kimurai} Downloads
Scrapy Selenium ⭐ 842
Scrapy middleware to handle javascript pages using selenium
dependent packages 2total releases 6most recent commit 6 months agopypi scrapy-selenium} Downloads
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7most recent commit 6 months ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42most recent commit 4 months agopypi icrawler} Downloads
Tweetscraper ⭐ 698
TweetScraper is a simple crawler/spider for Twitter Search without using API
total releases 8most recent commit 3 years ago
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
most recent commit 3 years ago
Python Fxxk Spider ⭐ 571
收集各种免费的 Python 爬虫项目
most recent commit 4 months ago
Domains ⭐ 508
World’s single largest Internet domains dataset
most recent commit 6 months ago
The 10 Most Depended On Crawler Scrapy Open Source Projects
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Random_user_agent ⭐ 79
A package to get list of user agents based on filters such as operating system, software name etc..
dependent packages 30total releases 9latest release December 26, 2018most recent commit 10 months agopypi random-user-agent} Downloads
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18latest release July 26, 2022most recent commit 5 months agopypi scrapy-redis} Downloads
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 4 months agopypi icrawler} Downloads
Scrapy Selenium ⭐ 842
Scrapy middleware to handle javascript pages using selenium
dependent packages 2total releases 6latest release January 24, 2019most recent commit 6 months agopypi scrapy-selenium} Downloads
Scrapely ⭐ 1,668
A pure-python HTML screen-scraping library
dependent packages 2total releases 13latest release November 28, 2019most recent commit 4 years agopypi scrapely} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
The 10 Latest Releases In Crawler Scrapy Open Source Projects
Aio Scrapy ⭐ 43
Implement scrapy with asyncio
total releases 33latest release October 20, 2023most recent commit 3 months ago
Scrapy Zyte Api ⭐ 30
Zyte API integration for Scrapy
dependent packages 1total releases 23latest release October 19, 2023most recent commit 3 months agopypi scrapy-zyte-api} Downloads
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7latest release September 20, 2023most recent commit 6 months ago
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Spidermon ⭐ 486
Scrapy Extension for monitoring spiders execution.
total releases 18latest release September 01, 2023most recent commit 5 months ago
Scrapingant Client Python ⭐ 26
ScrapingAnt API client for Python.
total releases 16latest release August 09, 2023most recent commit 9 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.