Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Spidermon | 486 | 5 months ago | 18 | September 01, 2023 | 52 | bsd-3-clause | Python | |||
Scrapy Extension for monitoring spiders execution. | ||||||||||
Scrapyredisbloomfilter | 144 | 3 years ago | 5 | January 03, 2021 | 6 | Python | ||||
Scrapy Redis Bloom Filter | ||||||||||
Scrapy Autounit | 42 | 3 years ago | 32 | July 09, 2021 | 10 | bsd-3-clause | Python | |||
Automatic unit test generation for Scrapy. | ||||||||||
Captcha Middleware | 38 | 3 years ago | gpl-3.0 | Python | ||||||
A middleware layer for Scrapy that detects CAPTCHA tests and solves them | ||||||||||
Scurl | 15 | 6 years ago | 1 | August 01, 2018 | 11 | apache-2.0 | Python | |||
Performance-focused replacement for Python urllib | ||||||||||
Scrapy Testmaster | 12 | 3 years ago | 1 | other | Python | |||||
The most advanced debugging and testing tool for Scrapy | ||||||||||
Flask Scrapy Test | 7 | 7 years ago | 1 | Python | ||||||
for kristjan | ||||||||||
Scrapy Mock | 6 | a year ago | 1 | January 06, 2020 | 1 | mit | Python | |||
Record Scrapy responses and use them as testing fixtures. | ||||||||||
Scrapy | 6 | 6 years ago | Python | |||||||
项目实例:一个学习scrapy的简单实例。帮助你快速的上手scrapy框架。 只需修改2个python文件。items.py 和spiders文件夹中的shushan.py。 需要修改的项,在2个python文件中均进行了备注。 大家可根据备注修改相关内容,再通过命令行运行爬虫程序。 命令行cd至spider目录,运行scrapy crawl shushan -o shushan.csv,生成csv文件,保存爬虫数据。 备注:保存的爬虫数据csv格式,需用WPS版excel打开,或是用txt直接打开。点击右上 star 按钮,喜欢的点个赞吧!(网站也是本人弄的,请放心使用) | ||||||||||
Jinritoutiaonews | 5 | 4 years ago | Python | |||||||
scrapy+pyppeteer,爬取今日头条中新闻及热门评论信息。 |