Scrapy
项目实例：一个学习scrapy的简单实例。帮助你快速的上手scrapy框架。 只需修改2个python文件。items.py 和spiders文件夹中的shushan.py。 需要修改的项，在2个python文件中均进行了备注。 大家可根据备注修改相关内容，再通过命令行运行爬虫程序。 命令行cd至spider目录,运行scrapy crawl shushan -o shushan.csv，生成csv文件，保存爬虫数据。 备注：保存的爬虫数据csv格式，需用WPS版excel打开，或是用txt直接打开。点击右上 star 按钮，喜欢的点个赞吧！（网站也是本人弄的，请放心使用）
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Rod | 4,505 | 140 | 4 months ago | 406 | November 06, 2023 | 106 | mit | Go | ||
| A Devtools driver for web automation and scraping | ||||||||||
| Skrape.it | 714 | 3 | 4 months ago | 14 | July 19, 2022 | 25 | mit | Kotlin | ||
| A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion. | ||||||||||
| Thea11ymachine | 553 |  | 4 | 2 | 4 years ago | 26 | February 08, 2017 | 35 | JavaScript | |
| The A11y Machine is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports. | ||||||||||
| Crawljax | 493 | 28 | 6 | a year ago | 18 | June 01, 2023 | 67 | apache-2.0 | Java | |
| Crawljax | ||||||||||
| Spidermon | 486 | 5 months ago | 18 | September 01, 2023 | 52 | bsd-3-clause | Python | |||
| Scrapy Extension for monitoring spiders execution. | ||||||||||
| Bolt | 482 | 4 months ago | 3 | gpl-3.0 | Python | |||||
| CSRF Scanner | ||||||||||
| Tarantula | 453 |  | 5 | 1 | 11 years ago | 13 | May 24, 2013 | 12 | mit | Ruby |
| a big hairy fuzzy spider that crawls your site, wreaking havoc | ||||||||||
| Black Widow | 168 | 2 years ago | 13 | June 17, 2022 | 2 | gpl-3.0 | Python | |||
| GUI based offensive penetration testing tool (Open Source) | ||||||||||
| Security Crawl Maze | 135 | 10 months ago | apache-2.0 | HTML | ||||||
| Security Crawl Maze is a comprehensive testbed for web security crawlers. It contains pages representing many ways in which one can link resources from a valid HTML document. | ||||||||||
| Reqman | 124 |  | 1 | 1 | 5 years ago | 6 | March 08, 2019 | 1 | mit | JavaScript |
| Reqman is a tool that can quickly help back-end engineers with api testing, as well as a nodejs-based crawler tool. | ||||||||||
Alternatives To ScrapySelect To Compare
Alternative Project Comparisons
Popular Crawler Projects
Popular Testing Projects
Popular Data Processing Categories
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Testing
Crawler
Spider
Scrapy
Xpath