Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Katana | 7,995 | 1 | 3 months ago | 8 | September 14, 2023 | 67 | mit | Go | ||
A next-generation crawling and spidering framework. | ||||||||||
Ferret | 5,540 | 5 | 4 months ago | 56 | March 28, 2023 | 52 | apache-2.0 | Go | ||
Declarative web scraping | ||||||||||
Gerapy | 3,144 | 8 | 3 months ago | 49 | July 19, 2023 | 60 | mit | Python | ||
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js | ||||||||||
Awesome Puppeteer | 2,245 | 4 months ago | 19 | |||||||
A curated list of awesome puppeteer resources. | ||||||||||
Bilix | 1,433 | 1 | 3 months ago | 77 | July 17, 2023 | 20 | apache-2.0 | Python | ||
⚡️Lightning-fast async download tool for bilibili and more | 快如闪电的异步下载工具,支持bilibili及更多 | ||||||||||
Fetchbot | 758 | 3 | 3 years ago | 7 | May 20, 2021 | 2 | bsd-3-clause | Go | ||
A simple and flexible web crawler that follows the robots.txt policies and crawl delays. | ||||||||||
Fictiondown | 601 | 4 months ago | 5 | February 17, 2020 | 3 | gpl-3.0 | Go | |||
小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对 | ||||||||||
Warcdb | 380 | 5 months ago | 4 | October 22, 2023 | 7 | apache-2.0 | Python | |||
WarcDB: Web crawl data as SQLite databases. | ||||||||||
Sitemap Generator Cli | 259 | 7 | 2 | a year ago | 30 | January 21, 2020 | 29 | mit | JavaScript | |
Creates an XML-Sitemap by crawling a given site. | ||||||||||
Comiccrawler | 251 | 3 months ago | 175 | December 10, 2023 | 25 | Python | ||||
An image crawler written in Python. |