Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Http Status Check | 587 | 4 | 1 | 4 months ago | 22 | June 14, 2023 | mit | PHP | ||
CLI tool to crawl a website and check HTTP status codes | ||||||||||
Curl Easy | 282 | 18 | 8 | 5 years ago | 9 | May 20, 2017 | 5 | mit | PHP | |
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot | ||||||||||
Google Group Crawler | 213 | 2 years ago | 6 | Shell | ||||||
[Deprecated] Get (almost) original messages from google group archives. Your data is yours. | ||||||||||
Awesome Java Crawler | 172 | 4 years ago | ||||||||
本仓库收集整理爬虫相关资源,开发语言以Java为主 | ||||||||||
Grawler | 128 | 3 years ago | 1 | mit | PHP | |||||
Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file. | ||||||||||
Seojs | 99 | 11 years ago | 1 | JavaScript | ||||||
Mycelium | 85 | 10 months ago | 2 | other | C++ | |||||
An open source information retrieval system written in C++11 and Python. Aspires to be an alternative to Nutch / Lucene. It uses MongoDB as an storage engine. | ||||||||||
Argus | 67 | 2 years ago | 3 | gpl-3.0 | Python | |||||
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9 | ||||||||||
Packagist Crawler | 50 | 4 years ago | 1 | January 01, 2015 | 1 | other | PHP | |||
make mirror of https://packagist.org | ||||||||||
Caterpillar | 39 | 8 years ago | other | PHP | ||||||
Caterpillar is a PHP library intended for website crawling and screen scraping. It handles parallel requests using the curl_multi functions. |