Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Crawler | 2,417 | 116 | 28 | 4 months ago | 101 | November 22, 2023 | mit | PHP | ||
An easy to use, powerful crawler implemented in PHP. Can execute Javascript. | ||||||||||
Weixin Game Helper | 1,352 | a year ago | 24 | gpl-3.0 | JavaScript | |||||
微信小游戏辅助合集(加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星途WeGoing、猜画小歌、知乎答题王、腾讯中国象棋、跳一跳、题多多黄金版) | ||||||||||
Crawler User Agents | 1,045 | 5 | 8 | 4 months ago | 118 | November 20, 2023 | 7 | mit | Python | |
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star: | ||||||||||
Supercrawler | 324 | 1 | 3 | 2 years ago | 45 | November 24, 2019 | 19 | apache-2.0 | JavaScript | |
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits. | ||||||||||
Robots Txt | 201 | 43 | 8 | 6 months ago | 18 | November 22, 2023 | mit | PHP | ||
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers | ||||||||||
Gflare Tk | 110 | 10 months ago | 7 | February 15, 2021 | 13 | gpl-3.0 | Python | |||
Open-Source Python Based SEO Web Crawler | ||||||||||
Auto Lighthouse | 105 | 2 | a year ago | 40 | August 18, 2022 | 14 | gpl-3.0 | HTML | ||
A utility package for automating lighthouse reporting | ||||||||||
Human_power | 97 | 2 | 9 years ago | 6 | May 12, 2014 | 8 | mit | Ruby | ||
Easy generation of robots.txt. Force the robots into submission! | ||||||||||
Roboto | 79 | 49 | 1 | 7 years ago | 6 | June 30, 2017 | 2 | mit | Ruby | |
Environment specific robots.txt for your Rails Apps | ||||||||||
Robots.txt | 69 | 3 years ago | other | |||||||
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites. |