Crawler User Agents
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Colly | 21,902 | 81 | 328 | a month ago | 22 | March 08, 2022 | 181 | apache-2.0 | Go | |
| Elegant Scraper and Crawler Framework for Golang | ||||||||||
| Easyspider | 20,149 | 16 days ago | 6 | other | JavaScript | |||||
| A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。 | ||||||||||
| Avbook | 8,777 | a year ago | 85 | PHP | ||||||
| AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database | ||||||||||
| Awesome Web Scraping | 6,060 | 5 months ago | 1 | other | Makefile | |||||
| List of libraries, tools and APIs for web scraping and data processing. | ||||||||||
| Awesome Crawler | 5,859 | 4 months ago | 27 | mit | ||||||
| A collection of awesome web crawler,spider in different languages | ||||||||||
| Douyin_tiktok_download_api | 4,844 | 5 months ago | 21 | September 23, 2023 | 60 | mit | Python | |||
| 🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。 | ||||||||||
| Browser Fingerprinting | 3,353 | a year ago | 7 | JavaScript | ||||||
| Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web? | ||||||||||
| Querylist | 2,598 |  | 168 | 73 | 2 years ago | 21 | July 05, 2021 | 12 | PHP | |
| :spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。 | ||||||||||
| Grab | 2,292 |  | 107 | 3 | 10 months ago | 120 | June 24, 2018 | 1 | mit | Python |
| Web Scraping Framework | ||||||||||
| Geziyor | 1,892 | 11 | a year ago | 6 | April 19, 2021 | 20 | mpl-2.0 | Go | ||
| Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering. | ||||||||||
Alternatives To Crawler User AgentsSelect To Compare
Alternative Project Comparisons
Popular Spider Projects
Popular Scraper Projects
Popular Data Processing Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Http
Robot
Scraper
Crawler
Spider