Awesome Open Source

Programming Languages

The Top 23 Crawler Spider Open Source Projects

Open source projects categorized as Crawler Spider

Categories > Data Processing > Crawler

Categories > Data Processing > Spider

Colly ⭐ 21,902

Elegant Scraper and Crawler Framework for Golang

dependent packages 328total releases 22most recent commit a month ago

Easyspider ⭐ 20,149

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化

most recent commit 15 days ago

Proxy_pool ⭐ 19,442

Python ProxyPool for web spider

most recent commit 3 months ago

Pyspider ⭐ 15,943

A Powerful Spider(Web Crawler) System in Python.

dependent packages 2total releases 17most recent commit 9 months ago

pypi pyspider} Downloads

Examples Of Web Crawlers ⭐ 13,142

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

most recent commit 4 months ago

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1most recent commit 3 months ago

Photon ⭐ 10,244

Incredibly fast crawler designed for OSINT.

total releases 18most recent commit 3 months ago

Avbook ⭐ 8,777

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

most recent commit a year ago

Spider Flow ⭐ 8,075

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

most recent commit 10 months ago

Infospider ⭐ 6,856

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透

most recent commit 6 months ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31most recent commit 3 months ago

npm crawler} Downloads

Awesome Web Scraping ⭐ 6,060

List of libraries, tools and APIs for web scraping and data processing.

most recent commit 5 months ago

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

most recent commit 4 months ago

Haipproxy ⭐ 5,384

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

total releases 7most recent commit a year ago

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

total releases 21most recent commit 5 months ago

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

most recent commit a year ago

Toapi ⭐ 3,417

Every web site provides APIs.

most recent commit 2 years ago

Novel Plus ⭐ 3,358

novel-plus 是一个多端（PC、WAP）阅读、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、

most recent commit 3 months ago

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

most recent commit a year ago

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

most recent commit 7 years ago

Toapi ⭐ 3,153

Every web site provides APIs.

total releases 40most recent commit 3 years ago

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

total releases 49most recent commit 3 months ago

Querylist ⭐ 2,598

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

dependent packages 73total releases 21most recent commit 2 years ago

packagist jaeger/querylist} Downloads

The 10 Most Depended On Crawler Spider Open Source Projects

Colly ⭐ 21,902

Elegant Scraper and Crawler Framework for Golang

dependent packages 328total releases 22latest release March 08, 2022most recent commit a month ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31latest release December 30, 2022most recent commit 3 months ago

npm crawler} Downloads

Crawler Detect ⭐ 1,842

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

dependent packages 110total releases 156latest release July 21, 2023most recent commit 5 months ago

packagist jaybizzle/crawler-detect} Downloads

Zhihu Api ⭐ 267

Unofficial API for zhihu.

dependent packages 103total releases 39latest release July 16, 2017most recent commit 7 years ago

npm zhihu-api} Downloads

Querylist ⭐ 2,598

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

dependent packages 73total releases 21latest release July 05, 2021most recent commit 2 years ago

packagist jaeger/querylist} Downloads

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15latest release October 03, 2021most recent commit 2 years ago

npm jssoup} Downloads

Anemone ⭐ 1,615

Anemone web-spider framework

dependent packages 34total releases 23latest release May 30, 2012most recent commit 4 years ago

rubygems anemone} Downloads

Laravel Crawler Detect ⭐ 262

A Laravel wrapper for CrawlerDetect - the web crawler detection library

dependent packages 16total releases 6latest release July 06, 2022most recent commit 2 years ago

packagist jaybizzle/laravel-crawler-detect} Downloads

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

dependent packages 16total releases 26latest release October 25, 2019most recent commit 3 months ago

rubygems spidr} Downloads

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

dependent packages 11total releases 6latest release April 19, 2021most recent commit a year ago

The 10 Latest Releases In Crawler Spider Open Source Projects

The fastest web crawler written in Rust. Maintained by @a11ywatch.

dependent packages 5total releases 368latest release December 11, 2023most recent commit 3 months ago

cargo spider} Downloads

Device_detector ⭐ 711

DeviceDetector is a precise and fast user agent parser and device detector written in Ruby

dependent packages 5total releases 26latest release December 01, 2023most recent commit 5 months ago

rubygems device_detector} Downloads

Go Crawler ⭐ 33

A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middleware and supports various parsing and storage methods. Additionally, it supports distributed deployment. 基于golang实现的爬虫框架，编写简单，性能强劲。内置了丰富的实用中间件，支持多种解析、保存方式，

total releases 26latest release November 23, 2023most recent commit 5 months ago

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

dependent packages 8total releases 118latest release November 20, 2023most recent commit 3 months ago

npm crawler-user-agents} Downloads

Web crawler based on Puppeteer

dependent packages 4total releases 23latest release November 13, 2023most recent commit 5 months ago

npm @opd/crawler} Downloads

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

dependent packages 1total releases 59latest release November 09, 2023most recent commit 3 months ago

npm x-crawl} Downloads

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months ago

npm webster} Downloads

Phpcreeper ⭐ 108

A new generation of multi-process asynchronous event-driven spider engine based on Workerman. http://www.phpcreeper.com

total releases 45latest release November 08, 2023most recent commit 3 months ago

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

dependent packages 3total releases 173latest release November 07, 2023most recent commit 3 months ago

pypi feapder} Downloads

Gathertool ⭐ 46

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口

total releases 58latest release November 01, 2023most recent commit 5 months ago

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.