Awesome Open Source

Programming Languages

Search results for crawler spider

368 search results found

Easyspider ⭐ 36,416

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化

Colly ⭐ 23,382

Elegant Scraper and Crawler Framework for Golang

Proxy_pool ⭐ 19,442

Python ProxyPool for web spider

Pyspider ⭐ 15,943

A Powerful Spider(Web Crawler) System in Python.

Examples Of Web Crawlers ⭐ 13,142

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Photon ⭐ 10,244

Incredibly fast crawler designed for OSINT.

Avbook ⭐ 8,777

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Spider Flow ⭐ 8,075

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Haipproxy ⭐ 5,384

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Toapi ⭐ 3,417

Every web site provides APIs.

Novel Plus ⭐ 3,358

novel-plus 是一个多端（PC、WAP）阅读、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Toapi ⭐ 3,153

Every web site provides APIs.

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Python3 Spider ⭐ 3,064

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Querylist ⭐ 2,598

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Lianjia Beike Spider ⭐ 2,464

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Decryptlogin ⭐ 2,375

DecryptLogin: APIs for loginning some websites by using requests.

Owllook ⭐ 2,340

owllook-小说搜索引擎

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Web Scraping Framework

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

Web crawling framework based on asyncio.

Web crawling framework based on asyncio.

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Crawler Detect ⭐ 1,842

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Async Python 3.6+ web scraping micro-framework based on asyncio

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Anemone ⭐ 1,615

Anemone web-spider framework

Open Source Search Engine ⭐ 1,504

Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.

Grab Site ⭐ 1,418

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Catvodtvspider ⭐ 1,365

Php Spider ⭐ 1,316

A configurable and extensible PHP web spider

Catvodtvspider ⭐ 1,270

Beanbun ⭐ 1,195

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

Python website crawler.

Kimuraframework ⭐ 874

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Baiduspider ⭐ 872

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索

Zhihu Crawler ⭐ 843

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

Icrawler ⭐ 792

A multi-thread crawler framework with many builtin image crawlers provided.

Crawly, a high-level web crawling & scraping framework for Elixir.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Baiduimagespider ⭐ 774

一个超级轻量的百度图片爬虫

Creeper ⭐ 762

🐾 Creeper - The Next Generation Crawler Framework (Go)

Spider_collection ⭐ 754

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

Device_detector ⭐ 711

DeviceDetector is a precise and fast user agent parser and device detector written in Ruby

Tweetscraper ⭐ 698

TweetScraper is a simple crawler/spider for Twitter Search without using API

Xxl Crawler ⭐ 650

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

Fictiondown ⭐ 601

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Newcrawler ⭐ 583

Free Web Scraping Tool with Java

Python Fxxk Spider ⭐ 571

收集各种免费的 Python 爬虫项目

Netdiscovery ⭐ 557

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

API of DouYin for Humans used to Crawl Popular Videos and Musics

Go_jobs ⭐ 536

带你了解一下Golang的市场行情

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

Tarantula ⭐ 453

a big hairy fuzzy spider that crawls your site, wreaking havoc

Awesome Scrapy ⭐ 450

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

Crack Js Spider ⭐ 442

JS破解逆向，破解JS反爬虫加密参数，已破解极验滑块w（2022.2.19），QQ音乐sign（20

Learnpython ⭐ 437

Python的基础练习代码与各种爬虫代码

《爬虫逆向进阶实战》书籍代码库

The fastest web crawler written in Rust. Maintained by @a11ywatch.

Html2article ⭐ 425

Html网页正文提取

Fbcrawl ⭐ 415

A Facebook crawler

Linkedin Profile Scraper Api ⭐ 404

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

Scrapybook ⭐ 378

Scrapy Book Code

Ants Go ⭐ 368

open source, distributed, restful crawler engine in golang

Gospider ⭐ 354

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Zhihu Login ⭐ 350

知乎模拟登录，支持提取验证码和保存 Cookies

91porn Api ⭐ 341

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新）及在线web预览

Linkedindumper ⭐ 337

Python 3 script to dump/scrape/extract company employees from LinkedIn API

Free_proxy_website ⭐ 333

获取免费socks/https/http代理的网站集合

Node Readability ⭐ 302

Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.

Webpalm ⭐ 295

WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.

Crawler ⭐ 288

K 哥爬虫代码分享，JS 逆向，爬虫进阶。关注公众号：K哥爬虫

Magic_google ⭐ 287

Google search results crawler, get google search results that you need

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Ppspider ⭐ 278

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（ne

A web crawler for Go

Zhihu Api ⭐ 267

Unofficial API for zhihu.

一个灵活、友好的爬虫框架

Laravel Crawler Detect ⭐ 262

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Hotel Review Analysis ⭐ 254

Sentiment analysis and aspect classification for hotel reviews using machine learning models with MonkeyLearn.

Lagoujob ⭐ 250

Job data mining repo for lagou.com

JavaScript + BeautifulSoup = JSSoup

Scrapy Jsonrpc ⭐ 238

Scrapy extension to control spiders using JSON-RPC

Go Movies ⭐ 232

golang spider Crawler 爬虫电影

Scrapy Deltafetch ⭐ 232

Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

Nudecrawler ⭐ 231

Crawl telegra.ph searching for nudes!

Infinitycrawler ⭐ 221

A simple but powerful web crawler library for .NET

Wayback Machine Scraper ⭐ 219

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Zhihuspider ⭐ 215

多线程知乎用户爬虫，基于python3

Fast Lianjia Crawler ⭐ 214

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

Finance_news_analysis ⭐ 206

金融新闻数据挖掘分析

Related Searches

Python Crawler (4,528)

Python Spider (2,155)

Javascript Crawler (1,142)

Spider Scrapy (982)

Scraper Crawler (896)

Java Crawler (593)

Crawler Scrapy (578)

Golang Crawler (509)

1-100 of 368 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2025 Awesome Open Source. All rights reserved.