Awesome Open Source

Programming Languages

Search results for spider

1,729 search results found

Colly ⭐ 21,902

Elegant Scraper and Crawler Framework for Golang

Awesome Spider ⭐ 20,170

Easyspider ⭐ 20,149

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化

Proxy_pool ⭐ 19,442

Python ProxyPool for web spider

Python Spider ⭐ 16,816

🌈Python3网络爬虫实战：淘宝、京东、网易云、B站、12306、抖音、笔趣阁、漫画小说下载、音

Pyspider ⭐ 15,943

A Powerful Spider(Web Crawler) System in Python.

Awesome Python Login Model ⭐ 15,679

😮python模拟登陆一些大型网站，还有一些简单的爬虫，希望对你们有所帮助❤️，如果喜欢记得给个s

Learn_python3_spider ⭐ 14,425

python爬虫教程系列、从0到1学习python爬虫，包括浏览器抓包，手机APP抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifu 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等

Examples Of Web Crawlers ⭐ 13,142

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Photon ⭐ 10,244

Incredibly fast crawler designed for OSINT.

Avbook ⭐ 8,777

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Spider Flow ⭐ 8,075

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Pholcus ⭐ 7,529

Pholcus is a distributed high-concurrency crawler software written in pure golang

Weibospider ⭐ 7,444

新浪微博爬虫，用python爬取新浪微博数据

Anti Anti Spider ⭐ 7,229

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库

Infospider ⭐ 6,856

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Awesome Web Scraping ⭐ 6,060

List of libraries, tools and APIs for web scraping and data processing.

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

Haipproxy ⭐ 5,384

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

Font Spider ⭐ 4,770

Smart webfont compression and format conversion tool

Toplist ⭐ 4,661

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:h

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Toapi ⭐ 3,417

Every web site provides APIs.

Novel Plus ⭐ 3,358

novel-plus 是一个多端（PC、WAP）阅读、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Weibospider ⭐ 3,294

持续维护的新浪微博采集工具🚀🚀🚀

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Toapi ⭐ 3,153

Every web site provides APIs.

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Scrapydweb ⭐ 2,839

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉

java项目实战练习

Scrapyd ⭐ 2,766

A service daemon to run Scrapy spiders

Wechatsync ⭐ 2,706

一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typec

Spiderkeeper ⭐ 2,685

admin ui for scrapy/open source scrapinghub

🔞 JAVClub - 让你的大姐姐不再走丢

Querylist ⭐ 2,598

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Python3 Spider ⭐ 2,582

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Scrapy Examples ⭐ 2,550

Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

Lianjia Beike Spider ⭐ 2,464

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

BitTorrent DHT Protocol && DHT Spider.

Decryptlogin ⭐ 2,375

DecryptLogin: APIs for loginning some websites by using requests.

Owllook ⭐ 2,340

owllook-小说搜索引擎

Torbot ⭐ 2,338

Dark Web OSINT Tool

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Web Scraping Framework

Wechat_articles_spider ⭐ 2,234

微信公众号文章的爬虫

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

Go Demo ⭐ 2,183

Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等

Web crawling framework based on asyncio.

Image Downloader ⭐ 2,029

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Web crawling framework based on asyncio.

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Bilibili Api ⭐ 1,871

哔哩哔哩的API调用模块

Crawler Detect ⭐ 1,842

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Video_spider ⭐ 1,805

短视频去水印：抖音,皮皮虾,火山,微视,微博,绿洲,最右,轻视频,快手,全民小视频,巴塞电影,陌陌, Vlog 小咖秀,皮皮搞笑,全民K歌,西瓜视频,逗拍,虎牙,6间房,梨视频,新片场,acfun,美拍...

Wechat Spider ⭐ 1,753

开源微信爬虫：爬取公众号所有文章、阅读量、点赞量和评论内容。易部署。持续维护！！！

Async Python 3.6+ web scraping micro-framework based on asyncio

Bilibili Api ⭐ 1,711

哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://gith

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Go_spider ⭐ 1,629

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Anemone ⭐ 1,615

Anemone web-spider framework

Open Source Search Engine ⭐ 1,504

Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.

Blackwidow ⭐ 1,448

A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.

Xsscrapy ⭐ 1,398

XSS spider - 66/66 wavsep XSS detected

Catvodtvspider ⭐ 1,365

Spider ⭐ 1,365

Unsurprising JavaScript - No longer active

Php Spider ⭐ 1,316

A configurable and extensible PHP web spider

Catvodtvspider ⭐ 1,270

Grab Site ⭐ 1,254

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Wechat_spider ⭐ 1,236

微信爬虫，获取文章内容、阅读量、点赞量、评论等，获取公众号所有历史文章链接。

Beanbun ⭐ 1,195

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Quotesbot ⭐ 1,178

This is a sample Scrapy project for educational purposes

Douyinliverecorder ⭐ 1,175

可循环值守和多人录制的直播录制软件，支持抖音、Tiktok、快手、虎牙、斗鱼、B站、小红书、网易cc

Checkinpanel ⭐ 1,175

一个主要运行在 𝐞𝐥𝐞𝐜𝐕𝟐𝐏 或 𝐪𝐢𝐧𝐠𝐥𝐨𝐧𝐠 等定时面板，同时支持系统运行环境的签到项目（环境：𝑷𝒚𝒕𝒉𝒐𝒏 3.8+ / 𝑵𝒐𝒅𝒆.𝒋𝒔 10+ / 𝑩𝒂𝒔𝒉 4+ / 𝑶𝒑𝒆𝒏𝑱𝑫𝑲8 / 𝑷𝒆𝒓𝒍5）

Pythoncrawler ⭐ 1,166

💗用python编写的爬虫项目集合

Qzoneexport ⭐ 1,163

QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Magic Api ⭐ 1,104

magic-api 是一个接口快速开发框架，通过Web页面编写脚本以及配置，自动映射为HTTP接口，无需定义Contro

Reptile ⭐ 1,081

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼腾讯视频豆瓣研招网微博笔趣阁小说百度热点 B站 CSDN 网易云阅读阿里文学百度股票今日头条微信公众号网易云音乐拉勾有道 unsplash 实习僧汽车之家英雄联盟盒子大众点评链家 LPL赛程台风梦幻西游、阴阳师藏宝阁天气牛客网百度文库睡前故事知乎 Wish

Django Dynamic Scraper ⭐ 1,069

Creating Scrapy scrapers via the Django admin interface

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

Antcolony ⭐ 1,042

Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )

Darknet_chinesetrading ⭐ 1,035

🚇暗网中文网监控爬虫(DEEPMIX)

V2rss ⭐ 1,030

🚀 采集|免费|优质|的-订?阅<；

Jspider ⭐ 1,006

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

磁力網站U3C3介紹以及域名更新

Oneblog ⭐ 952

👽 OneBlog，一个简洁美观、功能强大并且自适应的Java博客

Querido Diario ⭐ 944

📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.

Python website crawler.

A configurable web spider with a easy-to-use web console

Xsrfprobe ⭐ 897

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

Crawler ⭐ 897

A high performance web crawler / scraper in Elixir.

Vulstudy ⭐ 876

使用docker快速搭建各大漏洞靶场，目前可以一键搭建17个靶场。

Kimuraframework ⭐ 874

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Baiduspider ⭐ 872

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索

Related Searches

Python Spider (2,155)

Spider Scrapy (982)

Crawler Spider (709)

1-100 of 1,729 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.