Awesome Open Source

Programming Languages

Search results for golang crawler

94 search results found

👾 Fast and simple video download library and CLI tool written in Go

Colly ⭐ 21,902

Elegant Scraper and Crawler Framework for Golang

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Katana ⭐ 7,995

A next-generation crawling and spidering framework.

Ferret ⭐ 5,540

Declarative web scraping

A Devtools driver for web automation and scraping

Hakrawler ⭐ 4,120

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.

Proxypool ⭐ 3,173

自动抓取tg频道、订阅地址、公开互联网上的ss、ssr、vmess、trojan节点信息，聚合去重测

Crawlergo ⭐ 2,642

A powerful browser crawler for web vulnerability scanners

BitTorrent DHT Protocol && DHT Spider.

Torbot ⭐ 2,338

Dark Web OSINT Tool

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

Gryffin ⭐ 2,061

Gryffin is a large scale web security scanning platform.

Rendora ⭐ 1,950

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

Gocrawl ⭐ 1,929

Polite, slim and concurrent web crawler.

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Go_spider ⭐ 1,629

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Cariddi ⭐ 1,228

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Dungeonfs ⭐ 966

A FUSE filesystem and dungeon crawling adventure game engine

Goclone ⭐ 907

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Ipfs Search ⭐ 779

Search engine for the Interplanetary Filesystem.

Seccrawler ⭐ 777

一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客，持续更新

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

Creeper ⭐ 762

🐾 Creeper - The Next Generation Crawler Framework (Go)

Fetchbot ⭐ 758

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Go Dork ⭐ 677

The fastest dork scanner written in Go.

Staticgen ⭐ 668

Static website generator that lets you use HTTP servers and frameworks you already know

Fictiondown ⭐ 601

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

OSINT Swiss Army Knife

Go_jobs ⭐ 536

带你了解一下Golang的市场行情

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

Simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.

Dataflowkit ⭐ 394

Extract structured data from web sites. Web sites scraping.

Ants Go ⭐ 368

open source, distributed, restful crawler engine in golang

Gospider ⭐ 354

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Second Order ⭐ 295

Second-order subdomain takeover scanner

Webpalm ⭐ 295

WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Project for Go Search, a search engine for finding popular and relevant packages.

A web crawler for Go

Certgraph ⭐ 264

An open source intelligence tool to crawl the graph of certificate Alternate Names

Go Movies ⭐ 232

golang spider Crawler 爬虫电影

Gorecon ⭐ 227

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

Crawley ⭐ 208

The unix-way web crawler

Awesome_crawl ⭐ 206

腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

golang light-weight image crawler

Pryingdeep ⭐ 196

Prying Deep - An OSINT tool to collect intelligence on the dark web.

Leacrawler ⭐ 193

Web template crawler written by golang. golang开发的网站模板爬虫.

Ptt Alertor ⭐ 193

📢 Ptt 文章通知機器人！Notify Ptt Article in Realtime

A fast tool to fetch URLs from HTML attributes by crawl-in.

Chainwalker ⭐ 188

Rapid Smart Contract Crawler

A utility for crawling an AWS account and exporting all its resources for further analysis.

Digger is a powerful and flexible web crawler implemented by pure golang

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

🌌 A network agnostic DHT crawler, monitor, and measurement tool that exposes timely information about DHT networks.

Golang Proxy ⭐ 167

Golang version of proxy mining tool with evaluation system

Goribot ⭐ 162

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Eth Indexer ⭐ 139

An Ethereum project to crawl blockchain states into database

Crawler ⭐ 138

Go process used to crawl websites

Domain names collector - Crawl websites and collect domain names along with their availability status.

rotating open proxy multiplexer

Yispider ⭐ 127

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用

Lao Siji ⭐ 126

注意車速！！

Massivedl ⭐ 125

Download a large list of files concurrently

Interactive CLI Web Crawler

Scraply ⭐ 114

Scraply a simple dom scraper to fetch information from any html based website

Linkcrawler ⭐ 111

Cross-platform persistent and distributed web crawler 🔗

Graphquery ⭐ 104

GraphQuery is a query language and execution engine tied to any backend service.

Node Crawler ⭐ 98

Attempts to crawl the Ethereum network of valid Ethereum execution nodes and visualizes them in a nice web dashboard.

Gopa Abandoned ⭐ 97

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

Blinkist M4a Downloader ⭐ 97

Grabs all of the audio files from all of the Blinkist books

Open source SEO auditing tool.

Douban Movie ⭐ 91

Golang爬虫爬取豆瓣电影Top250

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Dataset of Adult Image Metadata for Training Spam Detection Models

Bathyscaphe ⭐ 83

Fast, highly configurable, cloud native dark web crawler.

Xcrawl3r ⭐ 79

A command-line interface (CLI) based utility to recursively crawl webpages. It is designed to systematically browse webpages' URLs and follow links to discover linked webpages' URLs.

Dorkscout ⭐ 78

DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

Gospider ⭐ 70

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

Ipproxypool ⭐ 66

Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra

Useragent ⭐ 65

User agent string parser in golang

Collyzar ⭐ 65

Distributed redis-based web crawler framework for colly

Ipfs Crawler ⭐ 62

A crawler for the IPFS network, code for our paper (https://arxiv.org/abs/2002.07747). Also holds scripts to evaluate the obtained data and make similar plots as in the paper.

Cross-platform persistent and distributed web crawler 🦀

Gocrawler ⭐ 60

A distributed web crawler implemented using Go, Postgres, RabbitMQ and Docker

Goscraper ⭐ 58

Golang pkg to quickly return a preview of a webpage (title/description/images)

Universal sitemap generator

Godataaccess ⭐ 54

🪲Data access framework in native Golang(Golang实现的类Scrapy框架)

Owlcrawler ⭐ 54

Crawl the web using nats.io and Go

Tool Gin ⭐ 54

基于go-gin框架建立减少冗余动作项目，如：下载一些工具

Fast website scraper and wordlist generator

Car Prices ⭐ 50

Golang爬虫爬取汽车之家二手车产品库

A distributed self-host DHT torrent search suite

Gathertool ⭐ 46

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口

Html Query ⭐ 45

A fluent and functional approach to querying HTML

Proxifier ⭐ 43

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Crawlerdetect ⭐ 43

Golang module to detect bots and crawlers via the user agent

🌱 goclone - clone websites in a matter of seconds

Go Crawler Distributed ⭐ 39

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息 gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。

Related Searches

Golang Command Line (8,308)

Golang Docker (7,769)

Python Crawler (4,545)

Golang Http (4,290)

Golang Server (4,285)

Javascript Golang (3,372)

Golang Database (2,927)

Golang Json (2,652)

Golang Proxy (2,577)

Golang Grpc (2,432)

1-94 of 94 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.