Awesome Open Source

Programming Languages

Search results for docker crawler

82 search results found

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Proxypool ⭐ 5,154

An Efficient ProxyPool with Getter, Tester and Server

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Packtpub Crawler ⭐ 701

Download your daily free Packt Publishing eBook https://www.packtpub.com/packt/offers/free-learnin

Newcrawler ⭐ 583

Free Web Scraping Tool with Java

Crawljax ⭐ 493

Browsertrix Crawler ⭐ 470

Run a high-fidelity browser-based crawler in a single Docker container

Scrapybook ⭐ 378

Scrapy Book Code

The simple, easy to use command line web crawler.

Go Movies ⭐ 232

golang spider Crawler 爬虫电影

Goose Parser ⭐ 222

Universal scraping tool, which allows you to extract data using multiple environments

Zhihuspider ⭐ 215

多线程知乎用户爬虫，基于python3

Make a ZIM file from any Web site and surf offline!

Portia Dashboard ⭐ 190

portia-dashboard is a visual web crawler based on scrapinghub/portia

Black Widow ⭐ 168

GUI based offensive penetration testing tool (Open Source)

Github_commit_crawler ⭐ 167

Tool used to continuously monitor a Github org for mistaken public commits

estela, an elastic web scraping cluster 🕸

Crawler ⭐ 138

Go process used to crawl websites

Acm Statistics ⭐ 137

An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge

Javbus Api ⭐ 136

一个自我托管的 JavBus API 服务

POOPAK - TOR Hidden Service Crawler

Councilor Voter Guide ⭐ 104

縣市長 / 議員投票指南

Automated fuzzing framework

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题；HTTP协议介绍； Requests使用；解析器Xpath介绍； MongoDB与MySQL；多线程爬虫； Scrapy介绍；Scrapy-redis介绍；使用docker部署；使用nomad管理docker集群；使用EFK查询docker日志

Open source SEO auditing tool.

Lcbo Api ⭐ 85

A crawler and API server for Liquor Control Board of Ontario retail data

Scrapper ⭐ 83

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.

Seleniumdemo ⭐ 80

Selenium automation test framework

A DHT crawler and torrent indexer

Movie Elasticsearch ⭐ 76

使用 SpringBoot2.0+ElasticSearch 实现的开源电影搜索引擎

Docker Diskover ⭐ 66

A Docker container for the Diskover space mapping application

Dig Etl Engine ⭐ 65

Download DIG to run on your laptop or server.

Gocrawler ⭐ 60

A distributed web crawler implemented using Go, Postgres, RabbitMQ and Docker

Fishfishjump ⭐ 57

Fish Fish Jump is a solution in the python that simply and basic for search engines. 🐟 🐟 🐟

Semantic Bus ⭐ 51

object flow treatment, data transformation

Snapcrawl ⭐ 51

Crawl a website and take screenshots

Docker Django Celery Tutorial ⭐ 45

docker-django-celery-tutorial 基本教學 📝

Browser As A Service ⭐ 43

A web browser 🌎 hosted as a service, to render your JavaScript web pages as HTML

Go Crawler Distributed ⭐ 39

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息 gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。

Docker Scrapy Crawler ⭐ 38

docker scrapyd scrapy boot2docker crawler - a spider Python application that can be "Dockerized".

Tw Stock Telegram Bot ⭐ 33

台股機器人，提供即時個股及大盤報價、走勢、新聞、盤後資料等 Telegram bot to query real-time TW stock quotes, charts, news, and other related information

Aio Vextractor ⭐ 33

解析视频网站/APP/H5 页面视频信息。支持抖音、腾讯视频、YouTube、Instagram 等40余个网站与APP

Openartbrowser ⭐ 33

Exploring the world of arts using open data

Pokemongo Map Poc ⭐ 27

🎃 POC project for Pokemon Go map

Cryptocurrency Trading ⭐ 27

How to make profits in cryptocurrency trading with machine learning

Teach daily is web crawl by GoLang from web dev.to, freecodecamp.com, medium.com, hashnode.com, logrocket.com,infoq.com

Crawler Project ⭐ 25

Google资深工程师深度讲解Go语言爬虫项目。

Blog_crawler ⭐ 24

Blog crawler using Scrapy and PostgreSQL.

Toripchanger ⭐ 24

Python powered way to get a unique Tor IP

Domain_discovery_tool_deprecated ⭐ 23

Seed acquisition tool to bootstrap focused crawlers

Screamingfrog Docker ⭐ 19

Docker image for ScreamingFrog version 16

A distributed image crawler

Twds Crawler ⭐ 18

Highly scalable webcrawler for towardsdatascience.com by using Python, Selenium, Docker, Kubernetes and the infrastructure of the Google Cloud Platform

Searchgar ⭐ 18

SearchGar - An actual Search Engine made using Python

Kathisto ⭐ 17

📦 Server-side rendering for Javascript based web-apps

一个分布式的爬虫项目

Newspaper Crawler ⭐ 15

Scrapy based crawler which crawls newspaper.

Coronavis ⭐ 15

This is the repository for coronavis.dbvis.de

Mongo Elasticsearch Nutch ⭐ 15

Docker image for creating a single Apache Nutch server, with mongodb as crawl storage and Elasticsearch for indexing

Goose Starter Kit ⭐ 14

This is a starter kit for redco/goose-parser

Fastcampus ⭐ 14

수업 내용 주제별 정리

Robots.txt ⭐ 13

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Intellead Getting Started ⭐ 12

Getting started with intellead.

Containerized Ferret worker

Crawler News ⭐ 11

Use python scrapy build crawler for real-time Taiwan NEWS website.

Microdocs ⭐ 11

Documentation that scales with your Microservices

Eis Warc Archiver ⭐ 10

ARCHIVED--Docker app to crawl URLs and generate WARCs

News Crawler ⭐ 10

Crawler that collects and extracts content of daily published news articles

Aws Fargate Demo ⭐ 10

AWS fargate demo for AWSKRUG-recap

Learning_scrapy ⭐ 10

精通python爬虫框架scrapy源码

Crawler Benchmark ⭐ 10

A Reference Framework for the Automated Exploration of Web Applications. Provides some general web features to let you test crawlers in a well defined environment.

Mb Checker ⭐ 10

Python script that traverses chrome Bookmark file and remove stale entries. Includes Jenkinsfile to generate docker images.

Ncov Channel Crawler ⭐ 9

Distributed web scraping framework

Dcss_tourney ⭐ 9

Dungeon Crawl Stone Soup tournament scripts

Product Categorization ⭐ 8

Product Categorization with Machine Learning

Docker Nutch Elasticsearch Mongodb ⭐ 8

Docker Image for Apache Nutch, Elasticsearch and MongoDB

Distributed Webcrawler ⭐ 8

Distributed WebCrawler built on top of Docker

A web crawler that, visits HTML pages within the same domain for a given url.

Docker Codesearch ⭐ 8

Code Search on Fess

Elastic Webcrawler ⭐ 8

Golang Webcrawler for Elasticsearch

Sql Crawler ⭐ 8

Connect to multiple SQL database servers and run queries to collect data.

Simplified Search Engine ⭐ 7

Multithreaded Web Crawler, Scraper, Indexer

Datasurvey ⭐ 7

Crawl a directory of files and generate a summary of what is available.

Xiaohongshu Spider Visualizer ⭐ 7

A distributed web crawler for xiaohongshu.com and visualization for the crawled content.

Nucypher Monitor ⭐ 7

NuCypher network intelligence crawler and web dashboard

91porn Docker ⭐ 6

download video on 91porn with aria2

Lolcrawler ⭐ 6

Headless web crawler for bugbounty and penetration-testing/redteaming

Linkchecker ⭐ 6

Recursively crawls a website and checks that URLs return 200.

Zapimoveis_scraper ⭐ 6

ZAP Imóveis crawler and scraper using Scrapy and Splash

Spotify News ⭐ 6

A Flask application to retrieve the singers' latest news according to your Spotify current playing song.

Japanese News Crawler ⭐ 6

A complete automated japanese news crawler built on the top of Scrapy framework

Imdb_crawler ⭐ 6

Crawler do site https://www.imdb.com/

Scheduler ⭐ 6

Go orchestrator process used to schedule website parsing

Sport News Retrieval ⭐ 6

Just a typical search engine in this universe 🔥🔥🔥

Paste2splunk ⭐ 5

Pastebin crawler which index pasties into Splunk

Rentea Crawler ⭐ 5

A crawler that provides timely response to data change on public rental house platform

Sparkler Crawl Environment - a packaged, dockerized version of http://github.com/USCDataScience/sparkler.git

Beatport Top Preview ⭐ 5

A Node.js web app to retrieve and create YouTube playlists of Beatport top 100s.

Related Searches

Shell Docker (20,660)

Docker Dockerfile (16,395)

Python Docker (16,341)

Javascript Docker (10,426)

Golang Docker (7,702)

Php Docker (6,192)

Java Docker (6,071)

Docker Nginx (5,238)

Typescript Docker (4,630)

Python Crawler (4,545)

1-82 of 82 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.