Awesome Open Source

Programming Languages

Search results for spider web crawler

49 search results found

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Spider Flow ⭐ 8,075

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Awesome Web Scraping ⭐ 6,060

List of libraries, tools and APIs for web scraping and data processing.

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Web Scraping Framework

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Django Dynamic Scraper ⭐ 1,069

Creating Scrapy scrapers via the Django admin interface

A configurable web spider with a easy-to-use web console

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Zhihu Spider ⭐ 719

A web spider for zhihu.com

Spidersuite ⭐ 447

Advance web spider/crawler for cyber security professionals

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A web crawler for Go

Lagoujob ⭐ 250

Job data mining repo for lagou.com

Dark Fantasy Hack Tool ⭐ 248

DDOS Tool: To take down small websites with HTTP FLOOD. Port scanner: To know the open ports of a site. FTP Password Cracker: To hack file system of websites.. Banner Grabber: To get the service or software running on a port. (After knowing the software running google for its vulnerabilities.) Web Spider: For gathering web application hacking information. Email scraper: To get all emails related to a webpage IMDB Rating: Easy way to access the movie database. Both .exe(compressed as zip) and .py

Nudecrawler ⭐ 231

Crawl telegra.ph searching for nudes!

Infinitycrawler ⭐ 221

A simple but powerful web crawler library for .NET

Wayback Machine Scraper ⭐ 219

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Awesome Web Scraper ⭐ 214

A collection of awesome web scaper, crawler.

Portia Dashboard ⭐ 190

portia-dashboard is a visual web crawler based on scrapinghub/portia

Ignareo Isml Auto Voter ⭐ 186

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

Crawlab Lite ⭐ 184

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Digger is a powerful and flexible web crawler implemented by pure golang

Zhihu Crawler People ⭐ 179

A simple distributed crawler for zhihu && data analysis

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

A threaded web-spider written in Clojure

Direct_web_spider ⭐ 143

A direct web spider framworks for Ruby

Scrapy Training ⭐ 141

Scrapy Training companion code

Not Your Average Web Crawler ⭐ 130

A web crawler (for bug hunting) that gathers more than you can imagine.

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

Openscraper ⭐ 80

An open source webapp for scraping: towards a public service for webscraping

Arachnid ⭐ 80

Powerful web scraping framework for Crystal

Yet Another Web Spider

Scrapy Wayback Machine ⭐ 70

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Robotstxt ⭐ 68

robots.txt file parsing and checking for R

Schweizermesser ⭐ 66

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

Talospider ⭐ 55

talospider - A simple,lightweight scraping micro-framework

Pythonscrapybasicsetup ⭐ 54

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

Hk0weather ⭐ 49

Web scraper project to collect the useful Hong Kong weather data from HKO website

Ronin Web ⭐ 40

ronin-web is a collection of useful web helper methods and commands.

Rust Web Crawler saving pages on Redis

House Renting Spider ⭐ 39

A crawler for accommodation rental information in Douban Group 豆瓣小组上海租房爬虫

Scrapemate ⭐ 39

Golang Crawling and scraping framework

A fast and powerful web scraping library

Netcloud ⭐ 37

NetCloud Web Spider

a web crawler and data analysis repo with Python3.5, R, Excel 2016 and TAGUL

Flink Crawler ⭐ 35

Continuous scalable web crawler built on top of Flink and crawler-commons

Goscrapy ⭐ 34

GoScrapy: Harnessing Go's power for efficient web scraping, inspired by Python's Scrapy framework.

A simple web-crawler development framework based on .Net Core.

A web spider for shodan.io without using the Developer API.

Scrapy Bench ⭐ 25

A CLI for benchmarking Scrapy.

a complex but scalable web spider

Scrapebox ⭐ 23

A simple, system independent infrastructure for performing web scraping. Utilizes Vagrant virtualbox interface and puppet provisioning to create and execute scraping of web content to structured data quickly and easily without modifying your core system.

The Web Spider Obstacle Course

Assessor Scraper ⭐ 22

A project to scrape the assessor's website and make the data accessible for advanced queries

Docker Crawler ⭐ 19

Gpt2_episode_summary_generator ⭐ 18

Utilizing webscraping and state-of-the-art NLP to generate TV show episode summaries.

Weibospider ⭐ 18

A Web Spider for Weibo(Chinese Twitter)

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

Iwata Asks Downloader ⭐ 17

Tool to download Iwata Asks interviews (none of which are stored in this repo)

a web scraper which can find movies ,series ,software ,etc.. direct links and download them

Spiderman ⭐ 16

your friendly neighborhood web crawler

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

Selenium_python ⭐ 15

Mailinglistscraper ⭐ 15

A python web scraper for public email lists.

Web Crawler ⭐ 14

Web crawler based on Puppeteer

Boris The (Web) Spider

Robots.txt ⭐ 13

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Spiderwebai ⭐ 12

The fastest way to scrape data to feed your AI models and LLMs

1024spider ⭐ 12

Only for researching, please pay attention to your health.

Crawlerflow ⭐ 12

Web Crawlers orchestration Framework that lets you create datasets from multiple web sources.

Floodesh ⭐ 11

Floodesh is a distributed web spider written with Nodejs.

Crawlerr ⭐ 11

A simple and fully customizable web crawler/spider for Node.js with server-side DOM. Comes with elegant and hell-simple APIs.

Lilhomie ⭐ 11

A Machine Learning Project implemented from scratch which involves web scraping, data engineering, exploratory data analysis and machine learning to predict housing prices in New York Tri-State Area.

Prodirectscraper ⭐ 10

👔 Web scraper for http://www.prodirectselect.com/ 👞

Octopus_spider ⭐ 10

基于Scala Akka的分布式主题网络爬虫

Web Scraping ⭐ 10

Tutorial for web scraping in Python

Php_web_spider ⭐ 10

A web crawler written in PHP php网络蜘蛛，信息收集工具A web spider, using php, based on cURL & simple html dom.

Scrapyteer ⭐ 9

Web crawling & scraping framework for Node.js on top of headless Chrome browser

Web Spider ⭐ 9

Multi threaded Web crawler

Spiderfetch ⭐ 9

A modular web spider

A web crawler that crawls the latest WeChat article

Simple web spider for smoke tests, link checking, etc

Parker is a Python-based web spider for collecting specific data across a set of configured sites.

Android Market Scraper ⭐ 8

Web spider for the Android App Market

Kpop_crawler ⭐ 8

A web crawler that fetches K-pop song details and lyrics from top charts

Scraper Crawler ⭐ 8

A collection of some of best web Web scraper,crawler,spider in different languages

Web Spider ⭐ 8

这是一个用superagent + phantomjs 写的一个小爬虫，尽量简单。

Alicrawler ⭐ 7

a fully functional spider for aliexpress.com

Anemone_lite ⭐ 7

Distributed web crawler using mongodb

Bbc Football Stats ⭐ 7

Web scraper that collects and returns football stats from BBC's sports site

Xiaohongshu Spider Visualizer ⭐ 7

A distributed web crawler for xiaohongshu.com and visualization for the crawled content.

Related Searches

Python Spider (2,155)

Scraper Web Crawler (1,388)

Crawler Spider (1,073)

Spider Scrapy (982)

1-49 of 49 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.