Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Ruby Crawler Open Source Projects

Open source projects categorized as Ruby Crawler
Arachni ⭐ 3,632
Web Application Security Scanner Framework
most recent commit a year ago
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 34total releases 23most recent commit 4 years agorubygems anemone} Downloads
Wombat ⭐ 1,297
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
dependent packages 5total releases 33most recent commit 3 months agorubygems wombat} Downloads
Kimuraframework ⭐ 874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
dependent packages 2total releases 10most recent commit 2 years agorubygems kimurai} Downloads
Spidr ⭐ 775
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dependent packages 16total releases 26most recent commit 3 months agorubygems spidr} Downloads
Device_detector ⭐ 711
DeviceDetector is a precise and fast user agent parser and device detector written in Ruby
dependent packages 5total releases 26most recent commit 5 months agorubygems device_detector} Downloads
Vessel ⭐ 586
Fast high-level web crawling Ruby framework
total releases 3most recent commit 4 months ago
Bounty Targets ⭐ 522
This project crawls bug bounty platform scopes (like Hackerone/Bugcrowd/Intigriti/etc) hourly and dumps them into the bounty-targets-data repo
most recent commit 4 months ago
Tarantula ⭐ 453
a big hairy fuzzy spider that crawls your site, wreaking havoc
dependent packages 1total releases 13most recent commit 11 years agorubygems tarantula} Downloads
Playdrone ⭐ 405
Google Play Crawler
most recent commit 9 years ago
Prerender_rails ⭐ 358
Rails middleware gem for prerendering javascript-rendered pages on the fly for SEO
dependent packages 1total releases 44most recent commit 5 months agorubygems prerender_rails} Downloads
Archivebot ⭐ 328
ArchiveBot, an IRC bot for archiving websites
most recent commit 5 months ago
Woothee ⭐ 304
User-Agent parser/classifier for multi languages
dependent packages 2total releases 14most recent commit 2 years agopackagist woothee/woothee-testset} Downloads
Harvestman ⭐ 278
Quick and dirty web crawling.
most recent commit 8 years ago
Sinew ⭐ 254
A Ruby DSL for structured web crawling, with a robust caching system.
dependent packages 1total releases 14most recent commit 8 months agorubygems sinew} Downloads
Rawler ⭐ 253
Rawler is a tool that crawls the links of your website
dependent packages 2total releases 17most recent commit 9 years agorubygems rawler} Downloads
Spidey ⭐ 179
A loose framework for crawling and scraping web sites.
dependent packages 3total releases 4most recent commit 8 years agorubygems spidey} Downloads
Voight Kampff ⭐ 171
Voight-Kampff is a Ruby gem that detects bots, spiders, crawlers and replicants
dependent packages 3total releases 17most recent commit a year agorubygems voight_kampff} Downloads
Instagram Crawler ⭐ 157
Crawl instagram photos, posts and videos for download.
total releases 5most recent commit 5 years ago
Rubyretriever ⭐ 139
Asynchronous Web Crawler & Scraper
dependent packages 1total releases 29most recent commit a year agorubygems rubyretriever} Downloads
Saushengine.v1 ⭐ 118
Simple Ruby-based search engine
most recent commit 10 years ago
Crawler_detect ⭐ 106
Ruby gem to detect bots and crawlers via the user agent
total releases 23most recent commit 4 months ago
Human_power ⭐ 97
Easy generation of robots.txt. Force the robots into submission!
total releases 6most recent commit 9 years ago
The 10 Most Depended On Ruby Crawler Open Source Projects
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 34total releases 23latest release May 30, 2012most recent commit 4 years agorubygems anemone} Downloads
Woothee Ruby ⭐ 70
Woothee ruby implementation
dependent packages 6total releases 27latest release May 19, 2020most recent commit 2 years agorubygems woothee} Downloads
Wombat ⭐ 1,297
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
dependent packages 5total releases 33latest release September 03, 2019most recent commit 3 months agorubygems wombat} Downloads
Device_detector ⭐ 711
DeviceDetector is a precise and fast user agent parser and device detector written in Ruby
dependent packages 5total releases 26latest release December 01, 2023most recent commit 5 months agorubygems device_detector} Downloads
Polipus ⭐ 95
Polipus: distributed and scalable web-crawler framework
dependent packages 3total releases 20latest release July 17, 2015most recent commit 9 years agorubygems polipus} Downloads
Ronin Web ⭐ 40
ronin-web is a collection of useful web helper methods and commands.
dependent packages 3total releases 16latest release April 04, 2023most recent commit 4 months agorubygems ronin-web} Downloads
Spidey ⭐ 179
A loose framework for crawling and scraping web sites.
dependent packages 3total releases 4latest release May 03, 2013most recent commit 8 years agorubygems spidey} Downloads
Voight Kampff ⭐ 171
Voight-Kampff is a Ruby gem that detects bots, spiders, crawlers and replicants
dependent packages 3total releases 17latest release September 30, 2018most recent commit a year agorubygems voight_kampff} Downloads
Scruber ⭐ 6
Ruby crawling framework
dependent packages 2total releases 9latest release September 06, 2018most recent commit 6 years agorubygems scruber} Downloads
The 10 Latest Releases In Ruby Crawler Open Source Projects
Device_detector ⭐ 711
DeviceDetector is a precise and fast user agent parser and device detector written in Ruby
dependent packages 5total releases 26latest release December 01, 2023most recent commit 5 months agorubygems device_detector} Downloads
Legitbot ⭐ 19
🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?
total releases 44latest release November 17, 2023most recent commit 5 months ago
Snapcrawl ⭐ 51
Crawl a website and take screenshots
total releases 27latest release July 27, 2023most recent commit 9 months ago
Proxycrawl Ruby ⭐ 14
ProxyCrawl API ruby gem for scraping and crawling
total releases 9latest release July 03, 2023most recent commit 10 months ago
Ronin Web ⭐ 40
ronin-web is a collection of useful web helper methods and commands.
dependent packages 3total releases 16latest release April 04, 2023most recent commit 4 months agorubygems ronin-web} Downloads
Tansaku ⭐ 10
Yet another dirbuster tool
total releases 10latest release March 14, 2022most recent commit 6 months ago
Wgit ⭐ 10
Wgit allows you to crawl and extract the data you want from the web
dependent packages 1total releases 34latest release November 26, 2021most recent commit 3 months agorubygems wgit} Downloads
Prerender_rails ⭐ 358
Rails middleware gem for prerendering javascript-rendered pages on the fly for SEO
dependent packages 1total releases 44latest release November 04, 2021most recent commit 5 months agorubygems prerender_rails} Downloads
Woothee ⭐ 304
User-Agent parser/classifier for multi languages
dependent packages 2total releases 14latest release October 31, 2021most recent commit 2 years agopackagist woothee/woothee-testset} Downloads
Crawler_detect ⭐ 106
Ruby gem to detect bots and crawlers via the user agent
total releases 23latest release September 05, 2021most recent commit 4 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.