Project Name	Stars	Packages Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Katana	7,995	1	3 months ago	8	September 14, 2023	67	mit	Go
A next-generation crawling and spidering framework.
Ani Cli	5,724		3 months ago			15	gpl-3.0	Shell
A cli tool to browse and play anime
Spider	907		6 years ago			3	gpl-3.0	Java
A configurable web spider with a easy-to-use web console
Fetchbot	758	3	3 years ago	7	May 20, 2021	2	bsd-3-clause	Go
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Xidel	611		5 months ago			18	gpl-3.0	Pascal
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Mov Cli	314	1	3 months ago	103	November 25, 2023	32	gpl-3.0	Python
A cli tool to browse and watch Movies/Shows/TV/Sports.
Crawley	208		4 months ago	60	December 07, 2023		mit	Go
The unix-way web crawler
Gotor	150		5 months ago	3	November 10, 2022	3	gpl-3.0	Go
This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
Anirip	144		2 years ago			21	mit	Go
:clapper: A Crunchyroll show/season ripper
Cascadia	128		10 months ago	9	June 30, 2023	1	mit	Go
Go cascadia package command line CSS selector

Alternatives To Spider

Select To Compare

Katana ⭐ 7,995

A next-generation crawling and spidering framework.

dependent packages 1total releases 8most recent commit 3 months ago

Ani Cli ⭐ 5,724

A cli tool to browse and play anime

most recent commit 3 months ago

Spider ⭐ 907

A configurable web spider with a easy-to-use web console

most recent commit 6 years ago

Fetchbot ⭐ 758

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

dependent packages 3total releases 7most recent commit 3 years ago

Xidel ⭐ 611

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

most recent commit 5 months ago

Mov Cli ⭐ 314

A cli tool to browse and watch Movies/Shows/TV/Sports.

dependent packages 1total releases 103most recent commit 3 months ago

Crawley ⭐ 208

The unix-way web crawler

total releases 60most recent commit 4 months ago

Gotor ⭐ 150

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

total releases 3most recent commit 5 months ago

Anirip ⭐ 144

:clapper: A Crunchyroll show/season ripper

most recent commit 2 years ago

Cascadia ⭐ 128

Go cascadia package command line CSS selector

total releases 9most recent commit 10 months ago

Suggest An Alternative To spider

Alternative Project Comparisons

Popular Web Crawler Projects

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months ago

Huginn ⭐ 40,328

Create agents that monitor and act on your behalf. Your agents are standing by!

dependent packages 52total releases 8latest release September 22, 2017most recent commit 3 months ago

Changedetection.io ⭐ 13,943

The best and simplest free open source website change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

most recent commit 3 months ago

Crawlee ⭐ 11,957

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 4 days ago

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1latest release March 03, 2019most recent commit 3 months ago

Popular Command Line Projects

Ohmyzsh ⭐ 168,213

🙃 A delightful community-driven (with 2,300+ contributors) framework for managing your zsh configuration. Includes 300+ optional plugins (rails, git, macOS, hub, docker, homebrew, node, php, python, etc), 140+ themes to spice up your morning, and an auto-update tool so that makes it easy to keep up with the latest updates from the community.

dependent packages 5total releases 1latest release December 07, 2014most recent commit 10 days ago

The Art Of Command Line ⭐ 142,328

Master the command line, in one page

most recent commit 4 months ago

Terminal ⭐ 93,043

The new Windows Terminal and the original Windows console host, all in the same place!

total releases 1latest release November 20, 2022most recent commit a month ago

Thefuck ⭐ 81,037

Magnificent app which corrects your previous console command.

dependent packages 9total releases 101latest release January 02, 2022most recent commit 3 months ago

Fzf ⭐ 57,036

:cherry_blossom: A command-line fuzzy finder

dependent packages 11total releases 63latest release November 30, 2023most recent commit 3 months ago

Popular Data Processing Categories