Robots

Support for Robots Exclusion Protocol, including parsing and matching against robot.txt directives.
Alternatives To Robots
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Webrobots244,14299 months ago16January 04, 20162bsd-2-clauseRuby
A Ruby library to help write robots.txt compliant web robots
Waterplus_map_tools12
a year agogpl-2.0C++
Map tools for the Waterplus Robots
Robots613 years ago6February 03, 20218bsd-3-clauseJava
Support for Robots Exclusion Protocol, including parsing and matching against robot.txt directives.
Robot_rules6
15 years ago1August 11, 2014mitRuby
A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
Alternatives To Robots
Select To Compare


Alternative Project Comparisons
Popular Robot Projects
Popular Uri Projects
Popular Artificial Intelligence Categories

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Java
Robot
Uri