Awesome Open Source
Awesome Open Source


Arachnid is a fast, soon to be multi-threading capable web crawler for Crystal. It recenty underwent a full rewrite for Crystal 0.35.1, so see the documentation below for updated usage instructions.


  1. Add the dependency to your shard.yml:

        github: watzon/arachnid
  2. Run shards install


First, of course, you need to require arachnid in your project:

require "arachnid"

The Agent

Agent is the class that does all the heavy lifting and will be the main one you interact with. To create a new Agent, use

agent =

The initialize method takes a bunch of optional parameters:


You can, if you wish, supply your own HTTP::Client instance to the Agent. This can be useful if you want to use a proxy, provided the proxy client extends HTTP::Client.


The user agent to be added to every request header. You can override this on a per-host basis with either :host_headers or :default_headers.


The default headers to be used in every request.


Headers to be applied on a per-host basis. This is a hash String (host name) => HTTP::Headers.


The Arachnid::Queue instance to use for storing links waiting to be processed. The default is a MemoryQueue (which is the only one for now), but you can easily implement your own Queue using whatever you want as a backend.


Whether or not to stop running when the queue is empty. This is true by default. If it's made false, the loop will continue when the queue empties, so be sure you have a way to keep adding items to the queue.


Whether or not to follow redirects (add them to the queue).

Starting the Agent

There are four ways to start your Agent once it's been created. Here are some examples:


#start_at starts the Agent running on a particular URL. It adds a single URL to the queue and starts there.

agent.start_at("") do
  # ...


#site starts the agent running at the given URL and adds a rule that keeps the agent restricted to the given site. This allows the agent to scan the given domain and any subdomains. For instance:"") do
  # ...

The above will match and, but not or any other site not within the * space.


#host is like site, but with the added restriction of just remaining on the current domain path. Subdomains are not included."") do
  # ...


Provided you already have URIs in the queue ready to be scanned, you can also just use #start to start the Agent running.



URI's can be filtered before being enqueued. There are two kinds of filters, accept and reject. Accept filters can be used to ensure that a URI matches before being enqueued. Reject filters do the opposite, keeping URIs from being enqueued if they do match.

For instance:

# This will filter out all sites where the host is not ""
agent.accept_filter { |uri| == "" }

If you want to ignore certain parts of the above filter:

# This will ignore paths starting with "/api"
agent.reject_filter { |uri| uri.path.to_s.starts_with?("/api") }

The #site and #host methods add a default accept filter in order to keep things in the given site or host.


All the above is useless if you can't do anything with the scanned resources, which is why we have the Resource class. Every scanned resource is converted into a Resource (or subclass) based on the content type. For instance, text/html becomes a Resource::HTML which is parsed using kostya/myhtml for extra speed.

Each resource has an associated Agent#on_ method so you can do something when one of those resources is scanned:

agent.on_html do |page|
  puts typeof(page)
  # => Arachnid::Resource::HTML

  puts page.title
  # => The Title of the Page

Currently we have:

  • #on_html
  • #on_image
  • #on_script
  • #on_stylesheet
  • #on_xml

There is also #on_resource which is called for every resource, including ones that don't match the above types. Resources all include, at minimum the URI at which the resource was found, and the response (HTTP::Client::Response) instance.


  1. Fork it (
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Bot (12,352
Crystal (3,375
Crawler (2,872
Webscraping (2,216
Spider (1,564
Webscraper (553
Crawling (358
Crystal Lang (244
Related Projects