Awesome Open Source
Awesome Open Source


Build Status Coverage Status Apache License 2.0

CoCrawler is a versatile web crawler built using modern tools and concurrency.

Crawling the web can be easy or hard, depending upon the details. Mature crawlers like Nutch and Heritrix work great in many situations, and fall short in others. Some of the most demanding crawl situations include open-ended crawling of the whole web.

The object of this project is to create a modular crawler with pluggable modules, capable of working well for a large variety of crawl tasks. The core of the crawler is written in Python 3.7+ using coroutines.


CoCrawler is pre-release, with major restructuring going on. It is currently able to crawl at around 170 megabits / 170 pages/sec on a 4 core machine.

Screenshot: Screenshot


We recommend that you use pyenv / virtualenv to separate the python executables and packages used by cocrawler from everything else.

You can install cocrawler from pypi using "pip install cocrawler".

For a more fresh version, clone the repo and install like this:

git clone
cd cocrawler
pip install . .[test]
make pytest
make test_coverage

The CI for this repo uses the latest versions of everything. To see exactly what worked last, click on the "Build Status" link above. Alternately, you can look at requirements.txt for a test combination that I probably ran before checking in.


CoCrawler draws on ideas from the Python 3.4 code in "500 Lines or Less", which can be found at It is also heavily influenced by the experiences that Greg acquired while working at blekko and the Internet Archive.


Apache 2.0

Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Python (892,779
Crawler (10,229
Screenshot (9,835
Web Crawler (4,276
Concurrency (3,375
Ranking (2,719
Aiohttp (962
Warc (254
Async Python (48
Aiohttp Client (26