Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
Alternatives To Crawlee
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Scrapy47,4174,1853732 days ago87March 01, 2022741bsd-3-clausePython
Scrapy, a fast high-level web crawling & scraping framework for Python.
Lux21,17252 days ago31May 05, 2022445mitGo
👾 Fast and simple video download library and CLI tool written in Go
Colly19,5828120120 days ago22March 08, 2022168apache-2.0Go
Elegant Scraper and Crawler Framework for Golang
Newspaper12,67822248a month ago18September 28, 2018495mitPython
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Webmagic10,770734152 months ago21July 22, 2021338apache-2.0Java
A scalable web crawler framework for Java.
15 hours ago6gpl-3.0JavaScript
A visual no-code/code-free web crawler/spider易采集:一个可视化爬虫软件,可以无代码图形化的设计和执行爬虫任务
3 months ago85PHP
AV 电影管理系统, avmoo , javbus , javlibrary 爬虫,线上 AV 影片图书馆,AV 磁力链接数据库,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Crawlee8,4001620 hours ago246September 23, 2022106apache-2.0TypeScript
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
Awesome Web Scraping5,777
a month ago1otherMakefile
List of libraries, tools and APIs for web scraping and data processing.
Awesome Crawler5,548
a month ago24mit
A collection of awesome web crawler,spider in different languages
Alternatives To Crawlee
Select To Compare

Alternative Project Comparisons

A web scraping and browser automation library

NPM latest version Downloads Chat on discord Build Status

ℹ️ Crawlee is the successor to Apify SDK. 🎉 Fully rewritten in TypeScript for a better developer experience, and with even more powerful anti-blocking features. The interface is almost the same as Apify SDK so upgrading is a breeze. Read the upgrading guide to learn about the changes. ℹ️

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.

Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.

Crawlee is available as the crawlee NPM package.

👉 View full documentation, guides and examples on the Crawlee project website 👈


We recommend visiting the Introduction tutorial in Crawlee documentation for more information.

Crawlee requires Node.js 16 or higher.

With Crawlee CLI

The fastest way to try Crawlee out is to use the Crawlee CLI and choose the Getting started example. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.

npx crawlee create my-crawler
cd my-crawler
npm start

Manual installation

If you prefer adding Crawlee into your own project, try the example below. Because it uses PlaywrightCrawler we also need to install Playwright. It's not bundled with Crawlee to reduce install size.

npm install crawlee playwright
import { PlaywrightCrawler, Dataset } from 'crawlee';

// PlaywrightCrawler crawls the web using a headless
// browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
    // Use the requestHandler to process each of the crawled pages.
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();`Title of ${request.loadedUrl} is '${title}'`);

        // Save results as JSON to ./storage/datasets/default
        await Dataset.pushData({ title, url: request.loadedUrl });

        // Extract links from the current page
        // and add them to the crawling queue.
        await enqueueLinks();
    // Uncomment this option to see the browser window.
    // headless: false,

// Add first URL to the queue and start the crawl.

By default, Crawlee stores data to ./storage in the current working directory. You can override this directory via Crawlee configuration. For details, see Configuration guide, Request storage and Result storage.

🛠 Features

  • Single interface for HTTP and headless browser crawling
  • Persistent queue for URLs to crawl (breadth & depth first)
  • Pluggable storage of both tabular data and files
  • Automatic scaling with available system resources
  • Integrated proxy rotation and session management
  • Lifecycles customizable with hooks
  • CLI to bootstrap your projects
  • Configurable routing, error handling and retries
  • Dockerfiles ready to deploy
  • Written in TypeScript with generics

👾 HTTP crawling

  • Zero config HTTP2 support, even for proxies
  • Automatic generation of browser-like headers
  • Replication of browser TLS fingerprints
  • Integrated fast HTML parsers. Cheerio and JSDOM
  • Yes, you can scrape JSON APIs as well

💻 Real browser crawling

  • JavaScript rendering and screenshots
  • Headless and headful support
  • Zero-config generation of human-like fingerprints
  • Automatic browser management
  • Use Playwright and Puppeteer with the same interface
  • Chrome, Firefox, Webkit and many others

Usage on the Apify platform

Crawlee is open-source and runs anywhere, but since it's developed by Apify, it's easy to set up on the Apify platform and run in the cloud. Visit the Apify SDK website to learn more about deploying Crawlee to the Apify platform.


If you find any bug or issue with Crawlee, please submit an issue on GitHub. For questions, you can ask on Stack Overflow, in GitHub Discussions or you can join our Discord server.


Your code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see


This project is licensed under the Apache License 2.0 - see the file for details.

Popular Crawler Projects
Popular Scraper Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Web Crawler
Headless Browsers
Headless Chrome