Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 10 Web Crawling Open Source Projects

Open source projects categorized as Web Crawling
Categories  > Web Crawling
Edit Category
apify/crawlee ⭐ 11,229
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 0 total releases 0 most recent commit over 2 years ago
omkarcloud/botasaurus ⭐ 4,728
The All in One Framework to Build Undefeatable Scrapers
dependent packages 0 total releases 0 most recent commit 3 months ago
internetarchive/heritrix3 ⭐ 2,579
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 0 total releases 0 most recent commit over 2 years ago
scrapinghub/scrapyrt ⭐ 793
HTTP API for Scrapy spiders
dependent packages 0 total releases 0 most recent commit over 2 years ago
DemonDamon/Listed-company-news-crawl-and-text-analysis ⭐ 689
从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本分析、提取特征集，然后利用SVM、随机森林等分类器进行训练，最后对实施抓取的新闻数据进行分类预测
dependent packages 0 total releases 0 most recent commit over 3 years ago
jaeksoft/opensearchserver ⭐ 419
Open-source Enterprise Grade Search Engine Software
dependent packages 0 total releases 0 most recent commit almost 5 years ago
crwlrsoft/crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
dependent packages 0 total releases 0 most recent commit over 2 years ago
TurnerSoftware/InfinityCrawler ⭐ 221
A simple but powerful web crawler library for .NET
dependent packages 0 total releases 0 most recent commit almost 3 years ago
scrapehero-code/amazon-scraper ⭐ 219
A simple web scraper to extract Product Data and Pricing from Amazon
dependent packages 0 total releases 0 most recent commit about 3 years ago
ayakashi-io/ayakashi ⭐ 177
:zap: Ayakashi.io - The next generation web scraping framework
dependent packages 0 total releases 0 most recent commit almost 3 years ago
Get A Weekly Email With Trending Web Crawling Projects
No Spam. Unsubscribe easily at any time.
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2026 Awesome Open Source.  All rights reserved.