Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Web Crawling Open Source Projects

Open source projects categorized as Web Crawling
Categories > Web Crawling
Crawlee ⭐ 12,250
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit 4 days agonpm crawlee} Downloads
Heritrix3 ⭐ 2,579
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 2total releases 9most recent commit 6 months ago
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7most recent commit 6 months ago
Listed Company News Crawl And Text Analysis ⭐ 689
从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本
most recent commit a year ago
Opensearchserver ⭐ 419
Open-source Enterprise Grade Search Engine Software
total releases 17most recent commit 3 years ago
Botasaurus ⭐ 331
The All in One Web Scraping Framework
total releases 31most recent commit 4 months ago
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25most recent commit 4 months ago
Infinitycrawler ⭐ 221
A simple but powerful web crawler library for .NET
total releases 7most recent commit 8 months ago
Amazon Scraper ⭐ 219
A simple web scraper to extract Product Data and Pricing from Amazon
most recent commit a year ago
Ayakashi ⭐ 177
⚡ Ayakashi.io - The next generation web scraping framework
dependent packages 2total releases 40most recent commit 10 months agonpm ayakashi} Downloads
Bet On Sibyl ⭐ 157
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
most recent commit 7 years ago
Gotor ⭐ 150
This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
total releases 3most recent commit 5 months ago
Ralger ⭐ 145
ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.
total releases 9most recent commit a year ago
Scrapy Training ⭐ 141
Scrapy Training companion code
most recent commit 5 years ago
Raspagem De Dados Para Iniciantes ⭐ 115
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
most recent commit a year ago
Bancocentralbrasil ⭐ 112
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
most recent commit 2 years ago
Seleniumcrawler ⭐ 105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
most recent commit 5 years ago
Krawler ⭐ 96
A web crawling framework written in Kotlin
most recent commit 4 years ago
Terpene Profile Parser For Cannabis Strains ⭐ 93
Parser and database to index the terpene profile of different strains of Cannabis from online databases
most recent commit a year ago
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 4 years ago
Malheatmap ⭐ 87
An extension for tracking your activities on myanimelist.net
most recent commit 4 months ago
Katastrophe ⭐ 86
Command Line Tool to download torrents
total releases 18most recent commit 7 years ago
Bancocentralbrasil ⭐ 71
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
most recent commit 4 years ago
The 10 Most Depended On Web Crawling Open Source Projects
Courlan ⭐ 55
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters
dependent packages 5total releases 25latest release November 28, 2023most recent commit 4 months agopypi courlan} Downloads
Amazon_scraper ⭐ 64
Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt
dependent packages 2total releases 13latest release June 10, 2022most recent commit 4 months agonpm @scrapingant/amazon-proxy-scraper} Downloads
Ayakashi ⭐ 177
:zap: Ayakashi.io - The next generation web scraping framework
dependent packages 2total releases 40latest release June 29, 2023most recent commit 10 months agonpm ayakashi} Downloads
Heritrix3 ⭐ 2,579
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 2total releases 9latest release July 27, 2022most recent commit 6 months ago
Newspaperjs ⭐ 63
News extraction and scraping. Article Parsing
dependent packages 2total releases 7latest release May 16, 2020most recent commit a year agonpm newspaperjs} Downloads
Scrawler ⭐ 11
Scala web crawling and scraping using fs2 streams
dependent packages 1total releases 5latest release July 03, 2017most recent commit 7 years ago
Genmine ⭐ 7
GenBank Record downloader for taxonomists
dependent packages 1total releases 20latest release September 26, 2023most recent commit 8 months agopypi GenMine} Downloads
Tibia.py ⭐ 32
API to parse tibia.com content into python objects.
dependent packages 1total releases 92latest release October 02, 2023most recent commit 4 months agopypi tibia.py} Downloads
Url Frontier ⭐ 34
API definition, resources and reference implementation of URL Frontiers
dependent packages 1total releases 11latest release November 21, 2022most recent commit 5 months ago
The 10 Latest Releases In Web Crawling Open Source Projects
Botasaurus ⭐ 331
The All in One Web Scraping Framework
total releases 31latest release December 11, 2023most recent commit 4 months ago
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25latest release December 01, 2023most recent commit 4 months ago
Courlan ⭐ 55
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters
dependent packages 5total releases 25latest release November 28, 2023most recent commit 4 months agopypi courlan} Downloads
Tibia.py ⭐ 32
API to parse tibia.com content into python objects.
dependent packages 1total releases 92latest release October 02, 2023most recent commit 4 months agopypi tibia.py} Downloads
Genmine ⭐ 7
GenBank Record downloader for taxonomists
dependent packages 1total releases 20latest release September 26, 2023most recent commit 8 months agopypi GenMine} Downloads
Scrapyrt ⭐ 793
HTTP API for Scrapy spiders
total releases 7latest release September 20, 2023most recent commit 6 months ago
Socials_regex ⭐ 8
🪡 Social account detection and extraction in ruby, e.g. for crawling/scraping.
total releases 1latest release July 02, 2023most recent commit 8 months ago
Ayakashi ⭐ 177
:zap: Ayakashi.io - The next generation web scraping framework
dependent packages 2total releases 40latest release June 29, 2023most recent commit 10 months agonpm ayakashi} Downloads
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.