Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Scraper Web Crawler Webscraper Open Source Projects

Open source projects categorized as Scraper Web Crawler Webscraper
Categories > Data Processing > Scraper
Categories > Data Processing > Web Crawler
Categories > Webscraper
Crawlee ⭐ 11,957
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit 7 days agonpm crawlee} Downloads
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 4 months ago
Autoscraper ⭐ 5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit a year agopypi autoscraper} Downloads
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27most recent commit 9 months agonpm osmosis} Downloads
Soup ⭐ 2,074
Web Scraper in Go, similar to BeautifulSoup
dependent packages 57total releases 10most recent commit 6 months ago
Tomorrow ⭐ 1,463
Magic decorator syntax for asynchronous code in Python
dependent packages 2total releases 5most recent commit 4 years agopypi tomorrow} Downloads
Stealth ⭐ 923
🚀 Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
most recent commit 7 months ago
Spidr ⭐ 775
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dependent packages 16total releases 26most recent commit 3 months agorubygems spidr} Downloads
Xidel ⭐ 611
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
most recent commit 5 months ago
Scrapers ⭐ 511
A list of scrapers from around the web.
most recent commit a year ago
Phpscraper ⭐ 486
A universal web-util for PHP.
total releases 34most recent commit 5 months ago
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
total releases 10most recent commit 5 years ago
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58most recent commit 3 months ago
Basketball_reference_web_scraper ⭐ 382
NBA Stats API via Basketball Reference
total releases 39most recent commit 3 months ago
Scrape Linkedin Selenium ⭐ 353
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
most recent commit 2 years ago
Hquery.php ⭐ 345
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
dependent packages 4total releases 23most recent commit 3 months agonpm hquery.php} Downloads
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25most recent commit 3 months ago
Gopa ⭐ 281
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
total releases 6most recent commit 3 years ago
Web Scraping ⭐ 276
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
most recent commit 2 years ago
Ant ⭐ 271
A web crawler for Go
total releases 8most recent commit a year ago
Rcrawler ⭐ 240
An R web crawler and scraper
total releases 9most recent commit 4 years ago
Summarizer ⭐ 236
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
most recent commit 2 years ago
Getsy ⭐ 234
A simple browser/client-side web scraper.
dependent packages 2total releases 5most recent commit 7 years agonpm getsy} Downloads
The 10 Most Depended On Scraper Web Crawler Webscraper Open Source Projects
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months agonpm osmosis} Downloads
Soup ⭐ 2,074
Web Scraper in Go, similar to BeautifulSoup
dependent packages 57total releases 10latest release January 16, 2022most recent commit 6 months ago
Html Metadata ⭐ 115
MetaData html scraper and parser for Node.js (supports Promises and callback style)
dependent packages 26total releases 20latest release November 20, 2019most recent commit 4 years agonpm html-metadata} Downloads
Nodejs Web Scraper ⭐ 54
dependent packages 5total releases 81latest release January 07, 2023most recent commit a year agonpm nodejs-web-scraper} Downloads
Hquery.php ⭐ 345
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
dependent packages 4total releases 23latest release July 19, 2019most recent commit 3 months agonpm hquery.php} Downloads
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58latest release November 11, 2023most recent commit 3 months ago
Yolo Scraper ⭐ 5
A simple way to structure your web scraper.
dependent packages 3total releases 6latest release February 20, 2021most recent commit 3 years agonpm yolo-scraper} Downloads
Kirinuki Core ⭐ 12
Kirinuki is a library that convert any html to JSON using CSS selectors.
dependent packages 3total releases 14latest release March 06, 2019most recent commit 10 months agonpm kirinuki-core} Downloads
The 10 Latest Releases In Scraper Web Crawler Webscraper Open Source Projects
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 7 years ago
Google News Scraper ⭐ 144
Lightweight scraper for Google News
dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months agonpm google-news-scraper} Downloads
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25latest release December 01, 2023most recent commit 3 months ago
Kenpompy ⭐ 61
A simple yet comprehensive web scraper for kenpom.com.
total releases 9latest release November 22, 2023most recent commit 4 months ago
Basketball_reference_web_scraper ⭐ 382
NBA Stats API via Basketball Reference
total releases 39latest release November 14, 2023most recent commit 3 months ago
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58latest release November 11, 2023most recent commit 3 months ago
Python Scrapfly ⭐ 14
Scrapfly Python SDK for headless browsers and proxy rotation
total releases 33latest release October 27, 2023most recent commit 4 months ago
Tt Mobile ⭐ 6
PWA and web scraper for swiss table tennis results
dependent packages 1total releases 11latest release August 03, 2023most recent commit 4 months agonpm tt-mobile-widgets} Downloads
Phpscraper ⭐ 486
A universal web-util for PHP.
total releases 34latest release June 18, 2023most recent commit 5 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.