Awesome Open Source

Programming Languages

Search results for scraper crawler

300 search results found

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

👾 Fast and simple video download library and CLI tool written in Go

Colly ⭐ 21,902

Elegant Scraper and Crawler Framework for Golang

Easyspider ⭐ 20,149

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化

Newspaper ⭐ 13,147

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee ⭐ 12,059

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Webmagic ⭐ 11,080

A scalable web crawler framework for Java.

Avbook ⭐ 8,777

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Awesome Web Scraping ⭐ 6,060

List of libraries, tools and APIs for web scraping and data processing.

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

Ferret ⭐ 5,540

Declarative web scraping

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Headless Chrome Crawler ⭐ 5,051

Distributed crawler powered by Headless Chrome

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

A Devtools driver for web automation and scraping

Mygptreader ⭐ 4,267

A community-driven way to read and chat with AI bots - powered by chatGPT.

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Querylist ⭐ 2,598

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Googlescraper ⭐ 2,540

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Web Scraping Framework

Weibo_terminater ⭐ 2,265

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

Awesome Puppeteer ⭐ 2,245

A curated list of awesome puppeteer resources.

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Scrapely ⭐ 1,668

A pure-python HTML screen-scraping library

Wombat ⭐ 1,297

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

Jd Autobuy ⭐ 1,291

Python爬虫，京东自动登录，在线抢购商品

Cariddi ⭐ 1,228

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Newpipeextractor ⭐ 1,070

NewPipe's core library for extracting data from streaming sites

Parliament Scraper ⭐ 1,049

Public Data Scraper for Parliament Data for the EU and other Parliaments

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

Querido Diario ⭐ 944

📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.

Mlscraper ⭐ 935

🤖 Scrape data from HTML websites automatically by just providing examples

Instagram Crawler ⭐ 922

Get Instagram posts/profile/hashtag data without using Instagram API

Crawler ⭐ 897

A high performance web crawler / scraper in Elixir.

Awesome Datahoarding ⭐ 892

List of data-hoarding related tools

Kimuraframework ⭐ 874

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

Crawly, a high-level web crawling & scraping framework for Elixir.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

[Unmaintained] A simple and clean video/music/image downloader 👾

Skrape.it ⭐ 714

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Bookcorpus ⭐ 698

Crawl BookCorpus

Google Play Scraper ⭐ 645

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Voyager ⭐ 618

crawl and scrape web pages in rust

Easy Scraping Tutorial ⭐ 618

Simple but useful Python web scraping tutorial code.

Newcrawler ⭐ 583

Free Web Scraping Tool with Java

Scrapedin ⭐ 572

LinkedIn Scraper (currently working 2020)

OSINT Swiss Army Knife

Jvppeteer ⭐ 549

Headless Chrome For Java （Java 爬虫）

Nintendo Switch Eshop ⭐ 513

Crawler for Nintendo Switch eShop

Spidermon ⭐ 486

Scrapy Extension for monitoring spiders execution.

Nodejs Stuff ⭐ 484

Node.js libs I want to keep in mind.

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

Pywebcopy ⭐ 455

Locally saves webpages to your hard disk with images, css, js & links as is.

Scrapple ⭐ 452

A framework for creating semi-automatic web content extractors

Awesome Scrapy ⭐ 450

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

Movie metadata scraper

The fastest web crawler written in Rust. Maintained by @a11ywatch.

Fbcrawl ⭐ 415

A Facebook crawler

Pulsarrpa ⭐ 413

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

Linkedin Profile Scraper Api ⭐ 404

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

Dataflowkit ⭐ 394

Extract structured data from web sites. Web sites scraping.

Scavenger ⭐ 384

Crawler (Bot) searching for credential leaks on paste sites.

Search Engines Scraper ⭐ 377

Search google, bing, yahoo, and other search engines with python

Coronadatascraper ⭐ 372

COVID-19 Coronavirus data scraped from government and curated data sources.

Moodle Dl ⭐ 355

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

Scrapy Zyte Smartproxy ⭐ 348

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

Hquery.php ⭐ 345

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

Linkedindumper ⭐ 337

Python 3 script to dump/scrape/extract company employees from LinkedIn API

Freshonions Torscraper ⭐ 313

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

Be nice on the web

Advanced python library to scrap Twitter (tweets, users) from unofficial API

Memorious ⭐ 302

Lightweight web scraping toolkit for documents and structured data.

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

Web Scraping ⭐ 281

Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Harvestman ⭐ 278

Quick and dirty web crawling.

Youtube Projects ⭐ 272

This repository contains all the code I use in my YouTube tutorials.

A web crawler for Go

Python Automation Scripts ⭐ 264

Simple yet powerful automation stuffs.

一个灵活、友好的爬虫框架

Ruiji.net ⭐ 261

crawler framework, distributed crawler extractor

Weibo_terminator_workflow ⭐ 259

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

Arachnid ⭐ 246

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Awesome Crawler Cn ⭐ 243

互联网爬虫，蜘蛛，数据采集器，网页解析器的汇总，因新技术不断发展，新框架层出不穷，此文会不断更新..

Pythonscraping ⭐ 240

The code of book: Python Scraping

Rcrawler ⭐ 240

An R web crawler and scraper

Lightnovel_epub ⭐ 233

🍭 epub generator for (light)novels (轻)小说 epub 生成器，支持站点：轻之国度、轻小说文库

Nudecrawler ⭐ 231

Crawl telegra.ph searching for nudes!

Goose Parser ⭐ 222

Universal scraping tool, which allows you to extract data using multiple environments

Wayback Machine Scraper ⭐ 219

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Make a ZIM file from any Web site and surf offline!

A loose framework for crawling and scraping web sites.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Related Searches

Python Crawler (4,545)

Python Scraper (3,513)

Javascript Scraper (2,047)

Scraper Scrape (1,534)

Scraper Web Crawler (1,528)

Javascript Crawler (1,142)

Crawler Spider (1,044)

Crawler Scrapy (1,002)

Java Crawler (806)

Html Scraper (757)

1-100 of 300 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.