The Top 41 Crawling Open Source Projects

Awesome Open Source

Awesome Open Source

Combined Topics

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210

The Top 41 Crawling Open Source Projects

Categories > Data Processing > Crawling

Scrapy ⭐ 40,401

Scrapy, a fast high-level web crawling & scraping framework for Python.

Colly ⭐ 13,678

Elegant Scraper and Crawler Framework for Golang

Newspaper ⭐ 10,928

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Headless Chrome Crawler ⭐ 4,956

Distributed crawler powered by Headless Chrome

Ferret ⭐ 4,463

Declarative web scraping

Apify Js ⭐ 2,828

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Nutch ⭐ 2,191

Apache Nutch is an extensible and scalable web crawler

Awesome Puppeteer ⭐ 1,566

A curated list of awesome puppeteer resources.

[Unmaintained] A simple and clean video/music/image downloader 👾

Scrapyrt ⭐ 651

HTTP API for Scrapy spiders

Easy Scraping Tutorial ⭐ 590

Simple but useful Python web scraping tutorial code.

Scrapy Selenium ⭐ 569

Scrapy middleware to handle javascript pages using selenium

Dataflowkit ⭐ 458

Extract structured data from web sites. Web sites scraping.

Crawly, a high-level web crawling & scraping framework for Elixir.

Isp Data Pollution ⭐ 426

ISP Data Pollution to Protect Private Browsing History with Obfuscation

Webster ⭐ 365

a reliable high-level web crawling & scraping framework for Node.js.

Spidermon ⭐ 320

Scrapy Extension for monitoring spiders execution.

一个灵活、友好的爬虫框架

Stopstalk Deployment ⭐ 278

Stop stalking and start StopStalking 😉

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

The simple, easy to use command line web crawler.

Memorious ⭐ 250

Distributed crawling framework for documents and structured data.

cdp4j - Chrome DevTools Protocol for Java

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Linkedin Profile Scraper ⭐ 181

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

네이버 뉴스 수집을 위한 도구

Instagram Bot ⭐ 166

An Instagram bot developed using the Selenium Framework

Holiday Cn ⭐ 165

📅🇨🇳 中国法定节假日数据自动每日抓取国务院公告

Bhban_rpa ⭐ 158

6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Crawler ⭐ 148

Go process used to crawl websites

Massivedl ⭐ 141

Download a large list of files concurrently

Corpuscrawler ⭐ 130

Crawler for linguistic corpora

Squidwarc ⭐ 127

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Dotnetcrawler ⭐ 104

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Grawler ⭐ 102

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

Dig Etl Engine ⭐ 83

Download DIG to run on your laptop or server.

Arachnid ⭐ 70

Powerful web scraping framework for Crystal

Python Crawling Tutorial ⭐ 57

Python crawling tutorial

Crawling Projects ⭐ 49

Web scraping and automation using python

Bots Zoo ⭐ 29

Pdf_downloader ⭐ 18

A Scrapy Spider for downloading PDF files from a webpage.

1-41 of 41 projects

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210