Awesome Open Source
Awesome Open Source
Combined Topics
crawling x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 352 Crawling Open Source Projects on Github

Categories > Data Processing > Crawling
Scrapy ⭐ 41,588
Scrapy, a fast high-level web crawling & scraping framework for Python.
Colly ⭐ 14,759
Elegant Scraper and Crawler Framework for Golang
Newspaper ⭐ 11,309
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
Ferret ⭐ 4,629
Declarative web scraping
Apify Js ⭐ 3,009
Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Nutch ⭐ 2,237
Apache Nutch is an extensible and scalable web crawler
Awesome Puppeteer ⭐ 1,552
A curated list of awesome puppeteer resources.
Lulu ⭐ 752
[Unmaintained] A simple and clean video/music/image downloader 👾
Scrapyrt ⭐ 651
HTTP API for Scrapy spiders
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
Scrapy Selenium ⭐ 611
Scrapy middleware to handle javascript pages using selenium
Crawly ⭐ 499
Crawly, a high-level web crawling & scraping framework for Elixir.
Isp Data Pollution ⭐ 400
ISP Data Pollution to Protect Private Browsing History with Obfuscation
Dataflowkit ⭐ 394
Extract structured data from web sites. Web sites scraping.
Webster ⭐ 369
a reliable high-level web crawling & scraping framework for Node.js.
Spidermon ⭐ 339
Scrapy Extension for monitoring spiders execution.
Stopstalk Deployment ⭐ 292
Stop stalking and start StopStalking 😉
Bhban_rpa ⭐ 282
6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.
Gopa ⭐ 281
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Spidy ⭐ 268
The simple, easy to use command line web crawler.
Sasila ⭐ 264
一个灵活、友好的爬虫框架
Memorious ⭐ 254
Lightweight web scraping toolkit for documents and structured data.
Holiday Cn ⭐ 235
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
Instagram Bot ⭐ 215
An Instagram bot developed using the Selenium Framework
Linkedin Profile Scraper ⭐ 191
🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
N2h4 ⭐ 179
네이버 뉴스 수집을 위한 도구
Antch ⭐ 177
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Crawler ⭐ 138
Go process used to crawl websites
Squidwarc ⭐ 131
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Corpuscrawler ⭐ 131
Crawler for linguistic corpora
Massivedl ⭐ 125
Download a large list of files concurrently
Cariddi ⭐ 110
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more...
Grawler ⭐ 71
Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.
Arachnid ⭐ 65
Powerful web scraping framework for Crystal
Dig Etl Engine ⭐ 65
Download DIG to run on your laptop or server.
Dotnetcrawler ⭐ 63
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
Robots.txt ⭐ 61
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
Pomp ⭐ 60
Screen scraping and web crawling framework
Harvester ⭐ 59
Web crawling and document processing through a usable interface.
Bathyscaphe ⭐ 59
Fast, highly configurable, cloud native dark web crawler.
Python Crawling Tutorial ⭐ 55
Python crawling tutorial
Talospider ⭐ 55
talospider - A simple,lightweight scraping micro-framework
Diffbot Php Client ⭐ 49
[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library
Linkedin Skills Crawler ⭐ 49
A simple Python script to crawl complete list of LinkedIn skills
Learn.scrapinghub.com ⭐ 49
Scrapinghub Learning Center. Report issues in Jira: Report issues in Jira: https://scrapinghub.atlassian.net/projects/WEB
Argus ⭐ 45
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9
Proxycrawl Python ⭐ 45
ProxyCrawl Python library for scraping and crawling
Double Agent ⭐ 44
A test suite of common scraper detection techniques. See how detectable your scraper stack is.
Crawling Projects ⭐ 43
Web scraping and automation using python
Proxifier ⭐ 43
A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.
Tech Seo Crawler ⭐ 43
Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.
Wget Lua ⭐ 40
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Jkcrawler ⭐ 40
使用 Scrapy 写成的 JK 爬虫，图片源自哔哩哔哩、Tumblr、Instagram，以及微博、Twitter
Pdf Crawler ⭐ 39
SimFin's open source PDF crawler
Jason The Miner ⭐ 39
⛏ A versatile Web scraper for Node.js
Flink Crawler ⭐ 35
Continuous scalable web crawler built on top of Flink and crawler-commons
Podcastcrawler ⭐ 33
PHP library to find podcasts
Warcworker ⭐ 33
A dockerized, queued high fidelity web archiver based on Squidwarc
Bots Zoo ⭐ 32
Engineeringteam ⭐ 32
와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.
Telegram Crawler ⭐ 32
Automatically detect changes made to the official Telegram sites.
Serverless Instagram Crawler ⭐ 29
serverless, instagram hashtag crawler with lambda, dynamoDB
Scrape Github Trending ⭐ 29
Tutorial for web scraping / crawling with Node.js.
Baiduspider ⭐ 27
项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。
Video Crawler ⭐ 26
Crawl websites for videos from Youtube, Vimeo, Soundcloud, etc
Bluebird ⭐ 23
Unofficial Python client for Twitter
Crawlkit ⭐ 23
A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.
Custom Crawler ⭐ 23
🌌 High productivity semi-automatic crawler generator 🛠️🧰
Trend Monitoring ⭐ 23
실시간 트렌드 데이터 분석/모니터링 시스템 tremo
Scrapy Distributed ⭐ 23
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Auctus ⭐ 22
Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index
Ferret Server ⭐ 22
Advanced declarative web scraping
Puppet Master ⭐ 22
Puppeteer as a service hosted on Saasify.
Spidyquotes ⭐ 22
Example site for web scraping tutorials
Zcrawl ⭐ 21
An open source web crawling platform
Xxx___dead___xxx ⭐ 21
b̶̡̪̬͒l̸̰̗̝̀ỏ̷̡̩g̴͇̑g̶̲̱̽͐i̵̹͗n̶̤̥͂̅̆g̴̮̾̅͜ ̷̧͎͆i̷̛͒͜͠n̸̥̺͒ ̶͚͚͊̿͜t̸̺͙̭̆̊̈́ḧ̶̟́̐e̸̱͔̟̓̓͝ ̶̨͔̾͛̑d̵̥̣̏ȧ̷̼̊r̷̰̝̥̅̌͝k̵̟̥̞̉̍͛
Crawling Framework ⭐ 21
Easily crawl news portals or blog sites using Storm Crawler.
Crowleer ⭐ 20
Powerful C++ web crawler based on libcurl
Slackwebhooksgithubcrawler ⭐ 19
Search for Slack Webhooks token publicly exposed on Github
Mimo Crawler ⭐ 19
A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Popular_restaurants_from_officials ⭐ 19
서울시 공무원의 업무추진비를 분석하여 진짜 맛집 찾기 프로젝트
Mal Analysis ⭐ 18
github repo for MyAnimeList analysis. Also links to the MAL dataset.
Crawler ⭐ 17
Crawl your own website with various clients for SEO and indexing purposes.
Pdf_downloader ⭐ 17
A Scrapy Spider for downloading PDF files from a webpage.
Crawlly ⭐ 16
A simple web crawller in go
Proxycrawl Node ⭐ 16
ProxyCrawl Node library for scraping and crawling
Go Scrapy ⭐ 16
Web crawling and scraping framework for Golang
Old_ver_bot ⭐ 15
파이썬 슬랙 크롤링 봇입니다. It's slack bot made by python+flask+bs4. version of go below
Ragno ⭐ 15
Common Lisp Web crawling library based on Psychiq.
Facebook Scraper For Non English User ⭐ 15
crawling facebok page
Kasthack.osp ⭐ 15
Генератор сырых дампов пользователей VK.
Feedsearch Crawler ⭐ 14
Crawl sites for RSS, Atom, and JSON feeds.
Img Cli ⭐ 14
An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL
Html Article Extractor ⭐ 14
A web page content extractor
Pyreptile ⭐ 14
web crawling & scraping framework for Python
Proxycrawl Php ⭐ 14
ProxyCrawl PHP library for scraping and crawling websites
Serritor ⭐ 13
Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.
The Seinfeld Chronicles ⭐ 13
A dataset for textual analysis on arguably the best written comedy television show ever.
Bilib ⭐ 13
整合多个B站原生API，并结合爬取技术的Python爬取用lib
1-100 of 352 projects

Related Projects

Python Crawling Projects (176)
Crawler Crawling Projects (142)
Scraping Crawling Projects (90)
Python Crawler Crawling Projects (53)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021