Awesome Open Source
Awesome Open Source
Combined Topics
python x
web-crawler x

The Top 1,255 Python Web Crawler Open Source Projects

Categories > Data Processing > Web Crawler
Awesome Web Scraping ⭐ 5,074
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 3 days ago
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
total releases 15most recent commit a year ago
Selenium Python Helium ⭐ 2,890
Selenium-python but lighter: Helium is the best Python library for web automation.
dependent packages 9total releases 26most recent commit a month agopypi helium} Downloads
Automatic Udemy Course Enroller Get Paid Udemy Courses For Free ⭐ 2,648
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
total releases 10most recent commit a month ago
Grab ⭐ 2,174
Web Scraping Framework
dependent packages 4total releases 120most recent commit 4 months agopypi grab} Downloads
30 Days Of Python ⭐ 1,725
Learn Python for the next 30 (or so) Days.
most recent commit 8 months ago
Pspider ⭐ 1,675
简单易用的Python爬虫框架，QQ交流群：597510560
most recent commit 20 days ago
Snoop ⭐ 1,455
Snoop — инструмент разведки на основе открытых данных (OSINT world)
most recent commit 17 days ago
Dat8 ⭐ 1,439
General Assembly's 2015 Data Science course in Washington, DC
most recent commit 6 years ago
100projectsofcode ⭐ 1,293
A list of practical knowledge-building projects.
most recent commit a year ago
Pythoncode Tutorials ⭐ 1,130
The Python Code Tutorials
most recent commit 2 days ago
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
total releases 65most recent commit 6 months ago
Requests Cache ⭐ 963
Transparent persistent cache for python requests
dependent packages 308total releases 79most recent commit 10 days agopypi requests-cache} Downloads
Webscraping Open Project ⭐ 935
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
most recent commit 21 days ago
Youtube_tutorials ⭐ 855
Collection of scripts corresponding to LucidProgramming YouTube tutorials
most recent commit 4 months ago
Faster Than Requests ⭐ 838
Faster requests on Python 3
dependent packages 4total releases 18most recent commit 4 days agopypi faster-than-requests} Downloads
Zhihu Spider ⭐ 719
A web spider for zhihu.com
most recent commit 7 years ago
Lightnovel Crawler ⭐ 661
Generate and download e-books from online sources.
dependent packages 1total releases 30most recent commit 5 days agopypi ebook-crawler} Downloads
Selectolax ⭐ 547
Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
dependent packages 17total releases 34most recent commit 9 days agopypi selectolax} Downloads
Gazpacho ⭐ 543
🥫 The simple, fast, and modern web scraping library
dependent packages 5total releases 19most recent commit a year agopypi gazpacho} Downloads
Trafilatura ⭐ 510
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
dependent packages 5total releases 30most recent commit 16 hours agopypi trafilatura} Downloads
Scrapy Fake Useragent ⭐ 471
Random User-Agent middleware based on fake-useragent
dependent packages 1total releases 12most recent commit 2 years agopypi scrapy-fake-useragent} Downloads
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
total releases 10most recent commit 3 years ago
Crosslinked ⭐ 441
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
most recent commit a month ago
Facepager ⭐ 431
Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
most recent commit 2 months ago
Instascrape ⭐ 429
Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically
total releases 56most recent commit 4 months ago
Company Crawler ⭐ 386
天眼查爬虫&企查查爬虫，指定关键字爬取公司信息
most recent commit 3 months ago
Proxy_requests ⭐ 367
a class that uses scraped proxies to make http GET/POST requests (Python requests)
dependent packages 2total releases 49most recent commit 2 years agopypi proxy-requests} Downloads
Learn To Identify Similar Images ⭐ 358
Record my python script about Iearning to identify similar images
most recent commit 4 years ago
Complete Life Cycle Of A Data Science Project ⭐ 306
Complete-Life-Cycle-of-a-Data-Science-Project
most recent commit a day ago
Dude ⭐ 305
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
total releases 36most recent commit 2 days ago
Basketball_reference_web_scraper ⭐ 300
NBA Stats API via Basketball Reference
total releases 38most recent commit 4 months ago
Scrape Linkedin Selenium ⭐ 288
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
most recent commit 10 months ago
Spidy ⭐ 287
The simple, easy to use command line web crawler.
total releases 12most recent commit 2 months ago
Web Scraping ⭐ 276
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
most recent commit 5 months ago
Youtube Projects ⭐ 272
This repository contains all the code I use in my YouTube tutorials.
most recent commit 9 months ago
Archivebot ⭐ 270
ArchiveBot, an IRC bot for archiving websites
most recent commit a month ago
Onlyfans Dl ⭐ 268
OnlyFans content downloader
most recent commit 5 months ago
City Scrapers ⭐ 253
Scrape, standardize and share public meetings from local government websites
most recent commit 3 months ago
Lagoujob ⭐ 250
Job data mining repo for lagou.com
most recent commit 3 years ago
Tradingview Data Scraper ⭐ 250
Extract price and indicator data from TradingView charts to create ML datasets
most recent commit a year ago
Summarizer ⭐ 236
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
most recent commit 4 months ago
Docbao ⭐ 233
Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam
most recent commit a year ago
Quora Api ⭐ 232
An unofficial API for Quora.
most recent commit 6 years ago
Kochat ⭐ 222
Opensource Korean chatbot framework
total releases 27most recent commit a year ago
Netflix Clone ⭐ 219
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
most recent commit 4 months ago
Wayback Machine Scraper ⭐ 219
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
total releases 8most recent commit a year ago
Short Jokes Dataset ⭐ 205
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
most recent commit 3 years ago
Espnff ⭐ 202
ESPN Fantasy Football API
total releases 11most recent commit 4 years ago
Portia Dashboard ⭐ 190
portia-dashboard is a visual web crawler based on scrapinghub/portia
most recent commit 4 years ago
Zhihu Crawler People ⭐ 179
A simple distributed crawler for zhihu && data analysis
most recent commit 3 years ago
Ignareo Isml Auto Voter ⭐ 175
Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns!
most recent commit 4 months ago
Trump Lies ⭐ 175
Tutorial: Web scraping in Python with Beautiful Soup
most recent commit 4 years ago
Twitter Intelligence ⭐ 169
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
most recent commit 2 years ago
First Web Scraper ⭐ 167
A step-by-step guide to writing a web scraper with Python
most recent commit a year ago
Hq_bot ⭐ 161
📲 Bot to help solve HQ trivia
most recent commit 4 years ago
Cocrawler ⭐ 159
CoCrawler is a versatile web crawler built using modern tools and concurrency.
total releases 8most recent commit 2 months ago
Bet On Sibyl ⭐ 157
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
most recent commit 5 years ago
Juno_crawler ⭐ 147
Scrapy crawler to collect data on the back catalog of songs listed for sale.
most recent commit 3 years ago
Web Database Analytics ⭐ 144
Web scrapping and related analytics using Python tools
most recent commit 2 years ago
Stardox ⭐ 143
Github stargazers information gathering tool
most recent commit 4 months ago
Netpwn ⭐ 143
Tool made to automate tasks of pentesting.
most recent commit 3 years ago
Scrapy Training ⭐ 141
Scrapy Training companion code
most recent commit 3 years ago
Learnpythonforresearch ⭐ 137
This repository provides everything you need to get started with Python for (social science) research.
most recent commit 2 years ago
Competitive_programming_score_api ⭐ 134
API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit
most recent commit 24 days ago
Dark Fantasy Hack Tool ⭐ 129
DDOS Tool: To take down small websites with HTTP FLOOD. Port scanner: To know the open ports of a site. FTP Password Cracker: To hack file system of websites.. Banner Grabber: To get the service or software running on a port. (After knowing the software running google for its vulnerabilities.) Web Spider: For gathering web application hacking information. Email scraper: To get all emails related to a webpage IMDB Rating: Easy way to access the movie database. Both .exe(compressed as zip) and .py
most recent commit a year ago
Social Media Profile Scrapers ⭐ 129
Fetch user's data across social media
most recent commit 3 months ago
Zillow ⭐ 129
Zillow Scraper for Python using Selenium
most recent commit 3 years ago
Scrapers ⭐ 128
Lots and lots of web scrapers
most recent commit 5 years ago
Fisherman ⭐ 117
CLI program that collects information from facebook user profiles via Selenium.
most recent commit 3 months ago
Not Your Average Web Crawler ⭐ 116
A web crawler (for bug hunting) that gathers more than you can imagine.
total releases 36most recent commit 8 months ago
Geeksforgeeksscrapper ⭐ 116
Scrapes g4g and creates PDF
most recent commit 2 years ago
Crawlbox ⭐ 112
Easy way to brute-force web directory.
most recent commit 3 years ago
Tiktokbot ⭐ 109
A TikTokBot that downloads trending tiktok videos and compiles them using FFmpeg
most recent commit a year ago
Hockey Scraper ⭐ 107
Python Package for scraping NHL Play-by-Play and Shift data
total releases 52most recent commit 3 months ago
Seleniumcrawler ⭐ 105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
most recent commit 3 years ago
Bancocentralbrasil ⭐ 102
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
total releases 3most recent commit 7 months ago
Zhihu_crawler ⭐ 100
a crawler for zhihu
most recent commit 5 years ago
Cowin Vaccine Notifier ⭐ 97
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
most recent commit a year ago
Wereadscan ⭐ 96
扫描“微信读书”已购图书并下载本地PDF的爬虫
total releases 7most recent commit 10 months ago
Zimit ⭐ 95
Make a ZIM file from any Web site and surf offline!
most recent commit 8 days ago
Brutescrape ⭐ 95
A web scraper for generating password files based on plain text found
most recent commit 3 years ago
Wswp ⭐ 95
Code for the second edition Web Scraping with Python book by Packt Publications
most recent commit 3 years ago
Ir ⭐ 93
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
most recent commit a year ago
Reader ⭐ 90
Extract clean(er), readable text from web pages via Mercury Web Parser.
most recent commit 2 months ago
Saveddit ⭐ 90
Bulk Downloader for Reddit
total releases 15most recent commit 8 months ago
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 2 years ago
Udemy_bot ⭐ 88
An automation bot for free Udemy courses
most recent commit a year ago
Nordvpn Switcher ⭐ 86
Rotate between different NordVPN servers with ease. Works both on Linux and Windows without any required changes to your code!
dependent packages 1total releases 21most recent commit 19 days agopypi nordvpn-switcher} Downloads
Webcrawler ⭐ 86
Web crawler to download pictures from zhihu.com
most recent commit 3 years ago
Letterboxd_recommendations ⭐ 84
Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username
most recent commit 3 months ago
Ebaymarketanalyzer ⭐ 83
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
most recent commit 10 months ago
Openscraper ⭐ 80
An open source webapp for scraping: towards a public service for webscraping
most recent commit 4 months ago
Tableau Scraping ⭐ 79
Tableau scraper python library. R and Python scripts to scrape data from Tableau viz
dependent packages 1total releases 40most recent commit 5 months agopypi TableauScraper} Downloads
Awesomewebscraping ⭐ 77
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 4 months ago
Web Scraping ⭐ 76
Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!
most recent commit 4 years ago
Python Web Scraping Second Edition ⭐ 76
Python Web Scraping Second Edition, published by Packt
most recent commit 2 years ago
Pymarketcap ⭐ 74
Python3 API wrapper and web scraper for https://coinmarketcap.com
total releases 92most recent commit 3 years ago
Raspagem De Dados Para Iniciantes ⭐ 74
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
most recent commit 2 years ago
Scraping Ebay ⭐ 73
Scraping Ebay's products using Scrapy Web Crawling Framework
most recent commit 4 months ago
1-100 of 1,255 projects
Related Awesome Lists
Python Django Projects (26,307)
Python Python3 Projects (22,972)
Python Dataset Projects (14,792)
Python Pytorch Projects (14,667)
Python Flask Projects (14,408)
Python Machine Learning Projects (14,100)
Python Docker Projects (13,757)
Python Tensorflow Projects (13,736)
Python Deep Learning Projects (13,093)
Python Jupyter Notebook Projects (12,976)
Python Network Projects (11,547)
Python Learning Projects (10,661)
Python Algorithms Projects (9,749)
Python Server Projects (7,843)
Python Neural Projects (7,444)
Python Neural Network Projects (7,064)
Javascript Python Projects (6,958)
Python Deep Projects (6,552)
Python Google Projects (6,420)
Python Keras Projects (6,284)
Python Search Projects (5,943)
Python Nlp Projects (5,233)
Python Cnn Projects (5,079)
Python Face Projects (4,910)
Python Discord Projects (4,815)
Python Recognition Projects (4,789)
Python Telegram Projects (4,627)
Python Language Projects (4,480)
Python Data Science Projects (4,282)
Python Web Projects (4,272)
Python Pandas Projects (4,272)
Python Computer Vision Projects (4,236)
Python Password Projects (3,712)
Python Tutorial Projects (3,517)
Python Line Projects (3,471)
Python Tree Projects (3,254)
Python String Projects (3,238)
Python Security Projects (3,208)
Python Lstm Projects (3,120)
Python Book Projects (2,978)
Python Css Projects (2,892)
Python Programming Projects (2,778)
Python Cli Projects (2,579)
Python Raspberry Pi Projects (2,420)
Python Mnist Projects (2,390)
Python Artificial Intelligence Projects (2,382)
Python Block Projects (2,374)
Python Asyncio Projects (2,315)
Python Documentation Projects (2,205)
Python Aws Projects (2,184)
Python Natural Language Processing Projects (2,138)
Python Object Detection Projects (2,102)
Python Numpy Projects (2,087)
Python Command Line Projects (2,079)
Python Annotations Projects (2,047)
Python Rnn Projects (1,975)
Python Encrypt Projects (1,958)
Python Chinese Projects (1,898)
Python Java Projects (1,851)
Python Voice Projects (1,781)
Python Stack Projects (1,696)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,059,739)
Python  (807,156)
Java  (388,454)
Php  (285,984)
Typescript  (246,254)
C Plus Plus  (238,787)
Ruby  (221,835)
C  (182,003)
Shell  (171,198)
C Sharp  (164,103)
Golang  (158,248)
Swift  (65,213)
R  (57,905)
Rust  (57,566)
Objective C  (57,184)
Dart  (52,612)
Kotlin  (46,446)
Lua  (33,878)
Matlab  (31,150)
Perl  (30,540)
Language  (30,062)
Scala  (28,665)
Haskell  (20,368)
Clojure  (19,133)
Powershell  (19,013)
Bash  (18,678)
Coffeescript  (17,292)
Elixir  (16,655)
Assembly  (14,679)
Processing  (13,307)
Julia  (12,743)
Basic  (10,739)
Flow  (10,547)
Emacs Lisp  (10,417)
Erlang  (9,065)
Groovy  (8,446)
Scheme  (8,438)
Solidity  (7,954)
Ocaml  (6,703)
Pascal  (6,439)
Hack  (6,398)
Lisp  (6,341)
Fortran  (5,540)
Elm  (5,484)
Common Lisp  (5,298)
Dlang  (4,745)
Shell Script  (4,655)
Python Library  (4,455)
Crystal  (4,390)
Scripting  (4,207)
Top Projects
Freecodecamp ⭐ 348,159
996.icu ⭐ 262,763
Free Programming Books ⭐ 237,303
Coding Interview University ⭐ 222,665
Awesome ⭐ 206,565
Developer Roadmap ⭐ 198,217
Public Apis ⭐ 197,271
Vue ⭐ 197,223
React ⭐ 190,606
System Design Primer ⭐ 184,236
Tensorflow ⭐ 166,046
Bootstrap ⭐ 158,216
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 151,618
Ohmyzsh ⭐ 146,966
Build Your Own X ⭐ 145,201
Javascript Algorithms ⭐ 143,983
Flutter ⭐ 142,246
Python ⭐ 139,248
Gitignore ⭐ 134,582
Linux ⭐ 133,899
Vscode ⭐ 133,454
Awesome Python ⭐ 131,759
Javascript ⭐ 124,014
Javaguide ⭐ 123,971
Python 100 Days ⭐ 120,066
Computer Science ⭐ 118,096
Youtube Dl ⭐ 111,107
Fucking Algorithm ⭐ 107,431
The Art Of Command Line ⭐ 106,741
React Native ⭐ 103,527
Electron ⭐ 102,301
D3 ⭐ 101,735
Go ⭐ 101,131
30 Seconds Of Code ⭐ 96,728
Create React App ⭐ 95,709
Axios ⭐ 94,209
Free Programming Books Zh_cn ⭐ 93,295
Awesome Selfhosted ⭐ 92,838
Kubernetes ⭐ 89,722
Next.js ⭐ 88,761
Node ⭐ 88,478
Terminal ⭐ 83,789
Deno ⭐ 83,340
Three.js ⭐ 83,174
Awesome Go ⭐ 83,058
Angular ⭐ 82,342
Typescript ⭐ 81,850
Ant Design ⭐ 80,877
Material Ui ⭐ 79,382
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.