Awesome Open Source
Awesome Open Source
Combined Topics
python x
web-crawler x

The Top 1,235 Python Web Crawler Open Source Projects

Categories > Data Processing > Web Crawler
Awesome Web Scraping ⭐ 4,941
List of libraries, tools and APIs for web scraping and data processing.
most recent commit a month ago
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
total releases 15most recent commit a year ago
Selenium Python Helium ⭐ 2,890
Selenium-python but lighter: Helium is the best Python library for web automation.
dependent packages 9total releases 26most recent commit 3 days agopypi helium} Downloads
Automatic Udemy Course Enroller Get Paid Udemy Courses For Free ⭐ 2,637
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
total releases 7most recent commit 6 days ago
Grab ⭐ 2,174
Web Scraping Framework
dependent packages 4total releases 120most recent commit 3 months agopypi grab} Downloads
30 Days Of Python ⭐ 1,725
Learn Python for the next 30 (or so) Days.
most recent commit 7 months ago
Pspider ⭐ 1,617
简单易用的Python爬虫框架，QQ交流群：597510560
most recent commit 4 months ago
Dat8 ⭐ 1,439
General Assembly's 2015 Data Science course in Washington, DC
most recent commit 6 years ago
Snoop ⭐ 1,427
Snoop — инструмент разведки на основе открытых данных (OSINT world)
most recent commit 13 days ago
100projectsofcode ⭐ 1,293
A list of practical knowledge-building projects.
most recent commit a year ago
Pythoncode Tutorials ⭐ 1,093
The Python Code Tutorials
most recent commit 4 days ago
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
total releases 65most recent commit 5 months ago
Requests Cache ⭐ 943
Transparent persistent cache for python requests
most recent commit 5 days ago
Youtube_tutorials ⭐ 855
Collection of scripts corresponding to LucidProgramming YouTube tutorials
most recent commit 3 months ago
Faster Than Requests ⭐ 828
Faster requests on Python 3
dependent packages 4total releases 18most recent commit 2 days agopypi faster-than-requests} Downloads
Zhihu Spider ⭐ 719
A web spider for zhihu.com
most recent commit 7 years ago
Lightnovel Crawler ⭐ 635
Generate and download e-books from online sources.
dependent packages 1total releases 30most recent commit 11 days agopypi ebook-crawler} Downloads
Gazpacho ⭐ 543
🥫 The simple, fast, and modern web scraping library
dependent packages 5total releases 19most recent commit a year agopypi gazpacho} Downloads
Selectolax ⭐ 535
Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
dependent packages 17total releases 34most recent commit 2 months agopypi selectolax} Downloads
Trafilatura ⭐ 473
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
dependent packages 4total releases 29most recent commit 8 days agopypi trafilatura} Downloads
Scrapy Fake Useragent ⭐ 471
Random User-Agent middleware based on fake-useragent
dependent packages 1total releases 12most recent commit 2 years agopypi scrapy-fake-useragent} Downloads
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
total releases 10most recent commit 3 years ago
Crosslinked ⭐ 441
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
most recent commit 8 days ago
Facepager ⭐ 431
Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
most recent commit 13 days ago
Instascrape ⭐ 429
Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically
total releases 56most recent commit 3 months ago
Company Crawler ⭐ 386
天眼查爬虫&企查查爬虫，指定关键字爬取公司信息
most recent commit 2 months ago
Proxy_requests ⭐ 367
a class that uses scraped proxies to make http GET/POST requests (Python requests)
dependent packages 2total releases 49most recent commit a year agopypi proxy-requests} Downloads
Learn To Identify Similar Images ⭐ 358
Record my python script about Iearning to identify similar images
most recent commit 3 years ago
Basketball_reference_web_scraper ⭐ 300
NBA Stats API via Basketball Reference
total releases 38most recent commit 3 months ago
Dude ⭐ 298
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
total releases 30most recent commit a day ago
Scrape Linkedin Selenium ⭐ 288
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
most recent commit 9 months ago
Spidy ⭐ 287
The simple, easy to use command line web crawler.
total releases 12most recent commit a month ago
Web Scraping ⭐ 276
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
most recent commit 4 months ago
Youtube Projects ⭐ 272
This repository contains all the code I use in my YouTube tutorials.
most recent commit 8 months ago
Archivebot ⭐ 270
ArchiveBot, an IRC bot for archiving websites
most recent commit 3 days ago
Onlyfans Dl ⭐ 268
OnlyFans content downloader
most recent commit 4 months ago
City Scrapers ⭐ 253
Scrape, standardize and share public meetings from local government websites
most recent commit 2 months ago
Tradingview Data Scraper ⭐ 250
Extract price and indicator data from TradingView charts to create ML datasets
most recent commit 10 months ago
Lagoujob ⭐ 250
Job data mining repo for lagou.com
most recent commit 3 years ago
Summarizer ⭐ 236
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
most recent commit 3 months ago
Docbao ⭐ 233
Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam
most recent commit a year ago
Quora Api ⭐ 232
An unofficial API for Quora.
most recent commit 6 years ago
Kochat ⭐ 222
Opensource Korean chatbot framework
total releases 27most recent commit a year ago
Netflix Clone ⭐ 219
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
most recent commit 3 months ago
Wayback Machine Scraper ⭐ 219
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
total releases 8most recent commit a year ago
Short Jokes Dataset ⭐ 205
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
most recent commit 3 years ago
Espnff ⭐ 202
ESPN Fantasy Football API
total releases 11most recent commit 4 years ago
Portia Dashboard ⭐ 190
portia-dashboard is a visual web crawler based on scrapinghub/portia
most recent commit 4 years ago
Zhihu Crawler People ⭐ 179
A simple distributed crawler for zhihu && data analysis
most recent commit 3 years ago
Trump Lies ⭐ 175
Tutorial: Web scraping in Python with Beautiful Soup
most recent commit 4 years ago
Ignareo Isml Auto Voter ⭐ 175
Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns!
most recent commit 3 months ago
Twitter Intelligence ⭐ 169
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
most recent commit 2 years ago
First Web Scraper ⭐ 167
A step-by-step guide to writing a web scraper with Python
most recent commit a year ago
Hq_bot ⭐ 161
📲 Bot to help solve HQ trivia
most recent commit 3 years ago
Cocrawler ⭐ 159
CoCrawler is a versatile web crawler built using modern tools and concurrency.
total releases 8most recent commit a month ago
Bet On Sibyl ⭐ 157
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
most recent commit 5 years ago
Juno_crawler ⭐ 147
Scrapy crawler to collect data on the back catalog of songs listed for sale.
most recent commit 3 years ago
Web Database Analytics ⭐ 144
Web scrapping and related analytics using Python tools
most recent commit 2 years ago
Netpwn ⭐ 143
Tool made to automate tasks of pentesting.
most recent commit 3 years ago
Stardox ⭐ 143
Github stargazers information gathering tool
most recent commit 3 months ago
Scrapy Training ⭐ 141
Scrapy Training companion code
most recent commit 3 years ago
Learnpythonforresearch ⭐ 137
This repository provides everything you need to get started with Python for (social science) research.
most recent commit 2 years ago
Dark Fantasy Hack Tool ⭐ 129
DDOS Tool: To take down small websites with HTTP FLOOD. Port scanner: To know the open ports of a site. FTP Password Cracker: To hack file system of websites.. Banner Grabber: To get the service or software running on a port. (After knowing the software running google for its vulnerabilities.) Web Spider: For gathering web application hacking information. Email scraper: To get all emails related to a webpage IMDB Rating: Easy way to access the movie database. Both .exe(compressed as zip) and .py
most recent commit a year ago
Zillow ⭐ 129
Zillow Scraper for Python using Selenium
most recent commit 3 years ago
Social Media Profile Scrapers ⭐ 129
Fetch user's data across social media
most recent commit 2 months ago
Scrapers ⭐ 128
Lots and lots of web scrapers
most recent commit 5 years ago
Fisherman ⭐ 117
CLI program that collects information from facebook user profiles via Selenium.
most recent commit a month ago
Geeksforgeeksscrapper ⭐ 116
Scrapes g4g and creates PDF
most recent commit 2 years ago
Not Your Average Web Crawler ⭐ 116
A web crawler (for bug hunting) that gathers more than you can imagine.
total releases 36most recent commit 7 months ago
Crawlbox ⭐ 112
Easy way to brute-force web directory.
most recent commit 3 years ago
Tiktokbot ⭐ 109
A TikTokBot that downloads trending tiktok videos and compiles them using FFmpeg
most recent commit a year ago
Hockey Scraper ⭐ 107
Python Package for scraping NHL Play-by-Play and Shift data
total releases 52most recent commit 2 months ago
Seleniumcrawler ⭐ 105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
most recent commit 3 years ago
Competitive_programming_score_api ⭐ 102
API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit
most recent commit 7 months ago
Bancocentralbrasil ⭐ 102
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
total releases 3most recent commit 6 months ago
Zhihu_crawler ⭐ 100
a crawler for zhihu
most recent commit 5 years ago
Cowin Vaccine Notifier ⭐ 97
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
most recent commit a year ago
Wereadscan ⭐ 96
扫描“微信读书”已购图书并下载本地PDF的爬虫
total releases 7most recent commit 9 months ago
Wswp ⭐ 95
Code for the second edition Web Scraping with Python book by Packt Publications
most recent commit 3 years ago
Brutescrape ⭐ 95
A web scraper for generating password files based on plain text found
most recent commit 3 years ago
Ir ⭐ 93
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
most recent commit 10 months ago
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 2 years ago
Reader ⭐ 90
Extract clean(er), readable text from web pages via Mercury Web Parser.
most recent commit 21 days ago
Saveddit ⭐ 90
Bulk Downloader for Reddit
total releases 15most recent commit 7 months ago
Udemy_bot ⭐ 88
An automation bot for free Udemy courses
most recent commit a year ago
Webcrawler ⭐ 86
Web crawler to download pictures from zhihu.com
most recent commit 3 years ago
Letterboxd_recommendations ⭐ 84
Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username
most recent commit 2 months ago
Zimit ⭐ 83
Make a ZIM file from any Web site and surf offline!
most recent commit 23 days ago
Ebaymarketanalyzer ⭐ 83
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
most recent commit 9 months ago
Openscraper ⭐ 80
An open source webapp for scraping: towards a public service for webscraping
most recent commit 3 months ago
Tableau Scraping ⭐ 79
Tableau scraper python library. R and Python scripts to scrape data from Tableau viz
dependent packages 1total releases 40most recent commit 4 months agopypi TableauScraper} Downloads
Awesomewebscraping ⭐ 77
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 2 months ago
Python Web Scraping Second Edition ⭐ 76
Python Web Scraping Second Edition, published by Packt
most recent commit a year ago
Web Scraping ⭐ 76
Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!
most recent commit 3 years ago
Pymarketcap ⭐ 74
Python3 API wrapper and web scraper for https://coinmarketcap.com
total releases 92most recent commit 3 years ago
Raspagem De Dados Para Iniciantes ⭐ 74
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
most recent commit 2 years ago
Scraping Ebay ⭐ 73
Scraping Ebay's products using Scrapy Web Crawling Framework
most recent commit 3 months ago
Leetcode Compensation ⭐ 72
Compensation analysis on the posts scraped from leetcode.com/discuss/compensation. At present, the reports have been generated only for Indian cities.
most recent commit 3 months ago
Bancocentralbrasil ⭐ 71
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
most recent commit 2 years ago
Tspider ⭐ 71
Yet Another Web Spider
most recent commit 5 years ago
1-100 of 1,235 projects
Related Awesome Lists
Python Django Projects (26,307)
Python Dataset Projects (14,792)
Python Flask Projects (14,408)
Python Machine Learning Projects (14,100)
Python Docker Projects (13,757)
Python Tensorflow Projects (13,736)
Python Deep Learning Projects (13,093)
Python Jupyter Notebook Projects (12,976)
Python Network Projects (11,547)
Python Learning Projects (10,661)
Python Database Projects (9,975)
Python Algorithms Projects (9,749)
Python Pytorch Projects (7,877)
Python Server Projects (7,843)
Python Neural Projects (7,444)
Python Neural Network Projects (7,064)
Javascript Python Projects (6,958)
Python Google Projects (6,420)
Python Keras Projects (6,284)
Python Search Projects (5,943)
Python Natural Language Processing Projects (5,233)
Python Convolutional Neural Networks Projects (5,079)
Python Face Projects (4,910)
Python Discord Projects (4,815)
Python Recognition Projects (4,789)
Python Telegram Projects (4,627)
Python Language Projects (4,480)
Python Data Science Projects (4,282)
Python Web Projects (4,272)
Python Pandas Projects (4,272)
Python Computer Vision Projects (4,236)
Python Password Projects (3,712)
Python Robot Projects (3,624)
Python Tutorials Projects (3,517)
Python Line Projects (3,471)
Python Tree Projects (3,254)
Python String Projects (3,238)
Python Security Projects (3,208)
Python Lstm Projects (3,120)
Python Book Projects (2,978)
Python Css Projects (2,892)
Python Programming Projects (2,778)
Python Command Line Projects (2,579)
Python Raspberry Pi Projects (2,420)
Python Mnist Projects (2,390)
Python Artificial Intelligence Projects (2,382)
Python Block Projects (2,374)
Python Asyncio Projects (2,315)
Python Documentation Projects (2,205)
Python Amazon Web Services Projects (2,184)
Python Object Detection Projects (2,102)
Python Numpy Projects (2,087)
Python Recurrent Neural Networks Projects (1,975)
Python Encryption Projects (1,958)
Python Chinese Projects (1,898)
Python Java Projects (1,851)
Python Voice Projects (1,781)
Python Stack Projects (1,696)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,048,735)
Python  (796,103)
Java  (385,413)
Php  (284,589)
Typescript  (239,722)
C Plus Plus  (236,213)
Ruby  (221,696)
C  (180,396)
Shell  (169,594)
C Sharp  (162,015)
Golang  (156,089)
Swift  (64,698)
R  (57,248)
Objective C  (57,169)
Rust  (56,117)
Dart  (51,466)
Kotlin  (45,511)
Lua  (33,293)
Matlab  (30,891)
Perl  (30,483)
Language  (30,085)
Scala  (28,601)
Haskell  (20,260)
Clojure  (19,072)
Powershell  (18,797)
Bash  (18,606)
Coffeescript  (17,331)
Elixir  (16,565)
Assembly  (14,526)
Processing  (13,298)
Julia  (12,536)
Basic  (10,765)
Flow  (10,528)
Emacs Lisp  (10,368)
Erlang  (9,058)
Scheme  (8,437)
Groovy  (8,431)
Solidity  (7,611)
Ocaml  (6,668)
Pascal  (6,378)
Hack  (6,365)
Lisp  (6,340)
Fortran  (5,483)
Elm  (5,477)
Common Lisp  (5,277)
Dlang  (4,747)
Shell Script  (4,633)
Python Library  (4,446)
Crystal  (4,374)
Scripting  (4,203)
Top Projects
Freecodecamp ⭐ 346,432
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,678
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,578
Vue ⭐ 196,186
Developer Roadmap ⭐ 195,398
Public Apis ⭐ 193,925
React ⭐ 188,417
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,144
Bootstrap ⭐ 157,483
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,459
Build Your Own X ⭐ 142,246
Flutter ⭐ 140,926
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,303
Gitignore ⭐ 133,450
Linux ⭐ 132,199
Vscode ⭐ 132,056
Awesome Python ⭐ 127,528
Javascript ⭐ 123,347
Javaguide ⭐ 121,667
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,239
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,862
Electron ⭐ 101,813
D3 ⭐ 100,901
Go ⭐ 99,493
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,174
Axios ⭐ 93,562
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,689
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,823
Next.js ⭐ 87,114
Terminal ⭐ 83,251
Deno ⭐ 82,597
Three.js ⭐ 82,214
Angular ⭐ 81,516
Awesome Go ⭐ 81,089
Typescript ⭐ 80,876
Ant Design ⭐ 80,416
Material Ui ⭐ 78,638
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.