Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for python web crawler
python
x
web-crawler
x
117 search results found
Scrapy
⭐
49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
Changedetection.io
⭐
13,943
The best and simplest free open source website change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
Autoscraper
⭐
5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Douyin_tiktok_download_api
⭐
4,844
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T
Helium
⭐
3,975
Lighter web automation for Python
Automatic Udemy Course Enroller Get Paid Udemy Courses For Free
⭐
3,010
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
Snoop
⭐
2,530
Snoop — инструмент разведки на основе открытых данных (OSINT world)
Trafilatura
⭐
2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
Grab
⭐
2,292
Web Scraping Framework
30 Days Of Python
⭐
1,926
Learn Python for the next 30 (or so) Days.
Pythoncode Tutorials
⭐
1,923
The Python Code Tutorials
Pspider
⭐
1,675
简单易用的Python爬虫框架,QQ交流群:597510560
Dat8
⭐
1,549
General Assembly's 2015 Data Science course in Washington, DC
Webscraping From 0 To Hero
⭐
1,305
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
100projectsofcode
⭐
1,293
A list of practical knowledge-building projects.
Scrapeghost
⭐
1,283
👻 Experimental library for scraping websites using OpenAI's GPT API.
Requests Cache
⭐
1,208
Persistent HTTP cache for python requests
Lightnovel Crawler
⭐
1,185
Generate and download e-books from online sources.
Django Dynamic Scraper
⭐
1,069
Creating Scrapy scrapers via the Django admin interface
Faster Than Requests
⭐
1,061
Faster requests on Python 3
Crosslinked
⭐
1,060
LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping
Curl_cffi
⭐
987
Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
Selectolax
⭐
921
Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
Youtube_tutorials
⭐
889
Collection of scripts corresponding to LucidProgramming YouTube tutorials
Zhihu Spider
⭐
719
A web spider for zhihu.com
Gazpacho
⭐
716
🥫 The simple, fast, and modern web scraping library
Scrapy Fake Useragent
⭐
654
Random User-Agent middleware based on fake-useragent
Instascrape
⭐
554
Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically
Complete Life Cycle Of A Data Science Project
⭐
499
Complete-Life-Cycle-of-a-Data-Science-Project
Jekyll
⭐
498
Jekyll-based static site for The Programming Historian
Facepager
⭐
490
Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
Company Crawler
⭐
466
天眼查爬虫&企查查爬虫,指定关键字爬取公司信息
Scrapple
⭐
452
A framework for creating semi-automatic web content extractors
Wereadscan
⭐
447
扫描“微信读书”已购图书并下载本地PDF的爬虫
Google Search Results Python
⭐
432
Google Search Results via SERP API pip Python Package
Dude
⭐
397
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
Kochat
⭐
383
Opensource Korean chatbot framework
Basketball_reference_web_scraper
⭐
382
NBA Stats API via Basketball Reference
Proxy_requests
⭐
381
a class that uses scraped proxies to make http GET/POST requests (Python requests)
Tarsier
⭐
372
Vision utilities for web interaction agents 👀
Learn To Identify Similar Images
⭐
358
Record my python script about Iearning to identify similar images
Http Proxy List
⭐
355
It is a lightweight project that, every 10 minutes, scrapes lots of free-proxy sites, validates if it works, and serves a clean proxy list.
Scrape Linkedin Selenium
⭐
353
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Archivebot
⭐
328
ArchiveBot, an IRC bot for archiving websites
Social Media Profile Scrapers
⭐
322
Fetch user's data across social media
City Scrapers
⭐
315
Scrape, standardize and share public meetings from local government websites
Mov Cli
⭐
314
A cli tool to browse and watch Movies/Shows/TV/Sports.
Uscrapper
⭐
298
Uscrapper 2.0, a powerful OSINT webscraper for personal data collection. Uscrapper uses web scraping to extract email IDs, social-media links, geolocations, phone numbers, and usernames from webpages, supports multithreading, has advanced Anti-webscraping bypassing modules, supports webcrawling to scrape from various sublinks within the same domain
Spidy
⭐
287
The simple, easy to use command line web crawler.
Web Scraping
⭐
281
Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup
Web Scraping
⭐
276
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
Youtube Projects
⭐
272
This repository contains all the code I use in my YouTube tutorials.
Tiktokbot
⭐
262
A TikTokBot that downloads trending tiktok videos and compiles them using FFmpeg
H4x Tools
⭐
250
Open source toolkit for scraping, OSINT and more.
Lagoujob
⭐
250
Job data mining repo for lagou.com
Tradingview Data Scraper
⭐
250
Extract price and indicator data from TradingView charts to create ML datasets
Dark Fantasy Hack Tool
⭐
248
DDOS Tool: To take down small websites with HTTP FLOOD. Port scanner: To know the open ports of a site. FTP Password Cracker: To hack file system of websites.. Banner Grabber: To get the service or software running on a port. (After knowing the software running google for its vulnerabilities.) Web Spider: For gathering web application hacking information. Email scraper: To get all emails related to a webpage IMDB Rating: Easy way to access the movie database. Both .exe(compressed as zip) and .py
Easyapplyjobsbot
⭐
247
A python bot to automatically apply all Linkedin,Glassdoor, etc Easy Apply jobs based on your preferences. Auto login, auto fill additional questions, apply automatically!
Netflix Clone
⭐
245
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
Summarizer
⭐
236
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
Docbao
⭐
233
Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam
Quora Api
⭐
232
An unofficial API for Quora.
Nudecrawler
⭐
231
Crawl telegra.ph searching for nudes!
Fb_friend_list_scraper
⭐
221
OSINT tool to scrape names and usernames from large friend lists on Facebook, without being rate limited.
Amazon Scraper
⭐
219
A simple web scraper to extract Product Data and Pricing from Amazon
Wayback Machine Scraper
⭐
219
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Zimit
⭐
209
Make a ZIM file from any Web site and surf offline!
Twitter Scraper Selenium
⭐
207
Python's package to scrap Twitter's front-end easily
Short Jokes Dataset
⭐
205
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
Espnff
⭐
202
ESPN Fantasy Football API
Everything Web Scraping
⭐
197
Learn everything web scraping with David Teather Codes on YouTube
Weboptout
⭐
191
Opt-Out tool to check Copyright reservations in a way that even machines can understand.
Portia Dashboard
⭐
190
portia-dashboard is a visual web crawler based on scrapinghub/portia
Letterboxd_recommendations
⭐
190
Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username
Awesomewebscraping
⭐
189
List of libraries, tools and APIs for web scraping and data processing.
Nba Prediction
⭐
186
A project to deploy an online app that predicts the win probability for each NBA game every day. Demonstrates end-to-end Machine Learning deployment.
Ignareo Isml Auto Voter
⭐
186
Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)
Daath Ai Parser
⭐
184
Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
Zhihu Crawler People
⭐
179
A simple distributed crawler for zhihu && data analysis
Musicer
⭐
176
旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体
Trump Lies
⭐
175
Tutorial: Web scraping in Python with Beautiful Soup
Twitter Intelligence
⭐
169
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
Crawler_shopee_public
⭐
169
蝦皮非同步爬蟲 + 競品賣家分析
Stardox
⭐
165
Github stargazers information gathering tool
Hq_bot
⭐
161
📲 Bot to help solve HQ trivia
Cocrawler
⭐
159
CoCrawler is a versatile web crawler built using modern tools and concurrency.
Bet On Sibyl
⭐
157
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
Ir
⭐
155
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
Facebook_page_scraper
⭐
150
Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV
Blackmaria
⭐
148
Python package for webscraping in Natural language
Scrape Up
⭐
148
A web-scraping-based python package that enables you to scrape data from various platforms like GitHub, Twitter, Instagram, or any useful website.
Juno_crawler
⭐
147
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Web Database Analytics
⭐
144
Web scrapping and related analytics using Python tools
Saveddit
⭐
143
Bulk Downloader for Reddit
Estela
⭐
142
estela, an elastic web scraping cluster 🕸
Scrapy Training
⭐
141
Scrapy Training companion code
Amazon Scraper
⭐
140
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
Learnpythonforresearch
⭐
137
This repository provides everything you need to get started with Python for (social science) research.
Hockey Scraper
⭐
134
Python Package for scraping NHL Play-by-Play and Shift data
Competitive_programming_score_api
⭐
134
API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit
Related Searches
Python Django (26,307)
Python Machine Learning (20,195)
Python Deep Learning (19,382)
Python Jupyter Notebook (18,308)
Python Dataset (14,792)
Python Flask (14,408)
Python Docker (14,113)
Python Tensorflow (13,736)
Python Command Line (13,351)
Python Network (11,646)
1-100 of 117 search results
Next >
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.