Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for python web crawler
python
x
web-crawler
x
117 search results found
Not Your Average Web Crawler
⭐
130
A web crawler (for bug hunting) that gathers more than you can imagine.
Zillow
⭐
129
Zillow Scraper for Python using Selenium
Scrapers
⭐
128
Lots and lots of web scrapers
Clicknium Docs
⭐
127
A next-generation GUI automation framework for Web and Desktop Application Testing and Automation.
Ospider
⭐
124
开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)
Extractnet
⭐
118
A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package
Algolisted
⭐
118
Algolisted is an AI-powered nonprofit analytics firm dedicated to assisting computer science students in preparing for placements and internships. Our services include tracking and analytics across various platforms and topics.
Educative.io_scraper
⭐
117
Educative.io Course Downloader developed using Python and Selenium. Refer Readme.md for setup instructions.
Geeksforgeeksscrapper
⭐
116
Scrapes g4g and creates PDF
Raspagem De Dados Para Iniciantes
⭐
115
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
Homeharvest
⭐
114
Python package for real estate scraping of MLS listing data
Leetcode Compensation
⭐
112
Compensation analysis of leetcode.com/discuss/compensation.
Crawlbox
⭐
112
Easy way to brute-force web directory.
Collector
⭐
111
Collector is a OSINT tool and information gathering. This tool can do information gathering phone numbers, github account, ip address and instagram account.
Gflare Tk
⭐
110
Open-Source Python Based SEO Web Crawler
Seleniumcrawler
⭐
105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
Scraper
⭐
104
Web scraper for scraping, tracking and visualizing prices of products on various websites.
Htmldate
⭐
101
Fast and robust date extraction from web pages, with Python or on the command-line
Zhihu_crawler
⭐
100
a crawler for zhihu
Reader
⭐
100
Extract clean(er), readable text from web pages via Mercury Web Parser.
Facebook Marketplace Scraper
⭐
99
This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.
Cowin Vaccine Notifier
⭐
97
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
Senpwai
⭐
97
A desktop app for tracking and batch downloading anime
Wswp
⭐
95
Code for the second edition Web Scraping with Python book by Packt Publications
Brutescrape
⭐
95
A web scraper for generating password files based on plain text found
Nba_betting
⭐
93
Using data analytics and machine learning to create a comprehensive and profitable system for predicting the outcomes of NBA games.
Terpene Profile Parser For Cannabis Strains
⭐
93
Parser and database to index the terpene profile of different strains of Cannabis from online databases
Python Sec
⭐
93
A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.
Scrapy Gui
⭐
93
A simple, Qt-Webengine powered web browser with built in functionality for basic scrapy webscraping support.
Rymscraper
⭐
92
Python API to extract data from rateyourmusic.com.
Scrapyd Cluster On Heroku
⭐
90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
Amacapy Bot Telegram Amazon Affiliates
⭐
90
Amacapy is a software that does web scraping to the Amazon website and publishes them on Telegram, searches the products by the keyword entered or the direct link of the product. Then you can publish these products on Telegram in a certain time. The technologies used were Flet, Beautiful Soup and Python.
Udemy_bot
⭐
88
An automation bot for free Udemy courses
Email Crawler Lead Generator
⭐
88
This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.
Webcrawler
⭐
86
Web crawler to download pictures from zhihu.com
Nordvpn Switcher
⭐
86
Rotate between different NordVPN servers with ease. Works both on Linux and Windows without any required changes to your code!
Web Poet
⭐
85
Web scraping Page Objects core library
Transfermarkt Api
⭐
84
API service to get data from Transfermarkt
Instatools
⭐
83
🧰 A collection of tools built for automating tasks on Instagram.
Ebaymarketanalyzer
⭐
83
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
Cianparser
⭐
83
Parser general information on the cian.ru website / Сбор данных с сайта объявлений Циан
Lisc
⭐
81
Literature Scanner: Automated collection & analyses of the scientific literature.
Openscraper
⭐
80
An open source webapp for scraping: towards a public service for webscraping
Tableau Scraping
⭐
79
Tableau scraper python library. R and Python scripts to scrape data from Tableau viz
Webscrapper
⭐
77
Simple and powerfull all in one Telegram Bot to scrap webpages using Requests, html5lib and Beautifulsoup
Python Web Scraping Second Edition
⭐
76
Python Web Scraping Second Edition, published by Packt
Goodreadsscraper
⭐
76
Scrape data from Goodreads using Scrapy and Selenium 📚
Web Scraping
⭐
76
Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!
Scrapfly Scrapers
⭐
76
Web scrapers for popular targets powered Scrapfly.io
Pymarketcap
⭐
74
Python3 API wrapper and web scraper for https://coinmarketcap.com
Bridgekeeper
⭐
73
Scrape, Hunt, and Transform names and usernames
Requests Random User Agent
⭐
73
Configures the requests library to randomly select a desktop User-Agent
Scraping Ebay
⭐
73
Scraping Ebay's products using Scrapy Web Crawling Framework
Spotifyscraper
⭐
72
Spotify Scraper to extract all the information from spotify, download mp3 with cover of the song
Bancocentralbrasil
⭐
71
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
Tspider
⭐
71
Yet Another Web Spider
Brokenlinkhijacker
⭐
71
A Fast Broken Link Hijacker Tool written in Python
Davedavefind
⭐
71
A simple search engine based on the web crawler developed in Udacity's CS101 course.
Scrapy Wayback Machine
⭐
70
A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Webkitcrawler
⭐
69
QtWebKit-based web crawler
Argus
⭐
67
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-0
Web Scraping
⭐
67
Web Scraping with Beautiful Soup and Selenium
Top Github Scraper
⭐
67
Scape top GitHub repositories and users based on keywords
Data Wrangling With Python
⭐
66
Simplify your ETL processes with these hands-on data sanitation tips, tricks, and best practices
Schweizermesser
⭐
66
🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
Cvpr2019
⭐
65
Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.
Phd Seeker
⭐
64
Finding latest fully funded PhD positions for international students through web scraping
Ping Sm
⭐
64
🍎 Receive an email or Telegram message as soon as Migros Sanalmarket is available for delivery in your neighborhood.
Malaysianpaygap
⭐
63
Scrapping malaysianpaygap & Extracting data from the Instagram posts
Covid_19_jhu_data_web_scrap_and_cleaning
⭐
61
This repository contains data and code used to get and clean data from https://github.com/CSSEGISandData/COVID-19 and https://www.worldometers.info/coronavirus/
Kenpompy
⭐
61
A simple yet comprehensive web scraper for kenpom.com.
Leek
⭐
61
Distributed task redisqueue(最简单python分布式函数调度框架)
Instagram Giveaways Winner
⭐
59
Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!
Keyword_based_sina_weibo_crawler
⭐
59
A web crawler for Sina, search and retrieve microblogs that contain certain keywords 一个简单的python爬虫实践,爬取包含关键词的新浪微博
Qtsapp
⭐
59
The Python Library For QtsApp which displays the option chain in near real-time. This program retrieves this data from the QtsApp site and then generates useful analysis of the Option Chain for the specified Index or Stock. It also continuously refreshes the Option Chain along with Implied Volatatlity (IV), Open Interest (OI), Delta, Theta, Vega, Gamma, Vanna, Charm, Speed, Zomma, Color, Volga, Veta at an interval of a second and visually displays the trend in various indicators useful for Techn
Song Cli
⭐
58
A command line interface for downloading Bollywood and punjabi songs
Scraping Tripadvisor With Python 2020
⭐
58
Python implementation of web scraping of TripAdvisor with Selenium in a new 2019 website
Nba Search
⭐
56
flask application designed to explore NBA data 🏀
Searchifyx
⭐
56
Fast flashcard searcher study tool
Tripadvisor Scraper
⭐
56
The basics of forming an input code for scraping travel industry pages with Tripadvisor Scraper API + an example of results.
Talospider
⭐
55
talospider - A simple,lightweight scraping micro-framework
Selectorlib
⭐
55
A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them
Slurp
⭐
54
One repo to rule them all !!?!?!! 🤓 😎
Pythonscrapybasicsetup
⭐
54
Basic setup with random user agents and IP addresses for Python Scrapy Framework.
Mobile Phone Dataset Gsmarena
⭐
53
Python script for creating Mobile Phones Dataset on GSMArena website.
Animex V2
⭐
52
animeX is a CLI tool for downloading anime directly to your PC
Llm_osint
⭐
52
LLM OSINT is a proof-of-concept method of using LLMs to gather information from the internet and then perform a task with this information.
Web_check
⭐
52
Script for checking changes in webpages
Linkedin Profiles Scraping
⭐
51
Automatically scrape the web data of people profiles on Linkedin based on a specific search query
Dashboard
⭐
50
A tkinter GUI collating various data
Comp_thinking_social_science
⭐
50
Computational Thinking for Social Scientists book project
Datadoubleconfirm
⭐
50
Simple datasets and notebooks for data visualization, statistical analysis and modelling - with write-ups here: http://projectosyo.wix.com/datadoubleconfirm.
Hk0weather
⭐
49
Web scraper project to collect the useful Hong Kong weather data from HKO website
Ds Ml Public
⭐
49
Python Scripts and Jupyter Notebooks
Pysearch
⭐
48
Web crawler and Search engine in Python.
Instagram Bot
⭐
48
🤖 Python bot to view stories, like and comment on Instagram
Python Libzim
⭐
48
Libzim binding for Python: read/write ZIM files in Python
Scrapy Craigslist
⭐
47
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
Python Assistant
⭐
47
Python Assistant (PA) is a voice command based assistant service written in Python 3.9+. It can recognize human speech or voice, talk to user and execute basic commands.
Bookingscraper
⭐
47
🌎 🏨 Scrape Booking.com 🏨 🌎
Related Searches
Python Django (26,307)
Python Machine Learning (20,195)
Python Deep Learning (19,382)
Python Jupyter Notebook (18,308)
Python Dataset (14,792)
Python Flask (14,408)
Python Docker (14,113)
Python Tensorflow (13,736)
Python Command Line (13,351)
Python Network (11,646)
101-117 of 117 search results
< Previous
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.