Awesome Open Source

Programming Languages

Search results for python webscraper

103 search results found

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

100projectsofcode ⭐ 1,293

A list of practical knowledge-building projects.

Lightnovel Crawler ⭐ 1,185

Generate and download e-books from online sources.

Faster Than Requests ⭐ 1,061

Faster requests on Python 3

Scrapple ⭐ 452

A framework for creating semi-automatic web content extractors

Wereadscan ⭐ 447

扫描“微信读书”已购图书并下载本地PDF的爬虫

Opensource Korean chatbot framework

Basketball_reference_web_scraper ⭐ 382

NBA Stats API via Basketball Reference

Proxy_requests ⭐ 381

a class that uses scraped proxies to make http GET/POST requests (Python requests)

Scrape Linkedin Selenium ⭐ 353

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Archivebot ⭐ 328

ArchiveBot, an IRC bot for archiving websites

Social Media Profile Scrapers ⭐ 322

Fetch user's data across social media

The simple, easy to use command line web crawler.

Web Scraping ⭐ 276

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

Lagoujob ⭐ 250

Job data mining repo for lagou.com

Summarizer ⭐ 236

A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

Amazon Scraper ⭐ 219

A simple web scraper to extract Product Data and Pricing from Amazon

Rightmove_webscraper.py ⭐ 219

Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object

Portia Dashboard ⭐ 190

portia-dashboard is a visual web crawler based on scrapinghub/portia

Ignareo Isml Auto Voter ⭐ 186

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

Daath Ai Parser ⭐ 184

Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

Zhihu Crawler People ⭐ 179

A simple distributed crawler for zhihu && data analysis

Musicer ⭐ 176

旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

Crawler_shopee_public ⭐ 169

蝦皮非同步爬蟲 + 競品賣家分析

Cocrawler ⭐ 159

CoCrawler is a versatile web crawler built using modern tools and concurrency.

Facebook_page_scraper ⭐ 150

Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

Not Your Average Web Crawler ⭐ 130

A web crawler (for bug hunting) that gathers more than you can imagine.

Ospider ⭐ 124

开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)

Geeksforgeeksscrapper ⭐ 116

Scrapes g4g and creates PDF

Raspagem De Dados Para Iniciantes ⭐ 115

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

Crawlbox ⭐ 112

Easy way to brute-force web directory.

Gflare Tk ⭐ 110

Open-Source Python Based SEO Web Crawler

Seleniumcrawler ⭐ 105

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Zhihu_crawler ⭐ 100

a crawler for zhihu

Facebook Marketplace Scraper ⭐ 99

This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.

A desktop app for tracking and batch downloading anime

Cowin Vaccine Notifier ⭐ 97

Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.

Brutescrape ⭐ 95

A web scraper for generating password files based on plain text found

Python Sec ⭐ 93

A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.

Terpene Profile Parser For Cannabis Strains ⭐ 93

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Webcrawler ⭐ 86

Web crawler to download pictures from zhihu.com

A Python package for scraping and downloading subtitles from AppleTV / iTunes movie pages.

Goodreadsscraper ⭐ 76

Scrape data from Goodreads using Scrapy and Selenium 📚

Pymarketcap ⭐ 74

Python3 API wrapper and web scraper for https://coinmarketcap.com

Shutterscrape ⭐ 73

Speedy, lightweight web scrapper for Shutterstock.

Spotifyscraper ⭐ 72

Spotify Scraper to extract all the information from spotify, download mp3 with cover of the song

Davedavefind ⭐ 71

A simple search engine based on the web crawler developed in Udacity's CS101 course.

Webkitcrawler ⭐ 69

QtWebKit-based web crawler

Top Github Scraper ⭐ 67

Scape top GitHub repositories and users based on keywords

Schweizermesser ⭐ 66

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

Cvpr2019 ⭐ 65

Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

Distributed task redisqueue(最简单python分布式函数调度框架)

Kenpompy ⭐ 61

A simple yet comprehensive web scraper for kenpom.com.

Wuxiaworld 2 Ebook ⭐ 59

This Python script will download chapters from novels availaible on wuxiaworld.com saves then into the .epub format

Instagram Giveaways Winner ⭐ 59

Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!

Keyword_based_sina_weibo_crawler ⭐ 59

A web crawler for Sina, search and retrieve microblogs that contain certain keywords 一个简单的python爬虫实践，爬取包含关键词的新浪微博

Song Cli ⭐ 58

A command line interface for downloading Bollywood and punjabi songs

Scraping Tripadvisor With Python 2020 ⭐ 58

Python implementation of web scraping of TripAdvisor with Selenium in a new 2019 website

Searchifyx ⭐ 56

Fast flashcard searcher study tool

Scraperx ⭐ 53

Library for scraping websites or apis at any scale

Linkedin Profiles Scraping ⭐ 51

Automatically scrape the web data of people profiles on Linkedin based on a specific search query

Dashboard ⭐ 50

A tkinter GUI collating various data

Pysearch ⭐ 48

Web crawler and Search engine in Python.

Bookingscraper ⭐ 47

🌎 🏨 Scrape Booking.com 🏨 🌎

Webcollector Python ⭐ 47

WebCollector-Python is an open source web crawler framework based on Python.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

Scrapy Craigslist ⭐ 47

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Ipfs Arxiv ⭐ 47

Machine Learning papers from arXiv hosted on IPFS website.

Trscraper ⭐ 47

TRScraper, doğal dil işleme uygulamalarında kullanılmak amacıyla geliştirilmiş, Türkçe içerik girilen büyük platformlarda metin madenciliği yapma imkanı sunan bir uygulamadır.

Lead Generation ⭐ 46

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

Uoft Scrapers ⭐ 44

Public web scraping scripts for the University of Toronto.

Price Comparison Project ⭐ 43

A webscraper for the Django Framework that compares the product prices for various UK supermarkets

Yellowpages Scraper ⭐ 43

Yellowpages.com Web Scraper written in Python and LXML to extract business details available based on a particular category and location.

Stockforecast ⭐ 42

🎯 predict the price trend of individual stocks using deep learning and natural language processing

Ajax_crawler ⭐ 41

A flexible web crawler based on Scrapy for fetching most of Ajax or other various types of web pages. Easy to use: To customize a new web crawler-You just need to write a config file and run.

Dead simple web crawler for Python

Learncpp Download ⭐ 39

An advanced web scraper tool that seamlessly fetches and combines over 200 online tutorials into a convenient offline PDF format.

wsvuls - website vulnerability scanner detect issues [ outdated server software and insecure HTTP headers.]

a web crawler and data analysis repo with Python3.5, R, Excel 2016 and TAGUL

Financial Web Scraper & Sentiment Classifier

Web Unblocker ⭐ 36

Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems.

Python Marmiton ⭐ 35

Python API to search & get recipes from the 'marmiton.com' website (web crawler, unofficial)

Cobweb Lnx ⭐ 34

CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

A lightweight API for Pornhub

Animal Crossing Scraper ⭐ 33

Web scraper for Animal Crossing - New Horizons data using bs4

Market Trend Prediction ⭐ 32

This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).

Hostpanic ⭐ 31

Find host header injections and perform Host Header attacks with other kind of bugs like web cache poissoning

Gcf Packs ⭐ 30

Library packs for google cloud functions

Videorecognition Realtime Autotrainer Alerts ⭐ 30

State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.

快速登录大连理工大学统一身份认证系统（SSO）的Python模块，可轻松实现成绩提醒、抢课、玉兰卡信

Linkedin Web Scraper ⭐ 28

Python Web Scraper for LinkedIn to collect and store company data (e.g. name, description, industry, etc.) into .xls file

Jobs_linkedin ⭐ 28

Finds Jobs on LinkedIn using web-scraping

Craigslistscraper ⭐ 27

Simple webscraper for Craigslist.

Email Report ⭐ 27

A modular template for scraping data from the web to send yourself scheduled email reports

Web Scraper Nabidek Pronajmu ⭐ 27

Nástroj pro hlídání nových nabídek nemovitostí na populárních realitních serverech. Nabídky jsou vypisovány do Discord roomky.

Pcpartpicker ⭐ 26

This is an unofficial API for the website pcpartpicker.com.

Hipposcraper ⭐ 26

A Linux terminal tool for parsing and scraping Holberton project pages to automate repetitive tasks.

Comicbookmaker ⭐ 26

Script to fetch webcomics and use them to create ebooks.

Webcrawler ⭐ 25

A web crawler based on requests-html, mainly targets for url validation test.

Scrapy Bench ⭐ 25

A CLI for benchmarking Scrapy.

Related Searches

Python Django (28,897)

Python Machine Learning (20,195)

Python Deep Learning (19,382)

Python Flask (17,643)

Python Jupyter Notebook (16,821)

Python Dataset (14,792)

Python Docker (13,758)

Python Tensorflow (13,736)

Python Command Line (13,351)

Python Network (11,495)

1-100 of 103 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.