Awesome Open Source

Programming Languages

Search results for scraping websites

scraping-websites x

167 search results found

Ferret ⭐ 5,540

Declarative web scraping

Cloudflare Scrape ⭐ 3,229

A Python module to bypass Cloudflare's anti-bot page.

Crawly, a high-level web crawling & scraping framework for Elixir.

Edu Mail Generator ⭐ 707

Generate Free Edu Mail(s) within minutes

Python_and_the_web ⭐ 662

Build Bots, Scrape a website or use an API to solve a problem.

译文：Puppeteer 与 Chrome Headless —— 从入门到爬虫

Sarenka ⭐ 533

OSINT tool - gets data from services like shodan, censys etc. in one app

Phpscraper ⭐ 486

A universal web-util for PHP.

Linkedin Profile Scraper Api ⭐ 404

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

Dataflowkit ⭐ 394

Extract structured data from web sites. Web sites scraping.

Tinking ⭐ 337

🧶 Extract data from any website without code, just clicks.

Dart Fss ⭐ 289

한국 금융감독원에서 운영하는 다트(Dart) 시스템 크롤링을 위한 라이브러리

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

Web Scraping ⭐ 281

Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

An Aggregator Engine for searching and downloading movies free - NO ADs!

Pupflare ⭐ 267

A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

Python Automation Scripts ⭐ 264

Simple yet powerful automation stuffs.

Requests Html ⭐ 207

Pythonic HTML Parsing for Humans™

Email Extractor ⭐ 134

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

Spyscrap ⭐ 124

CLI and GUI for OSINT. Are you very exhibited on the Internet? Check it! Twitter, Tinder, Facebook, Google, Yandex, BOE. It uses facial recognition to provide more accurate results.

Nintendeals ⭐ 124

Library with a set of tools for scraping information about Nintendo games and its prices across all regions (NA, EU and JP).

Newspaper3_usage_overview ⭐ 120

This repository provides usage examples for the Python module Newspaper3k.

Cloudflaresolverre ⭐ 117

Cloudflare Javascript & reCaptcha challenge (I'm Under Attack Mode or IUAM) solving / bypass .NET Standard library.

Leetcode Questions Scraper ⭐ 117

Scrape Algorithm Questions from leetcode and generate html and epub file

Apktrack ⭐ 115

ApkTrack is an Android app which checks if updates for installed APKs are available.

Scraply ⭐ 114

Scraply a simple dom scraper to fetch information from any html based website

Zippyshare Scraper ⭐ 110

A module to get direct downloadable links from zippyshare download page.

Html2rss ⭐ 106

📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.

Seleniumcrawler ⭐ 105

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

The Youtube Scraper ⭐ 101

Download YouTube video description and video comments without using the YouTube API.

Spidergram ⭐ 89

Structural analysis tools for complex web sites

Ebaymarketanalyzer ⭐ 83

Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel

Anitop is an unofficial simple API from https://anitrendz.net/ site

Youtube Audio ⭐ 66

extract videos from youtube in audio format using webscraping techniques 🎶

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Webreaper ⭐ 59

Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

Outscraper Python ⭐ 58

The library provides convenient access to the Outscraper API from applications written in the Python language. Allows using Outscraper's services from your code.

Proxycrawl Python ⭐ 57

ProxyCrawl Python library for scraping and crawling

Social Scraper ⭐ 56

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

Leetcode ⭐ 52

At present contains scraped data from around 1500 problems present on the site. More to follow....

Dart Scraper ⭐ 51

한국 금융감독원에서 운영하는 다트(Dart) 시스템을 이용한 기업 재무제표 추출 프로그램

Instagram To Discord ⭐ 49

Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2021!

Instagram Bot ⭐ 48

🤖 Python bot to view stories, like and comment on Instagram

Kal El Network Stress Test and Penetration Testing Toolkit

Fzbypassbot ⭐ 46

A Elegant Fast Multi Threaded Bypass Bot for Bigger Deeds. Try Now !!

Beautifulsoup Tutorial ⭐ 45

✨ 🍜 Scrape webpage metadata using BeautifulSoup.

Crawling Projects ⭐ 43

Web scraping and automation using python

Teslaservicemanualscraper ⭐ 42

This script will download the Tesla Service Manual onto a local doc folder for offline access.

Proxy_web_crawler ⭐ 39

Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords

Readability Cli ⭐ 35

A CLI for Mozilla Readability. Get clean, uncluttered, ready-to-read HTML from any webpage!

Botvid 19 ⭐ 34

Messenger Bot that scrapes for COVID-19 data and periodically updates subscribers via Facebook Messages. Created using Python/Flask, MYSQL, HTML, Heroku

Text Analysis ⭐ 32

Explaining textual analysis tools in Python. Including Preprocessing, Skip Gram (word2vec), and Topic Modelling.

Imdb Scraper ⭐ 32

🎬 An attempt at the most complete IMDb API

Fulldom Server ⭐ 31

Proxy-like server that will show you the DOM of a page after JS runs

Onlyfans Scrapper ⭐ 29

Script to download media, post and more from creators on OnlyFans

Linkedin Web Scraper ⭐ 28

Python Web Scraper for LinkedIn to collect and store company data (e.g. name, description, industry, etc.) into .xls file

Gopher Parse Sitemap ⭐ 28

A high effective golang library for parsing big-sized sitemaps and avoiding high memory usage. The sitemap parser was written on golang without external dependencies.

React Node Web Scraper ⭐ 28

Final Year project, scraping data of e-commerce stores and display in ReactJS app.

Sample Web Scraping With Electron ⭐ 26

Sample project for web scraping with Electron

Video Crawler ⭐ 26

Crawl websites for videos from Youtube, Vimeo, Soundcloud, etc

Phpmediaserver ⭐ 24

Manage and play your home videos in any browser

Instagram Downloader ⭐ 24

Instagram user's photos and videos downloader. Download all media files from any username. Working 2021!

Tripadvisor_crawler ⭐ 22

Python Crawler: Scrape Data From Tripadvisor

Metafetch ⭐ 21

NodeJS package that fetches a given URL's title, description, images, links etc.

Big Data Upf ⭐ 21

RECSM-UPF Summer School: Social Media and Big Data Research

Reason Rust Scraper ⭐ 21

🦀 Scraping & crawling websites using Rust, and ReasonML

Tradetheevent ⭐ 20

Implementation of "Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading." In Findings of ACL2021

Proxycrawl Node ⭐ 20

ProxyCrawl Node library for scraping and crawling

Multi Go ⭐ 20

A multi-tool made in Go, and aimed at security experts to make life a little more convenient

Pydolarvenezuela ⭐ 20

Esta librería desarrollada en Python te brinda la capacidad de consultar los precios del dólar y/o euro en diversos monitores en Venezuela, también como las tasas de cambio del BCV.

Scrapman ⭐ 20

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

Scrape Whoscored Event Data ⭐ 19

Get match event data from whoscored.com

Proxycrawl Php ⭐ 19

ProxyCrawl PHP library for scraping and crawling websites

Costco Scrape ⭐ 19

Indonesia News Scraper ⭐ 18

A news scraper for nodejs that help to scrap news from Indonesian news portal.

Sinaweiboscraper ⭐ 18

Web Scraper for Sina Weibo Search by Keywords

Torchestrator ⭐ 18

Spin up Tor containers and proxy HTTP requests via these tor instances.

Document Dl ⭐ 18

Command line program to download documents from web portals

Multporn ⭐ 16

A python library used to scrape and download from Multporn.net

Ecommerce Scraping Api ⭐ 16

eCommerce Scraping API code examples for Python, PHP and Node.js

Mapping sites to AniList and back.

Fbspider ⭐ 16

Scraping Facebook information

Selenium_python ⭐ 15

Firstcyclingapi ⭐ 15

An unofficial Python API wrapper for firstcycling.com

Ryuanime ⭐ 15

A free anime streaming , using the jkanime content by scraping the jkanime website.

Outscraper Php ⭐ 15

The library provides convenient access to the Outscraper API from applications written in the PHP language. Allows using Outscraper's services from your code.

Web Scraping Api ⭐ 15

Web Scraping API code examples for Python, PHP and Node.js

Lazyscraper ⭐ 15

Lazy helper tool to make easier scraping with simple tasks

Spb Unofficial Wrapper ⭐ 14

Unofficial NodeJS wrapper for the ScrapingBee API

Scavenger ⭐ 14

Scrape and take screenshots of dynamic and static webpages

Kick Off Web Scraping Python Selenium Beautifulsoup ⭐ 13

A tutorial-based introduction to web scraping with Python.

Benny Scraper ⭐ 13

Webnovel and Manga Scraper that stores Webnovels as Epubs, and mangas as either PDFs of Comicbook Archives

Lyrics Corpora ⭐ 13

An unofficial Python API that allows users to create a corpus of lyrical text from their favorite artists and billboard charts

Olx_scraper ⭐ 13

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Gochanges ⭐ 13

**[ARCHIVED]** website changes tracker 🔍

Containerized Ferret worker

Dataset Indian Companies ⭐ 12

Web Scraping "List of companies in India" from AmbitionBox Website using Python and Beautiful Soup

Webscraping_email_phone ⭐ 11

Web scraping of Emails and Phone numbers from various websites

✂ Agent library for scraping Facebook groups and pages in Node.js

Animeflv Scrapper ⭐ 11

🏯 AnimeFLV fake API made scrapping the AnimeFLV website

1-100 of 167 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.