Awesome Open Source

Programming Languages

Search results for web crawler

1,484 search results found

Palantiri ⭐ 15

Web crawler to collect data on ht

Mailinglistscraper ⭐ 15

A python web scraper for public email lists.

Tweet Transcriber ⭐ 15

A Reddit bot that transcribes tweets from comments and submissions links, mirrors their images and replies back with a formatted Markdown message.

Neural Scam Artist ⭐ 15

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Tarantula ⭐ 15

Another PHP crawler based on Guzzle.

Hotel_review_nlp ⭐ 15

NLP Multi-class classifcation of hotel reviews, with demo app. Please check README for more details.

A web scraper for TikTok using Playwright

Workers Tutorial ⭐ 15

This repository holds the code for a tutorial that teaches how to build to build a web-crawler using Node Workers.

Information Retrieval ⭐ 15

Elasticsearch, MongoDB, Tornado Server, RESTful API, Python, Information Retrieval, Machine Learning, Web Crawler

Supervised Machine Learning ⭐ 15

This repo contains regression and classification projects. Examples: development of predictive models for comments on social media websites; building classifiers to predict outcomes in sports competitions; churn analysis; prediction of clicks on online ads; analysis of the opioids crisis and an analysis of retail store expansion strategies using Lasso and Ridge regressions.

Pixarfilms ⭐ 15

🎥 R data package to explore Pixar films, the people, and reception data

Mirror Mirror ⭐ 15

A library to get images from social media

Senscritiquescraper ⭐ 15

Python API to extract data from senscritique.com.

Twitter_scraper ⭐ 15

Web Scraper for Twitter pages

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

Web Scraping Api ⭐ 15

Web Scraping API code examples for Python, PHP and Node.js

Tinyback ⭐ 15

A tiny web scraper

Constituicao ⭐ 15

Explorador da Constituição: a Constituição Federal e suas Emendas acessíveis para o mundo da Ciência de Dados

Indeed_job_scraper ⭐ 15

💼 Given the Skill and the Location, this script will scrape the Indeed Indian website and present the matching job details.

Mma Parser For Sherdog And Ufc Data ⭐ 15

Python web scraper for Sherdog & UFC data. Creates output of your choice in csv or json format.

Papercut ⭐ 15

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Webscrape ⭐ 15

A web scraper to scrape email's and phone numbers from Websites.

all-in collection of productivity scripts, CLI tools, utility libraries, fuse filesystems, and also some stuff

Anime Scraper ⭐ 15

[partially working] Scrape and add anime episode stream URLs to uGet (Linux) or IDM (Windows) ~ Python3

Aiohttp_chromium ⭐ 15

aiohttp-like interface to chromium. based on selenium_driverless to bypass cloudflare

Selenium_python ⭐ 15

Article Summary Deep Learning ⭐ 15

📖 Using deep learning and scraping to analyze/summarize articles! Just drop in any URL!

Python Scraperlib ⭐ 14

Collection of Python code to re-use across Python-based scrapers

Google_scraper_live_view ⭐ 14

Application for extracting large amounts of data from the Google search results page

A web scraper in Python using Django and Celery

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

Fetscrape ⭐ 14

Web Crawler ⭐ 14

Scrapher ⭐ 14

A web scraper for PHP to easily extract data from web pages

Sforswagbot ⭐ 14

A telegram chat bot for : Getting lyrics, Getting nearby restaurants and their menu and random quotes.

Covid19 India Discord Bot ⭐ 14

Actor Content Checker ⭐ 14

You can use this act to monitor any page's content and get a notification when content changes.

Parallelized web crawler written in Golang

🕸→ℹ️ Reap Information from Websites

Google Podcast Downloader ⭐ 14

CL tool to download entire google podcast library for the provided URL 🎵

parallel execution of RSelenium

Ktu Notifier ⭐ 14

An NLP based Telegram Bot that pushes KTU Announcements Notifications

Attendance Genie ⭐ 14

Never be "late" again ;)

Automation Scripts ⭐ 14

Simple scripts that I'm using to automate the boring things.

Connpassattendancechecker ⭐ 14

An attendance checking iOS application for https://connpass.com

Imdb Scraper ⭐ 14

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

Image Crawler ⭐ 14

An image scraper that scraps images from unsplash.com

Dropout Dl ⭐ 14

A tool for downloading dropout.tv episodes

Koronawirus Api ⭐ 14

API do śledzenia statystyk o koronawirusie w Polsce

Hsbcscraper ⭐ 14

Web scraper for downloading statements from HSBC Business Banking

Python Scrapfly ⭐ 14

Scrapfly Python SDK for headless browsers and proxy rotation

Comicbookcoversdownloader ⭐ 14

Webscraper for DC, Marvel and more Comicbook Wikias to download CB covers

Audiobooker ⭐ 14

Audio Book scrapper

Crawling and scraping the Web for fun and profit

Sig To Googlecalendar ⭐ 14

A python script to get class schedules on UFLA's SIG and convert to a .CSV file to use in Google Calendar

Option Scraper Blackscholes ⭐ 14

Repo for scraping option data required for the Black Scholes model. Data is scraped from S&P500 companies

City Scrapers Cle ⭐ 14

City Scrapers project for Cleveland

Stock Fundamental Data Scraping And Analysis ⭐ 14

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

Puppeteer Daily Menu Scraper ⭐ 14

Scraping daily menus of restaurants from downtown Budapest with Puppeteer.

Drag Race ⭐ 13

Project dedicated to collecting, organizing, and analyzing information about RuPaul's Drag Race and related franchises.

Bapcs Stock Checker ⭐ 13

Reddit price and stock checker bot - replies with useful info, and saves data for later analysis

Olx_scraper ⭐ 13

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Backlink Monitoring ⭐ 13

Backlink checker is a simple tool, which checks backlink quality, identifies problematic backlinks, and outputs them to a specific Slack channel

Hcaptcha Solver ⭐ 13

Automated hCaptcha solver using binary image classification networks

Pawn package manager for SA-MP

Gdscraper ⭐ 13

📦 R package to easily web scrape Glassdoor company reviews. Write up of demo:

Webmiddle ⭐ 13

Node.js framework for modular web scraping and data extraction

⚒ Web crawler that analyzes and dissects subtitles into database entries

Scripts used for training and teaching

Webcrawlertokopedia ⭐ 13

It is a web crawler and scrapper for https://www.Tokopedia.com. The project scrape the product-ID, product URL and product videos present under the product images present at right bottom of the page.

Robots.txt ⭐ 13

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Bdo Scraper ⭐ 13

A scraper for BDDatabase.net.

Web_scraper_ruby ⭐ 13

The program is implementing a scraper for search results of a job offers website. This scraper helps you to search by keywords on https://www.indeed.com webpage, to scrape the results and to display them in the terminal, or export it into a CSV or a Text files.

Crawler_pubg.op.gg ⭐ 13

This is a web crawler for pubg.op.gg, written by Ruichong Liu. 绝地求生游戏数据抓取

USC's Introduction to Health Data Science

Gramsleuth ⭐ 13

FOSS Instagram OSINT tool

M3u Audiobooks ⭐ 13

collection of 100k+ audio books, radio porgrams, music etc from archive.org in a easy to listen m3u playlist format

Webscraper ⭐ 13

Manga Api ⭐ 13

A Python based web scraping api built with fastapi that provides easy access to manga contents

Musicscraper ⭐ 13

CLI tool for scraping information from musical websites (Rateyourmusic, Metal Archives), with nice album ASCII art

Udemyscraper ⭐ 13

A Udemy Course Scraper built with bs4 and selenium, that fetches udemy course information. Get udemy course information and convert it to json, csv or xml file, without authentication!

Web Scraping 101 ⭐ 13

An Introduction to Web Scraping

Node Krawler ⭐ 13

Fast and lightweight web crawler with built-in cheerio, xml and json parser.

Web crawler based on Puppeteer

Yahoostats ⭐ 13

Webscrape stock statistic data from: yahoo finance, reuters, morningstar, zacks.

Boris The (Web) Spider

Requestsr ⭐ 13

R interface to Python requests module

Dse Cse Market Update ⭐ 13

Dhaka Stock Exchange & Chittagong Stock Exchange Share Market Real Time Update Data

WEB TAB makes it easy for you to stay up-to-date with the latest developer news, tools, jobs and events.

Proxy Gr ⭐ 13

Python-based Massive Proxy Grabber. This bot grabs proxies from public websites so you can use them.

Raspberry Pi Stock Checker ⭐ 13

A configurable python webscraper that checks raspberry pi stocks from verified sellers

Node Fetch Dom ⭐ 13

Magic utility that extract javascript global variables from a remote html page.

Linkedin Crawler ⭐ 12

Linkedin public profile parser in JS

R package to provide API to Melbourne pedestrian data

CpZen is an Online Integrated Development Environment (IDE) for competitive programmers made as the Lab project for CSE 4510: Software Development Lab.

Google Local Results Ai Parser ⭐ 12

A ruby gem to extract structured data from Google Local Search Results using the serpapi/bert-base-local-results model, enabling parsing, classification, and information extraction from English HTML content.

Versammlungen in Berlin: Konservieren historischer Daten.

Android App ⭐ 12

The must-have app for an Amritian.

Spiderwebai ⭐ 12

The fastest way to scrape data to feed your AI models and LLMs

Books Com Tw Crawler ⭐ 12

books.com.tw crawler 「博客來」資料爬蟲

Related Searches

Scraper Web Crawler (1,388)

701-800 of 1,484 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.