Awesome Open Source

Programming Languages

Search results for web crawler

1,484 search results found

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

AI based web-wrapper for web-content-extraction

Yellowpages Scraper ⭐ 43

Yellowpages.com Web Scraper written in Python and LXML to extract business details available based on a particular category and location.

Linkedin Job Scraper ⭐ 42

LinkedIn scraper to retrieve and store a live stream of job postings

Newsemble ⭐ 42

API for fetching data from news websites.

Stockforecast ⭐ 42

🎯 predict the price trend of individual stocks using deep learning and natural language processing

Website Stalker ⭐ 42

Track changes on websites via git

Ajax_crawler ⭐ 41

A flexible web crawler based on Scrapy for fetching most of Ajax or other various types of web pages. Easy to use: To customize a new web crawler-You just need to write a config file and run.

Tab Scraper ⭐ 41

Interface for downloading guitar tabs from Ultimate Guitar

Rust Web Crawler saving pages on Redis

Dead simple web crawler for Python

Ronin Web ⭐ 40

ronin-web is a collection of useful web helper methods and commands.

Jason The Miner ⭐ 40

⛏ A versatile Web scraper for Node.js

House Renting Spider ⭐ 39

A crawler for accommodation rental information in Douban Group 豆瓣小组上海租房爬虫

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

Scrapemate ⭐ 39

Golang Crawling and scraping framework

Learncpp Download ⭐ 39

An advanced web scraper tool that seamlessly fetches and combines over 200 online tutorials into a convenient offline PDF format.

Fifa Fut Data ⭐ 39

Web-scraping script that writes the data of all players from FutHead and FutBin to a CSV file or a DB

Browserext ⭐ 38

A PHP extension for web scraping and browser emulation based on QtWebKit. Supports javascript and AJAX.

Cambridge ⭐ 38

Terminal version of Cambridge Dictionary by default. Also supports Merrian-Webster Dictionary.

Projeto_etl_rfb_ibge_anp ⭐ 38

PYTHON E POSTGRESQL - EXTRACT TRANSFORM LOAD - ETL - DADOS PÚBLICOS DA RECEITA FEDERAL DO BRASIL - RFB, INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA - IBGE E AGÊNCIA NACIONAL DO PETRÓLEO, GÁS NATURAL E BIOCOMBUSTÍVEIS - ANP - PYTHON E POSTGRESQL

Goodreads_textmining ⭐ 38

Webscraping and analyzing book reviews on GoodReads

A fast and powerful web scraping library

Validate Website ⭐ 38

Web crawler for checking the validity of your documents.

Gpt Automated Web Scraper ⭐ 38

The GPT-based Universal Web Scraper MVP is a solution that leverages GPT models and web scraping libraries to generate scraper code based on user input and website analysis, simplifying the web scraping process.

Procyclingstats ⭐ 38

procyclingstats scraper

Pymultidictionary ⭐ 38

PyMultiDictionary is a dictionary module that gets meanings, translations, synonyms, and antonyms of words in 20 different languages

Notionsnapshot ⭐ 37

notion web scraper

Netcloud ⭐ 37

NetCloud Web Spider

Web scraper for Scala

a web crawler and data analysis repo with Python3.5, R, Excel 2016 and TAGUL

Funda Scraper ⭐ 37

FundaScaper scrapes data from Funda, the Dutch housing website. You can find listings from house-buying or rental market, and historical data.

Incapsula Cracker ⭐ 37

Use to bypass sites which use incapsula to block access to webscraping bots.

Collect and Clean Hockey Stats

⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs

Web Unblocker ⭐ 36

Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems.

Web crawler with a Ruby API

Redditsfinder ⭐ 36

Archive a reddit user's post history. Formatted overview of a profile, JSON containing every post, and picture downloads. Uses the pushshift API.

Linkedin Comments Scraper ⭐ 36

Script to scrape comments (including name, profile link, pfp, designation, email(if present), and comment) from a LinkedIn post from the URL of the post.

Us Stock Prediction Using Ml And Spark ⭐ 35

Predict stock price based on financial news feeds

Detourning The Web ⭐ 35

Syllabus and example code for 7-week class at NYU/ITP

Lc Webscraping ⭐ 35

Introduction to web scraping

Flink Crawler ⭐ 35

Continuous scalable web crawler built on top of Flink and crawler-commons

Webscraper ⭐ 35

iOS library for web scraping

Python Marmiton ⭐ 35

Python API to search & get recipes from the 'marmiton.com' website (web crawler, unofficial)

Goscrapy ⭐ 34

GoScrapy: Harnessing Go's power for efficient web scraping, inspired by Python's Scrapy framework.

Animeflv ⭐ 34

Animeflv is a custom API that has the entire catalog of the animeflv.net website. You can enjoy all the content with subtitles in Spanish and the latest in the world of anime for free.

Cobweb Lnx ⭐ 34

CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

🍷 Scraps data from Vivino and collects outstanding wine-based meta-data.

Public Roadmap ⭐ 34

Public roadmap for SerpApi, LLC (https://serpapi.com)

Scraping Assisted by Learning

Open Australian Legal Corpus Creator ⭐ 34

The code used to create and update the Open Australian Legal Corpus, the first and only multijurisdictional open corpus of Australian legislative and judicial documents.

A simple web-crawler development framework based on .Net Core.

Iranian Phonenumber Validation ⭐ 33

Regex collection for validating Iranian phone numbers

Web Scraping Using Python ⭐ 33

This project scrapes Wikipedia for its articles using BeautifulSoup to create a dataset and then draws analysis on the collected data.

Jsonscraper ⭐ 33

JSON configurable concurrent scraper

Animal Crossing Scraper ⭐ 33

Web scraper for Animal Crossing - New Horizons data using bs4

Paperboy ⭐ 32

A comprehensive (eventually) collection of webscraping scripts for news media sites

Paperscraper ⭐ 32

A web scraping tool to systematically extract the text of scientific papers and corresponding metadata from university accessible journals.

𝐫⟋ Get Reddit data

Htmlunit ⭐ 32

🕸🧰☕️Tools to Scrape Dynamic Web Content via the 'HtmlUnit' Java Library

Utlyz Cli ⭐ 32

Let's you to access your FB account from the command line and returns various things number of unread notifications, messages or friend requests you have.

Text Analysis ⭐ 32

Explaining textual analysis tools in Python. Including Preprocessing, Skip Gram (word2vec), and Topic Modelling.

Sourcely ⭐ 32

Movie Recommendation System With Sentiment Analysis ⭐ 32

Content based movie recommendation system with sentiment analysis

Linkedin Client ⭐ 32

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

Market Trend Prediction ⭐ 32

This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).

A Kotlin multiplatform HTML5 parsing library

Fredsroadtripstoryteller ⭐ 31

Hear local historical markers as you travel on your road-trip. 100% Shared Compose UI, Kotlin native cross-platform codebase. Includes Cocoapods, Google Maps, GPS Location, notifications, background location tracking, In-App purchases, web-scraping, networking, persistent storage, CommonFlow

Linkextractor ⭐ 31

A Docker tutorial using a link extraction application example

Supercharged Web Scraping With Asyncio ⭐ 31

Scrape websites asynchronously with Python 3.8+, Asyncio, & arsenic (aka Selenium for Async).

Google Search Results Java ⭐ 30

Google Search Results JAVA API via SerpApi

Videorecognition Realtime Autotrainer Alerts ⭐ 30

State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.

Bot_bandejao_ufmg ⭐ 30

🤖🍴 A Python script that scrapes UFMG's restaurants menus and publishes them @bot_RU_UFMG Twitter profile

Gcf Packs ⭐ 30

Library packs for google cloud functions

Scrape Google Scholar Py ⭐ 30

Extract data from all Google Scholar pages from a single Python module.

Goodreadsscraper ⭐ 30

📚 A GoodReads.com Scraper script to get books reviews including text and rating.

Spidyquotes ⭐ 30

Example site for web scraping tutorials

Crawler4j ⭐ 30

Open Source Simple Web Crawler for Java. Simple Flexible And Lightweight

A cli tool to browse and play anime in pt-br

Boilerpipe Ruby ⭐ 30

Pure ruby implementation of the Boilerpipe content extraction algorithm tuned for online articles

Python Data From Web ⭐ 29

API and web scraping workshops

Scraping Dynamic Javascript Ajax Websites With Beautifulsoup ⭐ 29

A guide on how to scrape JavaScript rendered websites with Python and BeautifulSoup.

Seeshell ⭐ 29

Documentation and example scripts for SeeShell Automation

2017 Summer Workshop ⭐ 29

Exercises, data, and more for our 2017 summer workshop (funded by the Estes Fund and in partnership with Project Jupyter and Berkeley's D-Lab)

No Fasel Scrapers ⭐ 29

The web scrapers used to generate the files used by the NoFasel App.

N8n Nodes Browserless ⭐ 29

n8n node to interact with browserless instance

快速登录大连理工大学统一身份认证系统（SSO）的Python模块，可轻松实现成绩提醒、抢课、玉兰卡信

Python Web Scraping Tutorial ⭐ 29

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

Google Covid Mobility Scrape ⭐ 29

Script for scraping Google's COVID19 Community Mobility Reports [ARCHIVED]

Stormscraper ⭐ 29

A Storm based web crawler with Cassandra backend

Pycraigslist ⭐ 29

Craigslist API wrapper

Non Api Fb Scraper ⭐ 29

Scrape public FaceBook posts from any group or user into a .csv file without needing to register for any API access

Danger Mention ⭐ 29

Danger plugin to automatically mention potential reviewers on pull requests

Web Scraping Framework

Botcity Framework Web Python ⭐ 28

BotCity Framework Web - Python

React Node Web Scraper ⭐ 28

Final Year project, scraping data of e-commerce stores and display in ReactJS app.

Linkedin Web Scraper ⭐ 28

Python Web Scraper for LinkedIn to collect and store company data (e.g. name, description, industry, etc.) into .xls file

web scraping tool for grailed.com

Professional Javascript ⭐ 28

Fast-track your web development career using the powerful features of advanced JavaScript

Related Searches

Scraper Web Crawler (1,388)

401-500 of 1,484 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.