Awesome Open Source

Programming Languages

Search results for web crawler

1,484 search results found

Web Miner ⭐ 28

Crawls sites, to find new content and scrap it

Professional Javascript ⭐ 28

Fast-track your web development career using the powerful features of advanced JavaScript

Appliedmathschoollectures ⭐ 28

Lectures on "crime and political corruption analysis using data mining, machine learning and complex networks" at the School of Applied Mathematics in the Institute of Mathematics and Computer Science at University of São Paulo

API Simples que retorna dados sobre tal ação/empresa da B3

A configurable command-line web scraper written in go with auto configuration capability

Pycreeper ⭐ 27

一个用来快速提取网页内容的信息采集（爬虫）框架，实现了对网页的动态加载与控制。

Pinterest ⭐ 27

The script has been working for a long time and it was really cool, but Pinterest prevent webscraping. You should be able to webscrap the first page of pins but not more thant that. -- A PHP and casperjs scripts to webscrap and display elegantly your Pinterest with WookMark jQuery plugin -- The script

Puppeteer Service ⭐ 27

🎠 Run headless Chrome (aka Puppeteer) as a service.

Rhodesapi ⭐ 27

API for Arknights

Selenium Grid Docker Swarm ⭐ 27

web scraping in parallel with Selenium Grid and Docker

Email Report ⭐ 27

A modular template for scraping data from the web to send yourself scheduled email reports

Web Scraper Nabidek Pronajmu ⭐ 27

Nástroj pro hlídání nových nabídek nemovitostí na populárních realitních serverech. Nabídky jsou vypisovány do Discord roomky.

Heroku Casper Node ⭐ 27

A sample Heroku application that cascades NodeJS on CasperJS, suitable for web scraping and automation.

Small Data Projects ⭐ 27

Repository of small data analysis and visualisation projects to try out new libraries and create new types of visualisations. Mostly using Python.

Wawebsessionhandler ⭐ 27

(DISCONTINUED) Save WhatsApp Web Sessions as files and open them everywhere!

Webtranspose ⭐ 27

Web scraping API for building AI applications.

Craigslistscraper ⭐ 27

Simple webscraper for Craigslist.

Scrapingant Client Python ⭐ 26

ScrapingAnt API client for Python.

Telegram Search ⭐ 26

Simple web scrapping to search from telegram

Rotating Proxies With Python ⭐ 26

Learn about how to rotate proxies by using Python.

Supercodingbot ⭐ 26

THE TELEGRAM BOT FOR COMPETITIVE PROGRAMMERS

Web Scraping Magic With Scrapy And Python ⭐ 26

This repository contains my experiments with Scrapy for advanced web scraping in Python

Comicbookmaker ⭐ 26

Script to fetch webcomics and use them to create ebooks.

Investigation Amazon Brands ⭐ 26

Materials to reproduce our findings in our stories, "Amazon Puts Its Own 'Brands' First Above Better-Rated Products" and "When Amazon Takes the Buy Box, it Doesn’t Give it up"

A web scraper to download intraday Fitbit data (previously) unavailable in the official API

A web spider for shodan.io without using the Developer API.

A fast Web API scraper written in C++ and built on Boost ASIO

Webcrawler ⭐ 25

A web crawler based on requests-html, mainly targets for url validation test.

Scrape The Gibson ⭐ 25

Code snippets for a workshop on web scraping.

Bd Medicine Scraper ⭐ 25

Scrapy-Django PostgreSQL integrated API with Proxy IP configuration that scrapes all medicine data (meds, prices, generics, companies, indications) from Bangladesh (30k+ pages)

Deviantart Gallery Downloader ⭐ 25

fetch deviantart's images using mechanize

Lemonderssreader ⭐ 25

📰 Read RSS feed from LeMonde.fr and display news inside the App

Scrapy Bench ⭐ 25

A CLI for benchmarking Scrapy.

Java web crawling library

Covid 19 India Data ⭐ 25

data and code for scrapping and cleaning data on covid-19 in India from https://www.mohfw.gov.in/ and https://www.covid19india.org/

Conactivity ⭐ 25

A tool built with Puppeteer that parses the LinkedIn profiles of a company's employees and returns the list of active employees.

Playwright Web Scraping ⭐ 25

A tutorial for web scraping using Playwright headless browser

Clojure library for web scraping

a complex but scalable web spider

City Scrapers Template ⭐ 25

Template for creating a City Scrapers project in your area

Newshound ⭐ 25

This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around the world in over 50 languages.

Scraper Whoscored ⭐ 24

Webscraper for the website www.whoscored.com using Python and Selenium

Perform Google Dork search with Dorkify

Glassdoor Interview Scraper ⭐ 24

Web scraper for Glassdoor interview review data

India Whatsappfakenews Dataset ⭐ 24

WhatsApps related deaths News Articles along with other articles across India during that period

Pinscrape ⭐ 24

A simple library to scrape Pinterest images written in Python

Veri_cekme ⭐ 24

Beautifulsoup and Selenium

Blog.brasil.io ⭐ 24

Blog do Brasil.IO

Scrappaper ⭐ 24

A web scrapping method to extract journal information from PubMed and Google Scholar using Python.

Freegamesonsteam ⭐ 24

Searching SteamDB for Free Games and Activating them using ArchiSteamFarm

Wattpad2epub ⭐ 24

Python Script to Scrape Wattpad Story and convert to Epub and html file. Easiest to use.

Gsoc Data Analyser ⭐ 24

Simple search for organisations participating/participated in the GSoC

Mimo Crawler ⭐ 24

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Sncrawler ⭐ 24

A web crawler written with pentesting in mind and some hacks for smart crawling

Nba_predictions ⭐ 23

Reworked NBA Predictions (in Python)

Otakudesu Scraper ⭐ 23

unofficial otakudesu.cam rest api

Burgoking ⭐ 23

Burger King - Free Burger Code Generator

Imdb Api ⭐ 23

[🚧 WIP] Cross-platform microservice to scrape the IMDb website.

Capitol Breach Scraper And Data ⭐ 23

⚖️ R code to scrape the DoJ Capitol Breach Cases and daily data files of said scraped info via GH actions

The Web Spider Obstacle Course

Add Cover Art ⭐ 23

Change cover photo of all your songs automatically using python

Springboard Data Science Immersive ⭐ 23

Lagou Job Report ⭐ 23

Using web crawler to dig information from lagou.com 从拉勾招聘小窥互联网行业发展

Eztrackr ⭐ 23

v3 of Eztrackr's Chrome extension. Designed to ease your job hunt by adding your jobs in an organized Trello board ✨

Distributed, asynchronous web crawler

Crawler.py ⭐ 23

async web crawler

Igscraperkit ⭐ 23

Create dynamic web scraper in Objective-C or Ruby!

Yet Another Streaming Tool

Sp Subway Scraper ⭐ 23

🚆This web scraper builds a dataset for São Paulo subway operation status

Animeflix Cli ⭐ 23

AnimeFlix CLI is a command-line interface (CLI) program that allows you to search for anime and stream it using webtorrent. The program fetches anime information from Nyaa and lets you select the anime to watch it.

Nian Crawler ⭐ 23

A web crawler for api.nian.so

Scrapebox ⭐ 23

A simple, system independent infrastructure for performing web scraping. Utilizes Vagrant virtualbox interface and puppet provisioning to create and execute scraping of web content to structured data quickly and easily without modifying your core system.

Interesting datasets for personal projects or submissions to #TidyTuesday

Scrapeadvisor ⭐ 22

A user-friendly python-based GUI which provides sentiment analysis of users' reviews toward a specific TripAdvisor facility

Media Crawler ⭐ 22

Web scraper for generating a graph of media connections via articles, twitter, reddit, and more

Restock Bot ⭐ 22

Basic Python script that watches products on Shopify stores for a restock and updates the user via SMS or attempts to purchase them once they become available.

API to access banking account

Hour_of_code_python_2015 ⭐ 22

Python Allrecipes ⭐ 22

Python API to search & get recipes from the 'allrecipes.com' website (web crawler, unofficial)

Spinarago ⭐ 22

A basic web crawler written in Go

Assessor Scraper ⭐ 22

A project to scrape the assessor's website and make the data accessible for advanced queries

Darkwebbot ⭐ 22

Dark Web Crawler for crawling the hidden onion sites and indexing them in Solr

Webscraping Selenium ⭐ 22

webscraping using selenium python

Cryptocurrency_data_downloader ⭐ 22

Download cryptocurrency historical data from Binance

Scraping Youtube Comments ⭐ 22

Scrape comments from any Youtube video

Puppeteer Railway Buildpack ⭐ 21

Installs dependencies needed in order to run puppeteer on Railway.

Streamlit Selenium ⭐ 21

Streamlit project to test Selenium running on Streamlit Cloud

Bunker Api ⭐ 21

This is a API/Website to see the attendance recorded in your college website along with how many days you can take days off OR to attend class!!

Bitcoin Bar ⭐ 21

Physical Bitcoin Stat Ticker

Tools for web-scraping

Pastebin Bisque ⭐ 21

Download all of a given user's public Pastebin pastes

Codepen Puppeteer ⭐ 21

Use Puppeteer to download pens from Codepen.io as single html pages

Scrapybook ⭐ 21

精通Scrapy网络爬虫

Codechef Rank Comparator ⭐ 21

Web application hosted on Heroku cloud platform based on web scraping in python using lxml library (XML Path Language).

Node Web Crawler ⭐ 21

Web Crawler in Node.js

Toronto Apartment Finder ⭐ 21

[really old and probably doesn't work] Slack bot to post relevant Toronto apartment listings from Kijiji & Craigslist

Zillow_scraper ⭐ 21

Repo for Zillow Web scraper

Linkedin Job Report Creator ⭐ 21

This program creates a PDF including the original post with graphs of the most used language from scraping a LinkedIn job post using Python.

A mean little DSL'd poltergeist (capybara) based web crawler that stuffs data into your Rails app.

Kindlefy ⭐ 20

📑 A way to automatically sync data with your kindle, such as RSS feeds, manga, and too much more.

Related Searches

Scraper Web Crawler (1,388)

501-600 of 1,484 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.