Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for web crawler
web-crawler
x
1,485 search results found
Bytepods
⭐
138
Daily podcasts generated by AI 🤖
Learnpythonforresearch
⭐
137
This repository provides everything you need to get started with Python for (social science) research.
Actor Page Analyzer
⭐
136
Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.org microdata and JSON-LD metadata, analyzes AJAX requests, etc.
Hockey Scraper
⭐
134
Python Package for scraping NHL Play-by-Play and Shift data
Competitive_programming_score_api
⭐
134
API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit
Ph Submissions
⭐
133
The repository and website hosting the peer review process for new Programming Historian lessons
Not Your Average Web Crawler
⭐
130
A web crawler (for bug hunting) that gathers more than you can imagine.
Zillow
⭐
129
Zillow Scraper for Python using Selenium
Cascadia
⭐
128
Go cascadia package command line CSS selector
Scrapers
⭐
128
Lots and lots of web scrapers
Clicknium Docs
⭐
127
A next-generation GUI automation framework for Web and Desktop Application Testing and Automation.
Gpt4v Scraper
⭐
126
AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.
Ospider
⭐
124
开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)
Proxy
⭐
123
A simple tool for fetching usable proxies from several websites.
Php Crawler
⭐
121
A php crawler that finds emails on the internets
Dyer
⭐
118
Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
Save For Offline
⭐
118
Android app for saving webpages for offline reading.
Algolisted
⭐
118
Algolisted is an AI-powered nonprofit analytics firm dedicated to assisting computer science students in preparing for placements and internships. Our services include tracking and analytics across various platforms and topics.
Extractnet
⭐
118
A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package
Educative.io_scraper
⭐
117
Educative.io Course Downloader developed using Python and Selenium. Refer Readme.md for setup instructions.
Evine
⭐
117
Interactive CLI Web Crawler
Nytcrossword
⭐
117
An exploration of New York Times crossword answers from 1994-2017, i.e. the Will Shortz era.
Geeksforgeeksscrapper
⭐
116
Scrapes g4g and creates PDF
Html Metadata
⭐
115
MetaData html scraper and parser for Node.js (supports Promises and callback style)
Raspagem De Dados Para Iniciantes
⭐
115
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
Homeharvest
⭐
114
Python package for real estate scraping of MLS listing data
Bancocentralbrasil
⭐
112
💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil
Leetcode Compensation
⭐
112
Compensation analysis of leetcode.com/discuss/compensation.
Crawlbox
⭐
112
Easy way to brute-force web directory.
Collector
⭐
111
Collector is a OSINT tool and information gathering. This tool can do information gathering phone numbers, github account, ip address and instagram account.
Gflare Tk
⭐
110
Open-Source Python Based SEO Web Crawler
Abotx
⭐
106
Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
Seleniumcrawler
⭐
105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
Node Web Crawler
⭐
104
A web scraper with a web user interface which shows scraping stats in realtime. Uses Node.JS, jQuery, socket.io and Express.
Scraper
⭐
104
Web scraper for scraping, tracking and visualizing prices of products on various websites.
Haikei
⭐
102
HaiKei is an anime streaming website that uses the consumet API
Htmldate
⭐
101
Fast and robust date extraction from web pages, with Python or on the command-line
Zhihu_crawler
⭐
100
a crawler for zhihu
Reader
⭐
100
Extract clean(er), readable text from web pages via Mercury Web Parser.
Facebook Marketplace Scraper
⭐
99
This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.
Cascadia.jl
⭐
98
A CSS Selector library in Julia
Crawl Anywhere
⭐
98
Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.
Get Sauce
⭐
97
A command line program to download Hentai videos and images from multiple websites
Cowin Vaccine Notifier
⭐
97
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
Senpwai
⭐
97
A desktop app for tracking and batch downloading anime
Krawler
⭐
96
A web crawling framework written in Kotlin
Brutescrape
⭐
95
A web scraper for generating password files based on plain text found
Polipus
⭐
95
Polipus: distributed and scalable web-crawler framework
Wswp
⭐
95
Code for the second edition Web Scraping with Python book by Packt Publications
Actor Scraper
⭐
93
House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
Python Sec
⭐
93
A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.
Scrapy Gui
⭐
93
A simple, Qt-Webengine powered web browser with built in functionality for basic scrapy webscraping support.
Nba_betting
⭐
93
Using data analytics and machine learning to create a comprehensive and profitable system for predicting the outcomes of NBA games.
Terpene Profile Parser For Cannabis Strains
⭐
93
Parser and database to index the terpene profile of different strains of Cannabis from online databases
Rymscraper
⭐
92
Python API to extract data from rateyourmusic.com.
Bing Ip2hosts
⭐
91
bingip2hosts is a Bing.com web scraper that discovers websites by IP address
Ps239t
⭐
91
Introduction to Computational Tools and Techniques for Social Research
Anmeldung Berlin
⭐
91
This app will find and book any service.berlin.de appointment that can be booked online.
Torrents Api
⭐
90
Torrent Api ✨
Scrapyd Cluster On Heroku
⭐
90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
Amacapy Bot Telegram Amazon Affiliates
⭐
90
Amacapy is a software that does web scraping to the Amazon website and publishes them on Telegram, searches the products by the keyword entered or the direct link of the product. Then you can publish these products on Telegram in a certain time. The technologies used were Flet, Beautiful Soup and Python.
Email Crawler Lead Generator
⭐
88
This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.
Udemy_bot
⭐
88
An automation bot for free Udemy courses
Splashr
⭐
88
💦 Tools to Work with the 'Splash' JavaScript Rendering Service in R
Animeez
⭐
88
AnimeEZ - An Anime Streaming website without any ads for free (Demo - https://animeez.live) BTW ITS MADE IN HTML
Instago
⭐
87
Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram
Boxrec
⭐
87
Retrieve information from BoxRec and return it in JSON format
Nordvpn Switcher
⭐
86
Rotate between different NordVPN servers with ease. Works both on Linux and Windows without any required changes to your code!
Tacocat
⭐
86
A platform displaying the latest software engineer job information to entry-level new graduates
Webcrawler
⭐
86
Web crawler to download pictures from zhihu.com
Web Poet
⭐
85
Web scraping Page Objects core library
Crabler
⭐
85
Web Crawler for Crabs
Jsongenius
⭐
85
Get structured JSON data from any page.
Transfermarkt Api
⭐
84
API service to get data from Transfermarkt
Scrapper
⭐
83
Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.
Raise
⭐
83
A simple (and unofficial) GitHub Trending client that lives in your menubar.
Cl Torrents
⭐
83
Searching torrents on popular trackers - CLI, readline, GUI, web client. Tutorial and binaries (issue tracker on https://gitlab.com/vindarel/cl-torrents/)
Bathyscaphe
⭐
83
Fast, highly configurable, cloud native dark web crawler.
Ebaymarketanalyzer
⭐
83
Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel
Cianparser
⭐
83
Parser general information on the cian.ru website / Сбор данных с сайта объявлений Циан
Instatools
⭐
83
🧰 A collection of tools built for automating tasks on Instagram.
Lisc
⭐
81
Literature Scanner: Automated collection & analyses of the scientific literature.
Arachnid
⭐
80
Powerful web scraping framework for Crystal
Openscraper
⭐
80
An open source webapp for scraping: towards a public service for webscraping
Tableau Scraping
⭐
79
Tableau scraper python library. R and Python scripts to scrape data from Tableau viz
Node Search Engine
⭐
79
Sample search engine with web crawler, built on Node.js + CouchDB + Limestone
Deuce
⭐
78
R package for web scraping of tennis data
Seomacroscope
⭐
78
SEO Macroscope is a website scanning tool, to check your website for broken links; including some technical SEO functionality, site scraping, Excel reporting, and more.
Tatooine
⭐
78
A powerful scraper for JavaScript Developers.
Webscrapper
⭐
77
Simple and powerfull all in one Telegram Bot to scrap webpages using Requests, html5lib and Beautifulsoup
Browser Pool
⭐
77
A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.
Instagram Scraper 2021
⭐
77
Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).
Goodreadsscraper
⭐
76
Scrape data from Goodreads using Scrapy and Selenium 📚
Python Web Scraping Second Edition
⭐
76
Python Web Scraping Second Edition, published by Packt
Web Scraping
⭐
76
Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Requests, BeautifulSoup, Asyncio, Pandas, Numpy, and more!
Scrapfly Scrapers
⭐
76
Web scrapers for popular targets powered Scrapfly.io
Pymarketcap
⭐
74
Python3 API wrapper and web scraper for https://coinmarketcap.com
Web_scraper
⭐
73
A very basic web scraper implementation to scrap html elements from a web page.
Requests Random User Agent
⭐
73
Configures the requests library to randomly select a desktop User-Agent
Scraping Ebay
⭐
73
Scraping Ebay's products using Scrapy Web Crawling Framework
Related Searches
Scraper Web Crawler (1,388)
201-300 of 1,485 search results
< Previous
Next >
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.