Awesome Open Source

Programming Languages

Search results for web crawler

1,485 search results found

City Scrapers ⭐ 315

Scrape, standardize and share public meetings from local government websites

Mov Cli ⭐ 314

A cli tool to browse and watch Movies/Shows/TV/Sports.

Be nice on the web

Uscrapper ⭐ 298

Uscrapper 2.0, a powerful OSINT webscraper for personal data collection. Uscrapper uses web scraping to extract email IDs, social-media links, geolocations, phone numbers, and usernames from webpages, supports multithreading, has advanced Anti-webscraping bypassing modules, supports webcrawling to scrape from various sublinks within the same domain

The simple, easy to use command line web crawler.

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

Pricewise ⭐ 284

Dive into web scraping and build a Next.js 13 eCommerce price tracker within a single video that teaches you data scraping, cron jobs, sending emails, deployment, and more.

Web Scraping ⭐ 281

Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Web Scraping ⭐ 276

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

Youtube Projects ⭐ 272

This repository contains all the code I use in my YouTube tutorials.

A web crawler for Go

H4x Tools ⭐ 269

Open source toolkit for scraping, OSINT and more.

Technicalconceptsforinterviews ⭐ 265

Various technical concepts for interviews - Feel free to contribute and make it better!

Tiktokbot ⭐ 262

A TikTokBot that downloads trending tiktok videos and compiles them using FFmpeg

Lagoujob ⭐ 250

Job data mining repo for lagou.com

Tradingview Data Scraper ⭐ 250

Extract price and indicator data from TradingView charts to create ML datasets

Dark Fantasy Hack Tool ⭐ 248

DDOS Tool: To take down small websites with HTTP FLOOD. Port scanner: To know the open ports of a site. FTP Password Cracker: To hack file system of websites.. Banner Grabber: To get the service or software running on a port. (After knowing the software running google for its vulnerabilities.) Web Spider: For gathering web application hacking information. Email scraper: To get all emails related to a webpage IMDB Rating: Easy way to access the movie database. Both .exe(compressed as zip) and .py

Easyapplyjobsbot ⭐ 247

A python bot to automatically apply all Linkedin,Glassdoor, etc Easy Apply jobs based on your preferences. Auto login, auto fill additional questions, apply automatically!

Football Data Collection ⭐ 246

Web Scraper used to create Kaggle European Soccer database

Netflix Clone ⭐ 245

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

Rcrawler ⭐ 240

An R web crawler and scraper

Nofasel ⭐ 239

A streaming app with no ADs.

Laravel ⭐ 238

Laravel adapter for Roach, the complete web scraping toolkit for PHP.

Summarizer ⭐ 236

A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

A simple browser/client-side web scraper.

Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam

Quora Api ⭐ 232

An unofficial API for Quora.

Nudecrawler ⭐ 231

Crawl telegra.ph searching for nudes!

News Crawl ⭐ 229

News crawling with StormCrawler - stores content as WARC

Fb_friend_list_scraper ⭐ 225

OSINT tool to scrape names and usernames from large friend lists on Facebook, without being rate limited.

Infinitycrawler ⭐ 221

A simple but powerful web crawler library for .NET

Nytimes Ios ⭐ 220

🗽 NY Times is an Minimal News 🗞 iOS app 📱 built to describe the use of SwiftSoup and CoreData with SwiftUI🔥

Wayback Machine Scraper ⭐ 219

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Amazon Scraper ⭐ 219

A simple web scraper to extract Product Data and Pricing from Amazon

Crawler Commons ⭐ 217

A set of reusable Java components that implement functionality common to any web crawler

Selenops ⭐ 215

A Swift Web Crawler 🕷

Awesome Web Scraper ⭐ 214

A collection of awesome web scaper, crawler.

91_python_mini_projects ⭐ 213

Make a ZIM file from any Web site and surf offline!

Crawley ⭐ 208

The unix-way web crawler

Twitter Scraper Selenium ⭐ 207

Python's package to scrap Twitter's front-end easily

Short Jokes Dataset ⭐ 205

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

Strong Web Crawler ⭐ 204

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascrip

ESPN Fantasy Football API

Everything Web Scraping ⭐ 197

Learn everything web scraping with David Teather Codes on YouTube

Imdb Api ⭐ 193

Serverless IMDB API powered by Cloudflare Worker

Humanoid ⭐ 191

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

Weboptout ⭐ 191

Opt-Out tool to check Copyright reservations in a way that even machines can understand.

Portia Dashboard ⭐ 190

portia-dashboard is a visual web crawler based on scrapinghub/portia

Letterboxd_recommendations ⭐ 190

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username

Awesomewebscraping ⭐ 189

List of libraries, tools and APIs for web scraping and data processing.

Nba Prediction ⭐ 186

A project to deploy an online app that predicts the win probability for each NBA game every day. Demonstrates end-to-end Machine Learning deployment.

Ignareo Isml Auto Voter ⭐ 186

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

Spider Less ⭐ 186

Web spider as a service, spider on serverless

Open Source web scraping API. Falkor turns web pages into queryable JSON

Crawlab Lite ⭐ 184

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Daath Ai Parser ⭐ 184

Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

Digger is a powerful and flexible web crawler implemented by pure golang

Zhihu Crawler People ⭐ 179

A simple distributed crawler for zhihu && data analysis

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Ayakashi ⭐ 177

⚡ Ayakashi.io - The next generation web scraping framework

Musicer ⭐ 176

旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

Trump Lies ⭐ 175

Tutorial: Web scraping in Python with Beautiful Soup

Scraping With Rust ⭐ 174

👾 scraping hacker news with rust

Goscrape ⭐ 172

Web scraper that can create an offline readable version of a website

Crawler_shopee_public ⭐ 169

蝦皮非同步爬蟲 + 競品賣家分析

Twitter Intelligence ⭐ 169

Twitter Intelligence OSINT project performs tracking and analysis of the Twitter

A threaded web-spider written in Clojure

Stardox ⭐ 166

Github stargazers information gathering tool

A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.

Collector Http ⭐ 162

Norconex Web Crawler (or spider) is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

📲 Bot to help solve HQ trivia

Tool made to automate tasks of pentesting.

Cocrawler ⭐ 159

CoCrawler is a versatile web crawler built using modern tools and concurrency.

Screenslicer ⭐ 159

Automatic, zero-config web scraping -- written in Java, has no dependency on Java EE or app servers, and the web scraper has a restful/JSON API. Currently unmaintained.

Bet On Sibyl ⭐ 157

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir

Animos - Clean and minimal Anime-streaming desktop application without any ads.

Imghash ⭐ 152

Perceptual image hashing for Node.js

Decryptr ⭐ 151

An extensible API for breaking captchas

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Blackmaria ⭐ 150

Python package for webscraping in Natural language

Facebook_page_scraper ⭐ 150

Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

Nokolexbor ⭐ 149

High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.

Web Scraper Chrome Extension ⭐ 149

Web data extraction tool implemented as chrome extension

Aws Pdf Textract Pipeline ⭐ 148

🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Scrape Up ⭐ 148

A web-scraping-based python package that enables you to scrape data from various platforms like GitHub, Twitter, Instagram, or any useful website.

可视化任务调度系统，精简到一个二进制文件 (Web visual task scheduler system , yes ! just one binary solve all the problems !)

Juno_crawler ⭐ 147

Scrapy crawler to collect data on the back catalog of songs listed for sale.

Webchem ⭐ 147

Chemical Information from the Web

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Google News Scraper ⭐ 144

Lightweight scraper for Google News

Web Database Analytics ⭐ 144

Web scrapping and related analytics using Python tools

🎬 A Crunchyroll show/season ripper

Saveddit ⭐ 143

Bulk Downloader for Reddit

Direct_web_spider ⭐ 143

A direct web spider framworks for Ruby

estela, an elastic web scraping cluster 🕸

Scrapy Training ⭐ 141

Scrapy Training companion code

Related Searches

Scraper Web Crawler (1,388)

101-200 of 1,485 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.