Awesome Open Source

Programming Languages

Search results for web crawler

1,484 search results found

Python_crawler ⭐ 20

Python-driven web crawler and scraper. Uses BeautifulSoup to gather all URLs from a target page, and initiates a crawl from a start URL, considering Whitelist/Blacklist criteria that are populated in crawl.py

Kindlefy ⭐ 20

📑 A way to automatically sync data with your kindle, such as RSS feeds, manga, and too much more.

Scrapegen ⭐ 20

A simple python tool that generates a requests/bs4 based web scraper

Hepsiburada Review Scraper ⭐ 20

Hepsiburada review/comment and rating scraper. Turkish text dataset creator for data science and NLP projects. 📜

Halfstaff ⭐ 20

🇺🇸 Is the US flag at half-staff?

Py Parsehub ⭐ 20

Python API for parsehub.com web scraping service

Uoft Timetable Generator ⭐ 20

A web application that generates timetables for university students at the University of Toronto

Web Scraping With Selenium ⭐ 20

In this guide on how to web scrape with Selenium, we will be using Python 3. The code should work with any version of Python above 3.6

Browser Automation Api ⭐ 20

Browser automation API for repetitive web-based tasks, with a friendly user interface. You can use it to scrape content or do many other things like capture a screenshot, generate pdf, extract content or execute custom Puppeteer, Playwright functions.

Covid19 Api ⭐ 20

Covid19 Data API (JSON) - LIVE

Reverse Twitter Scraper ⭐ 20

A request based package for scraping twitter data. No API Key required. Support for proxies and private twitter accounts

Chegg Solutions Saver ⭐ 20

Because Chegg only gave few days to access their solutions for free when I rent their books so I decided to create a web crawler to save all the needed solutions offline.

Web Scraper ⭐ 20

Web Scraper is a melange of Web tools for web hacking, reconnaissance, bug bounty so on. This tool consists of 20 most used web tools for security assessment

Society Email Scrape ⭐ 20

Scrapes Every Email Address of Every Society in Every University

Mexican Jobs 2020 ⭐ 20

Data ETL & Analysis on thousands of job listings from the official Mexican job board (2020 edition).

Seo Keyword Research Tool ⭐ 20

Python SEO keywords suggestion tool. Google Autocomplete, People Also Ask and Related Searches.

Scrapevin ⭐ 19

A python package for scraping vinted - all foreign versions aswell!

Bvcscrap ⭐ 19

Python library to scrape financial data from Casablanca Stock Exchange(Bourse des Valeurs de Casablanca)

Trackpurchase ⭐ 19

단 몇줄의 코드로 다양한 쇼핑 플랫폼에서 결제 내역을 긁어오자!

Frequent ⭐ 19

A utility for crawling websites and building frequency lists of words

Docker Crawler ⭐ 19

Parsemycf Contest ⭐ 19

A personal submission codeforces parser for CF, parsed by individual contests.The user is prompted for the username and has the flexibilty to parse last 'n' contests he participated in CF.

Restaurant Finder Featurereviews ⭐ 19

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

Words Scraper ⭐ 19

Selenium based web scraper to generate passwords list

A parallelised web crawler in Go

Browser extension to simulate browsing behaviour in search engines.

Python Webcrawler ⭐ 19

A web crawler written in Python

Tagalog Dictionary Scraper ⭐ 19

Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com

📔 教育部《重編國語辭典修訂本》網路爬蟲 :: A live web crawler for the Chinese-Chinese dictionary published by the Ministry of Education in Taiwan

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

Scrapy Azuresearch Crawler Samples ⭐ 19

Scrapy as a Web Crawler for Azure Search Samples

Machinera 2020 ⭐ 19

This is an AI Series where we will cover Machine Learning and Deep Learning topics from the very basics.

Openai Scraper ⭐ 18

This is a template repository for building a web scraper with OpenAI support. The repository provides a basic project structure with TypeScript and Puppeteer pre-configured, as well as OpenAI's GPT-3 API integration. With this template, you can easily build a scraper that uses machine learning to analyze and extract insights from the scraped data.

Random_proxies ⭐ 18

Python package to generate random proxy on the fly!

Amazon Product Scraping ⭐ 18

Web scraping amazon products based on user search using Selenium Webdriver.

Essaygen ⭐ 18

Essay generator

Depth controllable Web scraper and Sitemap Generator in Go

Webcrawlerpacktpublishing ⭐ 18

Sourcegrade ⭐ 18

A web scraper for www.gradesource.com built with MongoDB, Express.js, Node.js, React and Redux

Pipe2time.ir ⭐ 18

Web Crawler for Time.ir to Retrive JSON File, jalali, qamari, miladi JSON Calendar API.

Stackoverflow Crawler ⭐ 18

It is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.

Deepcrawler ⭐ 18

Deep learning based Smart Web Crawler

Weibospider ⭐ 18

A Web Spider for Weibo(Chinese Twitter)

Sinaweiboscraper ⭐ 18

Web Scraper for Sina Weibo Search by Keywords

Smart Url Fuzzer ⭐ 18

Explore URLs of domains fast and efficiently using fuzzing techniques

Pagination With Python ⭐ 18

A guide on how to deal with pagination via Python.

Gpt2_episode_summary_generator ⭐ 18

Utilizing webscraping and state-of-the-art NLP to generate TV show episode summaries.

A web scraper and discord bot for osu! mapping/modding related event feeds.

Pubcrawl ⭐ 18

*Deprecated* A short and sweet Python web crawler using Redis as the process queue, seen set and Memcache style rate limiter for robots.txt

Biblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

Scrapeium ⭐ 18

Scrapeium is a powerful query language for scraping the web

Tweetscraper ⭐ 17

A simple script to scrape a list of accounts or single account from Twitter into a neatly formatted Excel file.

Sleipnir ⭐ 17

A simple, performant web-crawler for clojure

Parsel Cli ⭐ 17

cli for evaluating css and xpath selectors

Anikimiapi ⭐ 17

A Simple, LightWeight, Statically-Typed Python3 API wrapper for GogoAnime.

Mdpi_explorer ⭐ 17

A simple package to explore MDPI´s articles by journal. A series of functions help to obtain lists of papers, obtain data from them (turnaround times, special issues and articles types) and create summary graphs.

Lambda Web Scraper Example ⭐ 17

An example of a web scraper running on AWS Lambda and Lambda Layers

Dark_characters_network ⭐ 17

A web crawler that collects data about characters from Netflix's Dark serie

Super Anime Downloader ⭐ 17

A program which takes an Anime name or URL and downloads the specified range of episodes.

Json Web Crawler ⭐ 17

Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.

Artemis Arrow ⭐ 17

Iwata Asks Downloader ⭐ 17

Tool to download Iwata Asks interviews (none of which are stored in this repo)

Web Crawler created with Node.js and Puppeteer

Imd Bolsas Bot ⭐ 17

Bot de alimentação do IMD Bolsas no Telegram e Twittter. Seu objetivo é alertar estudantes do instituto sobre bolsas e vagas de estágio.

Beer For Python ⭐ 17

[DEPRECATED] A simple web scraper for Beer Advocate

Fanfiction Manager ⭐ 17

Provides extreme flexibility with the help of Rule system to power-users* in downloading, tracking-status & reading novels from ANY site they want.

Stravaweblib ⭐ 17

Extends the Strava v3 API using web scraping

Wcofun.com_downloader ⭐ 17

A java & Kotlin cartoon and anime downloader for https://www.wcofun.com/

Aniscrape ⭐ 17

Web scraping tool to download videos from TürkAnimeTV and AnimeciX

Raspagem De Dados Fatec ⭐ 17

📓 Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

Youtube Scraping Selenium ⭐ 17

Automatically creates a Youtube channel dashboard

Amazon_tracker ⭐ 17

A simple amazon tracker that sends you an email when prices of your followed products fall down!

Amelia_2.0 ⭐ 17

An Artificial Intelligence Chat Bot and Service Provider written in Python and AIML.

Comic Scraper ⭐ 16

[Python] Scraps comics and manga from various websites and creates cbz files from them

Coursedownloader ⭐ 16

GUI app for downloading whole online courses with folder structure from one url

Bsctokenscanner ⭐ 16

Pulls Binance Smart Chain feed of newly-verified contracts every 30 seconds, then checks their contract code for links to socials.Returns only those with socials information included, and then submits the contract address to TokenSniffer to evaluate contract legitimacy

Mls Real Estate Scraper For Realtor.ca ⭐ 16

Python MLS and Real-Estate Data Scraper for the Realtor.ca Website

Campaign Critic ⭐ 16

A web app that helps Kickstarter creators increase their chances of being funded

Request_curl ⭐ 16

user friendly wrapper for pycurl

Scrape Interwebz ⭐ 16

Tutorial on extracting data via APIs and webscraping

Spiderman ⭐ 16

your friendly neighborhood web crawler

Notes for learning and applying R to questions about crime and the justice system

Web_scrapers_ruby ⭐ 16

Samples of Web Scraper using Mechanize, Delayed Job, Parallel, Mongo, etc.

Pycon_2017 ⭐ 16

Fantastic Data and Where To Find Them: An introduction to APIs, RSS, and Scraping

Ted Scraper ⭐ 16

💬 TED Talks web scraper

Js Crawler ⭐ 16

A short and simple python crawler, that uses Webkit and executes Javascript

Stock Market Predictor ⭐ 16

Stock Market Predictor with LSTM network. Web scraping and analyzing tools (ohlc, mean)

Selenider ⭐ 16

Concise, Lazy and Reliable Wrapper for 'chromote' and 'selenium'

Gumtree Scraper ⭐ 16

Are you looking for an apartment on gumtree.pl? You will like this a lot.

Mapping sites to AniList and back.

A Dart package to web scraping data from websites easily and faster using less code lines.

Twitter Source Bot ⭐ 16

Ever wanted to know the source of a tweet? Just @whosaidthis_bot and I'll tell you where it came from

a web scraper which can find movies ,series ,software ,etc.. direct links and download them

Covid News ⭐ 16

A data engineering personal project for applying some of my skills

Roguelike Universe ⭐ 16

Understanding game design inspiration of roguelike games via web scraping and network analysis.

Quickscraper ⭐ 16

An R package wrapping quickscrape, a node.js web scraper

Spring Boot Web Scraper ⭐ 16

Simple web scrapping app made using Spring Boot + Thymeleaf + Jsoup + Java 8 Lambdas & Streams

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

Web Scraping Api ⭐ 15

Web Scraping API code examples for Python, PHP and Node.js

Related Searches

Scraper Web Crawler (1,388)

601-700 of 1,484 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.