Awesome Open Source

Programming Languages

Search results for scraper crawler

300 search results found

Crawling, scraping and indexing application written in Clojure.

Recipe Scraper ⭐ 35

A library for scraping recipes from popular recipe sites.

Scrapy Kafka Redis ⭐ 35

Distributed crawling/scraping, Kafka And Redis based components for Scrapy

Cobweb Lnx ⭐ 34

CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

Scraping Assisted by Learning

Web scraping engines with Python and Scrapy

Chatgpt Line Bot ⭐ 33

🤖Free ChatGPT Line Bot with Horoscope, Music Broadcast, Google Image Search...

Go Crawler ⭐ 33

A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middleware and supports various parsing and storage methods. Additionally, it supports distributed deployment. 基于golang实现的爬虫框架，编写简单，性能强劲。内置了丰富的实用中间件，支持多种解析、保存方式，

Noscrape ⭐ 32

obfuscate text via node to make scraping your content really difficult

Chromium / Puppeteer site crawler

Extension ⭐ 32

web scraping extension

A web crawler / scraper engine written in Golang

Python Searchengine ⭐ 31

A simple search engine which utilizes whoosh, mongodb, a custom html scraper and simple crawler.

Sneakpeek ⭐ 31

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

This was the night of the crawling terror!

Bitcointalk Scraper ⭐ 31

Python-based scraper / crawler for members and messages on bitcointalk.org

Scrapeulous ⭐ 30

Cloud crawler functions for scrapeulous

Cygnusx1 ⭐ 30

A multithreaded tool for searching and downloading images from popular search engines. It is straightforward to set up and run!

Scrapy Zyte Api ⭐ 30

Zyte API integration for Scrapy

Spidyquotes ⭐ 30

Example site for web scraping tutorials

Iclr2023 Openreviewdata ⭐ 30

Crawl & Visualize ICLR 2023 Data from OpenReview

Python Web Scraping Tutorial ⭐ 29

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

Stormscraper ⭐ 29

A Storm based web crawler with Cassandra backend

Scrape Github Trending ⭐ 29

Tutorial for web scraping / crawling with Node.js.

Pttminer ⭐ 28

Parallel Searching and Crawling Data from PTT 🚀

Actor Amazon Crawler ⭐ 28

Amazon crawler - this configuration will extract items for a keywords that you will specify in the input, and it will automatically extract all pages for the given keyword. You can specify more keywords on the input for one run.

Selenium Friends Scraper ⭐ 28

Python code that simulates the activity of crawling Facebook to get friends of friends network without using the API.

Web Scraping Framework

Python Scrapilicious ⭐ 28

Unmaintained: A horridly implemented scrapy app that will scrape all (?) of Delicious' bookmarks.

Colly Sqlite3 Storage ⭐ 27

A SQLite3 storage back end for the Colly web crawling/scraping framework https://go-colly.org

Webtranspose ⭐ 27

Web scraping API for building AI applications.

Novelsave_sources ⭐ 27

A collection of webnovel sources offering varying amounts of scraping capability.

Trollhunter ⭐ 27

Twitter Troll & Fake News Hunter - Crawls news websites and twitter to identify fake news

A web spider for shodan.io without using the Developer API.

Nest Crawler ⭐ 26

An easiest crawling and scraping module for NestJS

Video Crawler ⭐ 26

Crawl websites for videos from Youtube, Vimeo, Soundcloud, etc

Webmagician Ui ⭐ 26

An admin UI project for a configurable web crawler platform

Scrapingant Client Python ⭐ 26

ScrapingAnt API client for Python.

Estate Crawler ⭐ 25

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

Telegram Groups Crawler ⭐ 25

A Telegram crawler made in Python to automatically search groups and channels and collect any type of data from them.

Simple Crawler ⭐ 25

A super simple webcrawler framework written in Python.

Instagram Downloader ⭐ 24

Instagram user's photos and videos downloader. Download all media files from any username. Working 2021!

Pinscrape ⭐ 24

A simple library to scrape Pinterest images written in Python

Mimo Crawler ⭐ 24

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Download public videos on TikTok using Python with Selenium

Crawlkit ⭐ 23

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

🕷Configuration based html scraper

Scrapy Mosquitera ⭐ 23

Restrict crawl and scraping scope using matchers.

Bluebird ⭐ 23

Unofficial Python client for Twitter

Feedsearch Crawler ⭐ 23

Crawl sites for RSS, Atom, and JSON feeds.

Dijnet Bot ⭐ 22

Az összes számlád még egy helyen :)

Ferret Server ⭐ 22

Advanced declarative web scraping

Exoskeleton ⭐ 22

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

scraper for facebook, gab, google and tiktok

An open source web crawling platform

Crawling Framework ⭐ 21

Easily crawl news portals or blog sites using Storm Crawler.

Reason Rust Scraper ⭐ 21

🦀 Scraping & crawling websites using Rust, and ReasonML

Proxycrawl Node ⭐ 20

ProxyCrawl Node library for scraping and crawling

Anime Tracker ⭐ 20

🕸️ All in one place to track your favorite animes

Darkspider ⭐ 20

Anatomy and Visualization of the Network structure of the Dark web using multi-threaded crawler

Actor Youtube Scraper ⭐ 20

Apify actor to scrape Youtube search results. You can set the maximum videos to scrape per page as well as the date from which to start scraping.

Scrapy Azuresearch Crawler Samples ⭐ 19

Scrapy as a Web Crawler for Azure Search Samples

Webscraper ⭐ 19

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Kimurai is a modern web scraping framework written in Ruby which works out of box with headless chromium/firefox, phantomjs, or simple HTTP requests and allows to scrape and interact with javascript rendered websites

Parsemycf Contest ⭐ 19

A personal submission codeforces parser for CF, parsed by individual contests.The user is prompted for the username and has the flexibilty to parse last 'n' contests he participated in CF.

Proxycrawl Php ⭐ 19

ProxyCrawl PHP library for scraping and crawling websites

⚡️An API for downloading or streaming your favorite anime.

Flixhq Core ⭐ 18

Nodejs library that provides an Api for obtaining the movies information from FlixHQ website.

Marvel Snap Scrapr ⭐ 18

Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

Inmet Api Temperature ⭐ 18

Crawler dos dados metereológicos de estações convencionais do INMET (BDMEP)

Web Crawler ⭐ 18

Python Web Crawler with Selenium and PhantomJS

Istanbul Transportation Network ⭐ 18

Istanbul Transportation Network Scraping & Analysis

Youtube_scraper ⭐ 18

Scrape data about an entire Channel or just a Playlist, or get stats about your Own Watch History.

Chan Downloader ⭐ 18

CLI to download all images/webms in a 4chan thread

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

Hide your IP with free proxies using Froxy 🔄

Dedomeno ⭐ 18

Dedomeno: A Spanish real estate (Idealista) python scraper

Web Crawler created with Node.js and Puppeteer

Google_news_scraper_and_sentiment_analyzer ⭐ 17

Downloads news articles from Google news and uses pre-trained NLP models to perform sentiment analysis

Flipkart Scraper ⭐ 17

Python is going to bite the flipkart soon... :) This application does 'night watchman' job for flipkart.com

Structominer ⭐ 17

Data scraping for a more civilized age

Asyncio web crawling framework. Work in progress.

Soccer Scrape ⭐ 16

📃 Scrape football data from Bet365

Go Scrapy ⭐ 16

Web crawling and scraping framework for Golang

Easily fetch, slice, dice, and output HTML content from remote pages in your CraftCMS templates.

Penjabarberita ⭐ 16

Extract the article list from its raw news HTML

Scrapy Crawl Asp ⭐ 16

heavy-duty scraping framework for crawling ASP.net pages

Django Scraper ⭐ 16

Django application which crawls and downloads online content following instructions

Reddit Post Exporter ⭐ 15

Export desired amount of posts from specified subreddit and category/sort without any API wrappers

Spidey Mongo ⭐ 15

Implements a MongoDB back-end for Spidey (https://github.com/joeyAghion/spidey), a framework for crawling and scraping web sites.

Product Integrations ⭐ 15

Code examples and general information

Android Apps Downloader ⭐ 15

📱 A tool to download android apps from Google Play Store and Xiaomi App Store (the famous Chinese Store).

Papercut ⭐ 15

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Mailinglistscraper ⭐ 15

A python web scraper for public email lists.

Instabot ⭐ 15

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Web Scraper ⭐ 15

Crawl and scrape dynamic Web sites. Scrape Web sites that dynamically load content or sites that render their HTML using JavaScipt.

Web Scraper Gcp ⭐ 15

Scrape all the pages and links of a given domain and write the results to Google Cloud BigQuery.

A PasteBin scrapper that doesnt rely on the PasteBin scrape API

Dotnetthirdpartynotices ⭐ 15

A .NET tool to generate file with third party legal notices

Zhihu_crawler ⭐ 15

本程序支持关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取

Related Searches

Python Crawler (4,545)

Python Scraper (3,513)

Javascript Scraper (2,047)

Scraper Scrape (1,534)

Scraper Web Crawler (1,528)

Javascript Crawler (1,142)

Crawler Spider (1,044)

Crawler Scrapy (1,002)

Java Crawler (806)

Html Scraper (757)

201-300 of 300 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.