Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Framework Crawler Open Source Projects

Open source projects categorized as Framework Crawler
Categories > Data Processing > Crawler
Categories > Frameworks > Framework
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit 3 months agopypi Scrapy} Downloads
Colly ⭐ 21,902
Elegant Scraper and Crawler Framework for Golang
dependent packages 328total releases 22most recent commit a month ago
Webmagic ⭐ 11,080
A scalable web crawler framework for Java.
dependent packages 22total releases 25most recent commit 3 months ago
Grab ⭐ 2,292
Web Scraping Framework
dependent packages 3total releases 120most recent commit 10 months agopypi grab} Downloads
Gain ⭐ 1,972
Web crawling framework based on asyncio.
total releases 5most recent commit 5 years ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42most recent commit 4 months agopypi icrawler} Downloads
Creeper ⭐ 762
🐾 Creeper - The Next Generation Crawler Framework (Go)
total releases 1most recent commit 7 years ago
Catgate ⭐ 681
CatGate is a small crawler framework based on Chrome extension . CatGate是一个基于浏览器插件的数据抓取工具。做成浏览器插件无需模拟登入，能最真实的模仿用户行为
most recent commit 7 years ago
Staticgen ⭐ 668
Static website generator that lets you use HTTP servers and frameworks you already know
total releases 3most recent commit 3 years ago
Nodejs Stuff ⭐ 484
Node.js libs I want to keep in mind.
most recent commit 4 years ago
Webster ⭐ 465
a reliable high-level web crawling & scraping framework for Node.js.
dependent packages 2total releases 42most recent commit 5 months agonpm webster} Downloads
Dude ⭐ 397
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
total releases 43most recent commit 3 months ago
Sasila ⭐ 264
一个灵活、友好的爬虫框架
total releases 17most recent commit 4 years ago
Rendler ⭐ 247
A rendering web crawler for Apache Mesos.
most recent commit 5 years ago
Web Bee ⭐ 186
🐝 Web vertical crawler framework for fun
most recent commit 4 months ago
Antch ⭐ 177
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
dependent packages 1total releases 1most recent commit 4 years ago
Jlitespider ⭐ 151
A lite distributed Java spider framework :-)
total releases 3most recent commit 7 years ago
Aioscpy ⭐ 138
An asyncio + aiolibs crawler imitate scrapy framework
total releases 46most recent commit 5 months ago
Httpcode.core ⭐ 134
简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。
dependent packages 3total releases 2most recent commit 5 years agonuget HttpCode.Core} Downloads
Abotx ⭐ 106
Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
most recent commit 7 months ago
Krawler ⭐ 96
A web crawling framework written in Kotlin
most recent commit 4 years ago
Ant_nest ⭐ 93
Simple, clear and fast Web Crawler framework build on python3.6+, powered by asyncio.
dependent packages 1total releases 59most recent commit 2 years agopypi ant-nest} Downloads
Bots Zoo ⭐ 90
most recent commit 5 months ago
The 10 Most Depended On Framework Crawler Open Source Projects
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Colly ⭐ 21,902
Elegant Scraper and Crawler Framework for Golang
dependent packages 328total releases 22latest release March 08, 2022most recent commit a month ago
Webmagic ⭐ 11,080
A scalable web crawler framework for Java.
dependent packages 22total releases 25latest release September 10, 2023most recent commit 3 months ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 4 months agopypi icrawler} Downloads
Httpcode.core ⭐ 134
简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。
dependent packages 3total releases 2latest release September 04, 2017most recent commit 5 years agonuget HttpCode.Core} Downloads
Grab ⭐ 2,292
Web Scraping Framework
dependent packages 3total releases 120latest release June 24, 2018most recent commit 10 months agopypi grab} Downloads
Aragog ⭐ 9
Distributed web scraping framework
dependent packages 2total releases 9latest release December 15, 2021most recent commit 2 years agonpm aragog-client} Downloads
Scruber ⭐ 6
Ruby crawling framework
dependent packages 2total releases 9latest release September 06, 2018most recent commit 6 years agorubygems scruber} Downloads
Webster ⭐ 465
a reliable high-level web crawling & scraping framework for Node.js.
dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months agonpm webster} Downloads
Spider ⭐ 27
A web spider framework
dependent packages 2total releases 1latest release October 21, 2020most recent commit 4 months agonpm gz-spider} Downloads
The 10 Latest Releases In Framework Crawler Open Source Projects
Webster ⭐ 465
a reliable high-level web crawling & scraping framework for Node.js.
dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months agonpm webster} Downloads
Spider_man ⭐ 15
SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.
total releases 23latest release October 24, 2023most recent commit 6 months ago
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Webmagic ⭐ 11,080
A scalable web crawler framework for Java.
dependent packages 22total releases 25latest release September 10, 2023most recent commit 3 months ago
Dude ⭐ 397
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
total releases 43latest release August 12, 2023most recent commit 3 months ago
Icrawler ⭐ 792
A multi-thread crawler framework with many builtin image crawlers provided.
dependent packages 5total releases 42latest release June 30, 2023most recent commit 4 months agopypi icrawler} Downloads
Aioscpy ⭐ 138
An asyncio + aiolibs crawler imitate scrapy framework
total releases 46latest release May 17, 2023most recent commit 5 months ago
Tegenaria ⭐ 8
Tegenaria is a crawler framework based on golang
total releases 27latest release March 23, 2023most recent commit a year ago
Merlin Framework ⭐ 16
Merlin - migration framework
total releases 9latest release October 28, 2022most recent commit 5 months ago
Scrapio ⭐ 16
Asyncio web crawling framework. Work in progress.
total releases 9latest release July 09, 2022most recent commit 9 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.