Awesome Open Source

Programming Languages

The Top 23 Spider Open Source Projects

Open source projects categorized as Spider

Categories > Data Processing > Spider

Edit Category

Dumpall ⭐ 865

一款信息泄漏利用工具，适用于.git/.svn/.DS_Store泄漏和目录列出

total releases 7most recent commit a year ago

Wechat_spider ⭐ 863

使用“代理”的方式来抓取微信公众账号文章，可以抓取阅读数、点赞数，基于 anyproxy。

most recent commit 4 years ago

Funpyspidersearchengine ⭐ 862

Word2vec 千人千面个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

most recent commit 2 years ago

Zhihu_spider ⭐ 855

most recent commit 8 years ago

Spider Crack Js ⭐ 845

破解各种网站的JS [知乎] [豆瓣][Incapsula-CDN][百度贴吧_BSK][七麦analysis]

most recent commit 3 years ago

Zhihu Crawler ⭐ 843

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬

most recent commit 5 years ago

Overlappingmarkerspiderfier ⭐ 811

Deals with overlapping markers in Google Maps JS API v3, Google Earth-style

most recent commit a year ago

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

total releases 7most recent commit 6 months ago

Icrawler ⭐ 792

A multi-thread crawler framework with many builtin image crawlers provided.

dependent packages 5total releases 42most recent commit 4 months ago

pypi icrawler} Downloads

Crawly, a high-level web crawling & scraping framework for Elixir.

total releases 18most recent commit 4 months ago

Android Charts ⭐ 787

Open-source native Android graph/chart framework includes line chart,stick chart,candlestick chart,pie chart,spider-web chart etc.

most recent commit 7 years ago

Manspider ⭐ 779

Spider entire networks for juicy files sitting on SMB shares. Search filenames or file content - regex supported!

total releases 5most recent commit 6 months ago

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

dependent packages 16total releases 26most recent commit 3 months ago

rubygems spidr} Downloads

Baiduimagespider ⭐ 774

一个超级轻量的百度图片爬虫

most recent commit a year ago

👉一个基于spring boot 实现的java股票爬虫(仅支持A股)，如果你❤️请⭐ . V2升级版正在开发中！

most recent commit a year ago

Baiduyunspider ⭐ 769

百度云网盘搜索引擎，包含爬虫 & 网站

most recent commit 5 years ago

Creeper ⭐ 762

🐾 Creeper - The Next Generation Crawler Framework (Go)

total releases 1most recent commit 7 years ago

Spider_collection ⭐ 754

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos

most recent commit 6 months ago

Core Scrapy ⭐ 753

python-scrapy demo

most recent commit 6 years ago

Programmable spidering of web sites with node.js and jQuery

dependent packages 3total releases 2most recent commit 5 years ago

npm spider} Downloads

Qq Groups Spider ⭐ 733

QQ Groups Spider（QQ 群爬虫）

most recent commit 6 years ago

Spider_python ⭐ 732

most recent commit a year ago

Zhihu Spider ⭐ 719

A web spider for zhihu.com

most recent commit 9 years ago

The 10 Most Depended On Spider Open Source Projects

Penjabarberita ⭐ 16

Extract the article list from its raw news HTML

dependent packages 2total releases 7latest release July 15, 2022most recent commit a year ago

npm @ariya/penjabarberita} Downloads

Webpalm ⭐ 295

WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.

dependent packages 2total releases 8latest release June 19, 2023most recent commit 6 months ago

Migale was born out of a need to extract data quickly and with a very low development cost. This package is not intended to replace complete and structured libraries like DotnetSpider.

dependent packages 2total releases 4latest release January 30, 2023most recent commit a year ago

nuget Migale.Core} Downloads

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months ago

npm webster} Downloads

Password Stretcher ⭐ 17

FETCH THE PASSWORD STRETCHER

dependent packages 1total releases 7latest release February 18, 2022most recent commit 2 years ago

pypi password-stretcher} Downloads

Webcrawler Verifier ⭐ 27

Java library providing functionality to verify that user-agents are who they claim to be.

dependent packages 1total releases 1latest release April 05, 2015most recent commit 7 years ago

Scrapy Rotating Proxies ⭐ 474

use multiple proxies with Scrapy

dependent packages 1total releases 13latest release May 25, 2019most recent commit 3 years ago

pypi scrapy-rotating-proxies} Downloads

Tarantula ⭐ 453

a big hairy fuzzy spider that crawls your site, wreaking havoc

dependent packages 1total releases 13latest release May 24, 2013most recent commit 11 years ago

rubygems tarantula} Downloads

Scrapy Rotated Proxy ⭐ 13

A scrapy middleware to use rotated proxy ip list.

dependent packages 1total releases 13latest release August 23, 2018most recent commit 6 years ago

pypi scrapy-rotated-proxy} Downloads

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

dependent packages 1total releases 1latest release December 25, 2017most recent commit 4 years ago

The 10 Latest Releases In Spider Open Source Projects

Oceanmonkey ⭐ 5

OceanMonkey is a High-Level Distributed Web Crawling and Web Scraping framework base on multi-process and multi-coroutines, used to crawl websites and extract structured data from their pages like the classical scrapy framework.

total releases 1latest release January 04, 2022most recent commit 2 years ago

Net Spider ⭐ 6

A graph database middleware to maintain a time-varying graph

total releases 20latest release January 02, 2022most recent commit 2 years ago

Wechat_articles_spider ⭐ 2,234

微信公众号文章的爬虫

total releases 29latest release December 31, 2021most recent commit 7 months ago

Scrapy Scrapingbee ⭐ 26

JavaScript support and proxy rotation for Scrapy with ScrapingBee.

total releases 5latest release December 26, 2021most recent commit 2 years ago

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

dependent packages 1total releases 17latest release December 25, 2021most recent commit 5 months ago

Distributed web scraping framework

dependent packages 2total releases 9latest release December 15, 2021most recent commit 2 years ago

npm aragog-client} Downloads

Weibospider ⭐ 7,444

新浪微博爬虫，用python爬取新浪微博数据

total releases 26latest release December 14, 2021most recent commit 5 months ago

Scrapyscript ⭐ 92

Run a Scrapy spider programmatically from a script or a Celery task - no project required.

total releases 10latest release December 11, 2021most recent commit 2 years ago

Lrabbit_scrapy ⭐ 73

a quick start python mutil thread crawl

total releases 13latest release December 10, 2021most recent commit 2 years ago

Glyphhanger ⭐ 47

Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.

dependent packages 12total releases 19latest release December 06, 2021most recent commit 3 years ago

npm glyphhanger} Downloads

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.