Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Superset | 51,096 | 2 | 15 hours ago | 3 | April 29, 2022 | 1,311 | apache-2.0 | TypeScript | ||
Apache Superset is a Data Visualization and Data Exploration Platform | ||||||||||
Akshare | 6,193 | 10 | 16 hours ago | 1,593 | July 06, 2022 | 1 | mit | Python | ||
AKShare is an elegant and simple financial data interface library for Python, built for human beings! 开源财经数据接口库 | ||||||||||
Deeplake | 5,331 | 52 | 1 | 18 hours ago | 149 | June 28, 2022 | 65 | mpl-2.0 | Python | |
Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai | ||||||||||
Igel | 2,964 | a year ago | 34 | November 19, 2021 | 5 | mit | Python | |||
a delightful machine learning tool that allows you to train, test, and use models without writing code | ||||||||||
Fiftyone | 2,700 | 2 | 15 hours ago | 81 | June 24, 2022 | 432 | apache-2.0 | Python | ||
The open-source tool for building high-quality datasets and computer vision models | ||||||||||
Gopup | 2,206 | 4 months ago | 38 | May 15, 2021 | 59 | Python | ||||
数据接口:百度、谷歌、头条、微博指数,宏观数据,利率数据,货币汇率,千里马、独角兽公司,新闻联播文字稿,影视票房数据,高校名单,疫情数据… | ||||||||||
Datascience Pizza | 2,199 | 5 months ago | 9 | mpl-2.0 | ||||||
🍕 Repositório para juntar informações sobre materiais de estudo em análise de dados e áreas afins, empresas que trabalham com dados e dicionário de conceitos | ||||||||||
Whylogs | 2,134 | 15 hours ago | 20 | March 09, 2022 | 20 | apache-2.0 | Jupyter Notebook | |||
The open standard for data logging | ||||||||||
Diffgram | 1,632 | 7 days ago | 379 | other | Python | |||||
Integrate Human Supervision into your Platform. For all Training Data Types, Image, Video, 3D, Text, Geo, Audio, Compound, Grid, LLM, GPT, Conversational, and more. | ||||||||||
Codesearchnet | 1,548 | a year ago | 7 | mit | Jupyter Notebook | |||||
Datasets, tools, and benchmarks for representation learning of code. |
[ English | Français | 简体中文 | Türkçe | 한글 | Bahasa Indonesia] | Русский]
Note: the translations of this document may not be up-to-date. For the latest version, please check the README in English.
Software 2.0 needs Data 2.0, and Hub delivers it. Most of the time Data Scientists/ML researchers work on data management and preprocessing instead of training models. With Hub, we are fixing this. We store your (even petabyte-scale) datasets as single numpy-like array on the cloud, so you can seamlessly access and work with it from any machine. Hub makes any data type (images, text files, audio, or video) stored in cloud usable as fast as if it were stored on premise. With same dataset view, your team can always be in sync.
Hub is being used by Waymo, Red Cross, World Resources Institute, Omdena, and others.
Visualization of a dataset uploaded to Hub via app.activeloop.ai (free tool).
Work with public or your own data, locally or on any cloud.
To load a public dataset, one needs to write dozens of lines of code and spend hours accessing and understanding the API as well as downloading the data. With Hub, you only need 2 lines of code, and you can get started working on your dataset in under 3 minutes.
pip3 install hub
To be able to download datasets stores on hub platform, you would need to login: You can register a free account at Activeloop and authenticate locally:
hub register
hub login
# alternatively, add username and password as arguments (use on platforms like Kaggle)
hub login -u username -p password
Access public datasets in Hub by following a straight-forward convention which merely requires a few lines of simple code. Run this excerpt to get the first thousand images in the MNIST database in the numpy array format:
from hub import Dataset
mnist = Dataset("activeloop/mnist") # loading the MNIST data lazily
# saving time with *compute* to retrieve just the necessary data
mnist["image"][0:1000].compute()
You can find all the other popular datasets on app.activeloop.ai.
Load the data and train your model directly. Hub is integrated with PyTorch and TensorFlow and performs conversions between formats in an understandable fashion. Take a look at the example with PyTorch below:
from hub import Dataset
import torch
mnist = Dataset("activeloop/mnist")
# converting MNIST to PyTorch format
mnist = mnist.to_pytorch(lambda x: (x["image"], x["label"]))
train_loader = torch.utils.data.DataLoader(mnist, batch_size=1, num_workers=0)
for image, label in train_loader:
# Define your training loop here
If you want to work on your own data locally, you can start by creating a dataset:
from hub import Dataset, schema
import numpy as np
ds = Dataset(
"./data/dataset_name", # file path to the dataset
shape = (4,), # follows numpy shape convention
mode = "w+", # reading & writing mode
schema = { # named blobs of data that may specify types
# Tensor is a generic structure that can contain any type of data
"image": schema.Tensor((512, 512), dtype="float"),
"label": schema.Tensor((512, 512), dtype="float"),
}
)
# filling the data containers with data (here - zeroes to initialize)
ds["image"][:] = np.zeros((4, 512, 512))
ds["label"][:] = np.zeros((4, 512, 512))
ds.flush() # executing the creation of the dataset
You can also specify s3://bucket/path
, gcs://bucket/path
or azure path. Here you can find more information on cloud storage.
Also, if you need a publicly available dataset that you cannot find in the Hub, you may file a request. We will enable it for everyone as soon as we can!
Register a free account at Activeloop and authenticate locally:
activeloop register
activeloop login
# alternatively, add username and password as arguments (use on platforms like Kaggle)
activeloop login -u username -p password
Then create a dataset, specifying its name and upload it to your account. For instance:
from hub import Dataset, schema
import numpy as np
ds = Dataset(
"username/dataset_name",
shape = (4,),
mode = "w+",
schema = {
"image": schema.Tensor((512, 512), dtype="float"),
"label": schema.Tensor((512, 512), dtype="float"),
}
)
ds["image"][:] = np.zeros((4, 512, 512))
ds["label"][:] = np.zeros((4, 512, 512))
ds.flush()
Access it from anywhere else in the world, on any device having a command line:
from hub import Dataset
ds = Dataset("username/dataset_name")
For more advanced data pipelines like uploading large datasets or applying many transformations, please refer to our documentation.
The examples directory has a series of examples and the notebooks has some notebooks with use cases. Some of the notebooks are listed of below.
Notebook | Description | |
---|---|---|
Uploading Images | Overview on how to upload and store images on Hub | |
Uploading Dataframes | Overview on how to upload Dataframes on Hub | |
Uploading Audio | Explains how to handle audio data in Hub | |
Retrieving Remote Data | Explains how to retrieve Data | |
Transforming Data | Briefs on how data transformation with Hub | |
Dynamic Tensors | Handling data with variable shape and sizes | |
NLP using Hub | Fine Tuning Bert for CoLA | |
Getting Started with Text on Hub | Overview on using Text datasets in Hub |
There are quite a few dataset management libraries which offer functionality that might seem similar to Hub. In fact, quite a few users migrate data from PyTorch or Tensorflow Datasets to Hub. Here are a few startling differences you will encounter after switching to Hub:
Join our Slack community to get help from Activeloop team and other users, as well as stay up-to-date on dataset management/preprocessing best practices.
We'd love your feedback by completing our 3-minute survey.
on Twitter.
As always, thanks to our amazing contributors!
Made with contributors-img.
Please read CONTRIBUTING.md to know how to get started with making contributions to Hub.
Activeloop's Hub format lets you achieve faster inference at a lower cost. We have 30+ popular datasets already on our platform. These include:
Check these and many more popular datasets on our visualizer web app and load them directly for model training!
Using Hub? Add a README badge to let everyone know:
[](https://github.com/activeloopai/Hub)
By default, we collect anonymous usage data using Bugout (here's the code that does it). It only logs Hub library's own actions and parameters, and no user/ model data is collected.
This helps the Activeloop team to understand how the tool is used and how to deliver maximum value to the community by building features that matter to you. You can easily opt-out of usage tracking during login.
Similarly to other dataset management packages, Hub
is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.
If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!
This technology was inspired from our experience at Princeton University and would like to thank William Silversmith @SeungLab with his awesome cloud-volume tool. We are heavy users of Zarr and would like to thank their community for building such a great fundamental block.