Envision the Edge like never before...

OpenIssues

EnvisEdge allows users to simulate an edge computing environment to test their ideas and models before putting them in place on the edge. It takes care of all the complex stuff such as diversity across operating systems, computation power and communication mediums, allowing you to focus on the idea rather than the setup.

EnvisEdge allows researchers, developers and data scientists to experiment and test their hypotheses, and produce production-ready code without having direct access to the edge devices. Creating a path for global research and growth in the domains of federated learning and edge computing.

Key features 🌟

Provides a platform for global or remote teams to run and test their systems/models prior to deployment.
Run, train and test FL algorithms and ML models.
Can setup environment of your choice with any arbitrary hardware constraints such as RAM, CPU and more.
Experience Edge on cloud and your devices.

Repo Structure

NimbleEdge/EnvisEdge
CONTRIBUTING.md                         <-- Please go through the contributing guidelines before starting 
README.md                               <-- You are here 
datasets                                <-- Sample datasets
docs                                    <-- Tutorials and walkthroughs 
experiments                             <-- Recommendation models used by our services
fedrec                                  <-- Whole magic takes place here 
      communication_interfaces              <-- Modules for communication interfaces eg. Kafka
      data_models                           <-- All data modules that will be used for communication and thier serializers and  deserializers
      modules                               <-- All the modules related to transformers, embeddings etc.
      multiprocessing                       <-- Modules to run parallel worker jobs
      optimization                          <-- Modules realted to torch optimizers and gradient decesnt etc.
      python_executors                      <-- Contains worker modules eg. trainer and aggregator
      serialization                         <-- serialization interfaces for data models
      user_modules                          <-- Envis modules for wrapping toech modules for users.
      utilities                             <-- Helper modules
fl_strategies                           <-- Federated learning algorithms for our services.
notebooks                               <-- Jupyter Notebook examples
scala-core                              <-- Backbone of EnvisEdge
scripts                                 <-- bash scripts for creating and removing kafka topics.
tests                                   <-- tests

Quickstart

Update the config files of the model (can be found here) you are going to use with logging directory:

log_dir:
  PATH: <path to your logging directory>

Download kafka from Here and start the kafka server using the following commands

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Create kafka topics for the job executor

cd scripts
$ bash add_topics.sh
Enter path to kafka Directory : <Enter the path to the kafka directory>
kafka url: <Enter the URL on which kafka is listening e.g if you are running it on localhost it would be 127.0.0.1>
Creating Topics...

Installation

Install the dependencies using virtual environment

mkdir env
cd env
virtualenv envisedge
source envisedge/bin/activate
pip3 install -r requirements.txt

Download the federated dataset

$ bash download.sh -f
Enter global data path : <Enter the path you want your dataset to be saved>
Enter model : <Enter the config file of the model to update with the dataset path>
Downloading femnist dataset...

Run data preprocessing with preprocess_data . Using this dataset, you will prepare a client_id mapping in the dataset that will be sent to Python workers for training the model.

python preprocess_data.py --config configs/regression.yml

To start the multiprocessing executor run the following command:

$ python executor.py --config configs/regression.yml

To see how traning is done run the following command:

$ python tests/integration_tests/integration_test.py --config configs/regression.yml

Getting started

Tuitorials : to help you get started with understanding and implementing envisedge.

Resources

Start Contributing

Before you begin, please read our CONTRIBUTOR'S GUIDELINES.
Introduce yourself in the #introduction channel on Discord ( Most of the talks and discussions happen here.)
Look for an open issue that interests you such as good first issue, python, scala, documentation and more. Liverage labels feature as shown below.
Star, Fork, and Clone the repo.
Get down to business. Do your work.
Push to your fork.
Send a PullRequest to NimbleEdge/EnvisEdge.

This project follows the all-contributors specification.Contributions of any kind are welcome!!

FAQ and Other Questions

You can know more about Nimbleedge on the about nimbleedge page.If you have question about the code or any general question about the repository or project, please head towards our discord.

License

EnvisEdge has a Apache License 2.0

Envisedge