EnvisEdge allows users to simulate an edge computing environment to test their ideas and models before putting them in place on the edge. It takes care of all the complex stuff such as diversity across operating systems, computation power and communication mediums, allowing you to focus on the idea rather than the setup.
EnvisEdge allows researchers, developers and data scientists to experiment and test their hypotheses, and produce production-ready code without having direct access to the edge devices. Creating a path for global research and growth in the domains of federated learning and edge computing.
NimbleEdge/EnvisEdge
CONTRIBUTING.md <-- Please go through the contributing guidelines before starting
README.md <-- You are here
datasets <-- Sample datasets
docs <-- Tutorials and walkthroughs
experiments <-- Recommendation models used by our services
fedrec <-- Whole magic takes place here
communication_interfaces <-- Modules for communication interfaces eg. Kafka
data_models <-- All data modules that will be used for communication and thier serializers and deserializers
modules <-- All the modules related to transformers, embeddings etc.
multiprocessing <-- Modules to run parallel worker jobs
optimization <-- Modules realted to torch optimizers and gradient decesnt etc.
python_executors <-- Contains worker modules eg. trainer and aggregator
serialization <-- serialization interfaces for data models
user_modules <-- Envis modules for wrapping toech modules for users.
utilities <-- Helper modules
fl_strategies <-- Federated learning algorithms for our services.
notebooks <-- Jupyter Notebook examples
scala-core <-- Backbone of EnvisEdge
scripts <-- bash scripts for creating and removing kafka topics.
tests <-- tests
Update the config files of the model (can be found here) you are going to use with logging directory:
log_dir:
PATH: <path to your logging directory>
Download kafka from Here and start the kafka server using the following commands
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
Create kafka topics for the job executor
cd scripts
$ bash add_topics.sh
Enter path to kafka Directory : <Enter the path to the kafka directory>
kafka url: <Enter the URL on which kafka is listening e.g if you are running it on localhost it would be 127.0.0.1>
Creating Topics...
Install the dependencies using virtual environment
mkdir env
cd env
virtualenv envisedge
source envisedge/bin/activate
pip3 install -r requirements.txt
Download the federated dataset
$ bash download.sh -f
Enter global data path : <Enter the path you want your dataset to be saved>
Enter model : <Enter the config file of the model to update with the dataset path>
Downloading femnist dataset...
Run data preprocessing with preprocess_data . Using this dataset, you will prepare a client_id mapping in the dataset that will be sent to Python workers for training the model.
python preprocess_data.py --config configs/regression.yml
To start the multiprocessing executor run the following command:
$ python executor.py --config configs/regression.yml
To see how traning is done run the following command:
$ python tests/integration_tests/integration_test.py --config configs/regression.yml
good first issue
, python
, scala
, documentation
and more. Liverage labels feature as shown below.
This project follows the all-contributors specification.Contributions of any kind are welcome!!
You can know more about Nimbleedge on the about nimbleedge page.If you have question about the code or any general question about the repository or project, please head towards our discord.
EnvisEdge has a Apache License 2.0