EnvisEdge allows users to simulate an edge computing environment to test their ideas and models before putting them in place on the edge. It takes care of all the complex stuff such as diversity across operating systems, computation power and communication mediums, allowing you to focus on the idea rather than the setup.
EnvisEdge allows researchers, developers and data scientists to experiment and test their hypotheses, and produce production-ready code without having direct access to the edge devices. Creating a path for global research and growth in the domains of federated learning and edge computing.
NimbleEdge/EnvisEdge CONTRIBUTING.md <-- Please go through the contributing guidelines before starting README.md <-- You are here datasets <-- Sample datasets docs <-- Tutorials and walkthroughs experiments <-- Recommendation models used by our services fedrec <-- Whole magic takes place here communication_interfaces <-- Modules for communication interfaces eg. Kafka data_models <-- All data modules that will be used for communication and thier serializers and deserializers modules <-- All the modules related to transformers, embeddings etc. multiprocessing <-- Modules to run parallel worker jobs optimization <-- Modules realted to torch optimizers and gradient decesnt etc. python_executors <-- Contains worker modules eg. trainer and aggregator serialization <-- serialization interfaces for data models user_modules <-- Envis modules for wrapping toech modules for users. utilities <-- Helper modules fl_strategies <-- Federated learning algorithms for our services. notebooks <-- Jupyter Notebook examples scala-core <-- Backbone of EnvisEdge scripts <-- bash scripts for creating and removing kafka topics. tests <-- tests
Update the config files of the model (can be found here) you are going to use with logging directory:
log_dir: PATH: <path to your logging directory>
Download kafka from Here and start the kafka server using the following commands
bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties
Create kafka topics for the job executor
cd scripts $ bash add_topics.sh Enter path to kafka Directory : <Enter the path to the kafka directory> kafka url: <Enter the URL on which kafka is listening e.g if you are running it on localhost it would be 127.0.0.1> Creating Topics...
Install the dependencies using virtual environment
mkdir env cd env virtualenv envisedge source envisedge/bin/activate pip3 install -r requirements.txt
Download the federated dataset
$ bash download.sh -f Enter global data path : <Enter the path you want your dataset to be saved> Enter model : <Enter the config file of the model to update with the dataset path> Downloading femnist dataset...
Run data preprocessing with preprocess_data . Using this dataset, you will prepare a client_id mapping in the dataset that will be sent to Python workers for training the model.
python preprocess_data.py --config configs/regression.yml
To start the multiprocessing executor run the following command:
$ python executor.py --config configs/regression.yml
To see how traning is done run the following command:
$ python tests/integration_tests/integration_test.py --config configs/regression.yml
good first issue,
documentationand more. Liverage labels feature as shown below.
This project follows the all-contributors specification.Contributions of any kind are welcome!!
You can know more about Nimbleedge on the about nimbleedge page.If you have question about the code or any general question about the repository or project, please head towards our discord.
EnvisEdge has a Apache License 2.0