Skip to content

Commit

Permalink
Add ray grouped runtime (securefederatedai#910)
Browse files Browse the repository at this point in the history
* Fix flake8 error in local runtime (#764)

* Removes unnecessary dict comprehension

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Removes unnecessary dict comprehension

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update ROADMAP.md (#765)

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update README.md

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update GOVERNANCE.md

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update ROADMAP.md (#785)

Typos

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated integrations to GaNDLF (#781)

* renaming loader and runner

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

* updated plan to pick the new names

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

* new key name

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

* allow the ability to pass a file to `gandlf_config_dict` in addition to fully-fledged parameters

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

* checking this differently

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

* rename variable for clarity

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>

---------

Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update README.md

Removed references to Intel's ownship, given it's now owned by the LF AI and Data.

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Fix Flake8 C419 for Ubuntu CI (#800)

C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Introduced shard descriptor based collaborator private attributes

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Adding batch size for train, and test in config.yaml file
Files modified:
1. config.yaml
2. mnist_shard_descriptor.py
3. Workflow_Interface_101_MNIST.ipynb
4. participants.py

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Introducing multiple config yaml files

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Removing unnecessary config.yaml file.

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added collaborator private atribute delayed initialization
for local_runtime

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Incorporated review comments

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added multi-pricessing ray backend support and, aggregator yaml file

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated multi-processing code

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* RayExecutor class moved from participants.py to localruntime.py

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* RayExecytor moved from interface/pariticipants.py to
runtime/local_runtime.py

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added Aggregator private attribute initialation in runtime

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Removed unnecessary import statements

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Code cleaned up, validated checkpoints manually

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Refactored, and added some new doc string

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolved Flake8 instructions

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Recusrsion removal + Serialization removal integrated

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Incoporated Review Comments

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Removed configuration YAML files, and
added functionality to initialize private attributes by calling a
callback function created by end-user

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Removed commented code

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Implemented new approach, two example files given
1. Workflow_Interface_101_MNIST.py
2. Workflow_Interface_301_MNIST_Watermarking.py

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Internal Review Comments Incorporated

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* No private attributes are required

If private attributes are not provided, by default take an empty dictionary no need to pass a callable function.

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update participants.py

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added a check for GPU Resource Allocation

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Modified error message for resource allocation

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolved bug found during testing phase

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Modifide all the test cases, and following tutorials
1. Privacy Meter
2. FedProx

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added following test cases:
1. Workflow_Interface_101_MNIST.ipynb
2. Workflow_Interface_102_Aggregator_Validation.ipynb
3. Workflow_Interface_301_MNIST_Watermarking.ipynb
4. Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb
5. Workflow_Interface_103_Cyclic_Institutional_Incremental_Learning.ipynb

Signed-off-by: Parth Mandaliya <Parthx.Mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Modified and Added Global_DP tutorials.

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Modified and Added tutorial Workflow-Interface_201_Exclusive_GPUs_with_Ray.ipynb

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Modified documentation for Workflow_Interface_201 tutorial.

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* fixed flake-8 errors

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* reverted import module code

Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolved merge conflicts in local_runtime.py
---------
Fix flake8 error in local runtime (#764)

* Removes unnecessary dict comprehension

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Removes unnecessary dict comprehension

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
---------

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update README.md

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Fix warnings and issues in docs (#825)

* Fix warnings and issues in docs

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>

* fixup! Fix warnings and issues in docs

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>

---------

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Add Logo (#827)

* Add Logo

* Update README.md

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Change OpenFL documentation font to improve accessibility (#809)

This replaces the font of OpenFL documents with Intel One Mono font for low vision developers.

    Known issues:
    1. The text font within the images has not been changed.
    2. Some icons that do not exist in the new font cannot be displayed properly.

    Fixes securefederatedai#799

    Co-authored-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: He, Dan H <dan.h.he@intel.com>
    Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
    Signed-off-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Co-authored-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update unit tests to improve code coverage (#821)

* Update ci config

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add unit test for following files

- openfl/federated/plan/plan.py
- openfl/interface/aggregation_functions/core/adaptive_aggregation.py

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add some test cases for databases module

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Fix bugs for databases module unittest

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Update unit tests for component module

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Restore workflow config and update some comments

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Enable save_ test case.
Add yaml under test dir for unit test usage.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove plan to new dir.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove plan to new dir.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add aggregator start test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 2 aggregator test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 1 aggregator test case.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Format code.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Refactor code.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add collaborator start test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 1 collaborator test case.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Format with flake8

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove TODO comments

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

---------

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>
Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com>
Co-authored-by: Lei5 Chen <lei5.chen@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Add PyTorch linear regression example (#808)

This adds a new tutorial example on distributing a linear regression task over OpenFL cluster.

    The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters.

    Fixes #797

    Co-authored-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: He, Dan H <dan.h.he@intel.com>
    Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
    Signed-off-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* This prints out the hash of the CSR to disk for both the aggregator and (#813)

* This prints out the hash of the CSR to disk for both the aggregator and
collaborator. The user then compares and approves this hash with the
hash printed out of the file to validate the CSR. In addition, a warning
message is pritned if certify is run in silent mode.

Fixes securefederatedai#692

Signed-off-by: Grant Baker <grant.baker@intel.com>

* Refactor read_csr function to use get_csr_hash

Signed-off-by: Grant Baker <grant.baker@intel.com>

* Ask to check hashes before prompt

---------

Signed-off-by: Grant Baker <grant.baker@intel.com>
Co-authored-by: Grant Baker <grant.baker@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Improve workspace requirements import (#810)

Remove the dump_requirement_file operation in export_ method.

            Fixes securefederatedai#767

            Co-authored-by: Li, Qingqing <qingqing.li@intel.com>
            Co-authored-by: Wu, Caili <caili.wu@intel.com>
            Signed-off-by: He, Dan H <dan.h.he@intel.com>
            Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
            Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
            Signed-off-by: Wang, Le <le3.wang@intel.com>
            Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Issue 506 Added Example using FedProx (#818)

* created new ineractive_api dir to hold pytorch fedprox mnist example

corrected files

changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example

renamed FedProx notebook

used mode.parrameters() to get pytorch model weights

got weights using state_dict

changed old wieghts to list (for serialization) and fixed README

input wieghts before zero_grad

[Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api

This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer.

Fixes: #506

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api

This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer.

Fixes: securefederatedai#506

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* Update README.md

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README.

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

---------

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* [Bug: 768] FX CLI: Separate create, cert gen commands (#807)

This change separates existing command
"fx collaborator.py generate-cert-request" command into two
commands.
"fx collaborator create -n {NAME} -d {DATA_PATH: optional}".
"fx collaborator generate-cert-request -n {NAME}".

Fixes #768

Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Add new tutorial example to OpenFL interactive API (#812)

* Add new tutorial example to OpenFL interactive API

This adds a new tutorial example on distributing a linear regression task over OpenFL cluster

The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters.

Fixes #798

Co-authored-by: Beverly Klemme <beverly.klemme@intel.com>
Co-authored-by: Grant Baker <grant.baker@intel.com>

Signed-off-by: Yi CAO <yi2.cao@intel.com>

* reduced requirements.txt in workspace

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

---------

Signed-off-by: Yi CAO <yi2.cao@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Co-authored-by: Yi CAO <yi2.cao@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (#776)

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow (#777)

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Running a federation with GaNDLF Documentation (#794)

* Initial commit of Running the federation with GaNDLF Documentation

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update docs/running_the_federation_with_gandlf.rst

Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update README.md

Removed references to Intel's ownship, given it's now owned by the LF AI and Data.

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix Flake8 C419 for Ubuntu CI (#800)

C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update README.md

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix warnings and issues in docs (#825)

* Fix warnings and issues in docs

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>

* fixup! Fix warnings and issues in docs

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>

---------

Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add Logo (#827)

* Add Logo

* Update README.md

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Change OpenFL documentation font to improve accessibility (#809)

This replaces the font of OpenFL documents with Intel One Mono font for low vision developers.

    Known issues:
    1. The text font within the images has not been changed.
    2. Some icons that do not exist in the new font cannot be displayed properly.

    Fixes securefederatedai#799

    Co-authored-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: He, Dan H <dan.h.he@intel.com>
    Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
    Signed-off-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Co-authored-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update unit tests to improve code coverage (#821)

* Update ci config

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add unit test for following files

- openfl/federated/plan/plan.py
- openfl/interface/aggregation_functions/core/adaptive_aggregation.py

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add some test cases for databases module

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Fix bugs for databases module unittest

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Update unit tests for component module

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Restore workflow config and update some comments

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Enable save_ test case.
Add yaml under test dir for unit test usage.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove plan to new dir.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove plan to new dir.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add aggregator start test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 2 aggregator test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 1 aggregator test case.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Format code.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Refactor code.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add collaborator start test cases.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Add 1 collaborator test case.

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Format with flake8

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

* Remove TODO comments

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>

---------

Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>
Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com>
Co-authored-by: Lei5 Chen <lei5.chen@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add PyTorch linear regression example (#808)

This adds a new tutorial example on distributing a linear regression task over OpenFL cluster.

    The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters.

    Fixes #797

    Co-authored-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: He, Dan H <dan.h.he@intel.com>
    Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
    Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
    Signed-off-by: Wang, Le <le3.wang@intel.com>
    Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* This prints out the hash of the CSR to disk for both the aggregator and (#813)

* This prints out the hash of the CSR to disk for both the aggregator and
collaborator. The user then compares and approves this hash with the
hash printed out of the file to validate the CSR. In addition, a warning
message is pritned if certify is run in silent mode.

Fixes securefederatedai#692

Signed-off-by: Grant Baker <grant.baker@intel.com>

* Refactor read_csr function to use get_csr_hash

Signed-off-by: Grant Baker <grant.baker@intel.com>

* Ask to check hashes before prompt

---------

Signed-off-by: Grant Baker <grant.baker@intel.com>
Co-authored-by: Grant Baker <grant.baker@intel.com>

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Improve workspace requirements import (#810)

Remove the dump_requirement_file operation in export_ method.

            Fixes securefederatedai#767

            Co-authored-by: Li, Qingqing <qingqing.li@intel.com>
            Co-authored-by: Wu, Caili <caili.wu@intel.com>
            Signed-off-by: He, Dan H <dan.h.he@intel.com>
            Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com>
            Signed-off-by: Li, Qingqing <qingqing.li@intel.com>
            Signed-off-by: Wang, Le <le3.wang@intel.com>
            Signed-off-by: Wu, Caili <caili.wu@intel.com>

Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Issue 506 Added Example using FedProx (#818)

* created new ineractive_api dir to hold pytorch fedprox mnist example

corrected files

changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example

renamed FedProx notebook

used mode.parrameters() to get pytorch model weights

got weights using state_dict

changed old wieghts to list (for serialization) and fixed README

input wieghts before zero_grad

[Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api

This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer.

Fixes: #506

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api

This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer.

Fixes: securefederatedai#506

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* Update README.md

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

* addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README.

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

---------

Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* [Bug: 768] FX CLI: Separate create, cert gen commands (#807)

This change separates existing command
"fx collaborator.py generate-cert-request" command into two
commands.
"fx collaborator create -n {NAME} -d {DATA_PATH: optional}".
"fx collaborator generate-cert-request -n {NAME}".

Fixes #768

Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add new tutorial example to OpenFL interactive API (#812)

* Add new tutorial example to OpenFL interactive API

This adds a new tutorial example on distributing a linear regression task over OpenFL cluster

The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters.

Fixes #798

Co-authored-by: Beverly Klemme <beverly.klemme@intel.com>
Co-authored-by: Grant Baker <grant.baker@intel.com>

Signed-off-by: Yi CAO <yi2.cao@intel.com>

* reduced requirements.txt in workspace

Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>

---------

Signed-off-by: Yi CAO <yi2.cao@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Co-authored-by: Yi CAO <yi2.cao@intel.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>

* build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (#776)

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* build(deps): bump tensorflow (#777)

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update GaNDLF repo location and test CI with master branch

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update GaNDLF repo location and test CI with master branch

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update GaNDLF repo location and test CI with master branch

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix documentation links. Change path names and templates for CI

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix paths

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix paths

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix breaking tests

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add compatible onnx version to requirements.txt file

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix wrong csv file name

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix wrong csv file name

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix wrong names in workflow file

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix wrong data path

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix lint in test_gandlf.py

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix lint errors

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>
Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Signed-off-by: Yi CAO <yi2.cao@intel.com>
Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com>
Co-authored-by: akantak <aleksander.kantak@intel.com>
Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com>
Co-authored-by: He, Dan H <dan.h.he@intel.com>
Co-authored-by: xiaoranf <desolatefang@gmail.com>
Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com>
Co-authored-by: Lei5 Chen <lei5.chen@intel.com>
Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com>
Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Co-authored-by: Yi CAO <yi2.cao@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (#841)

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Fix GaNDLF documentation links (#842)

* Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix links in GaNDLF Documentation

* Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix links in GaNDLF Documentation

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Fix incorrectly formatted link in docs (#839)

Signed-off-by: Francis Storr <francis.storr@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolving merge conflicts in local-runtime.py

Integrated aggregator as stateful actor branch, tested.

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>

--------
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump onnx in /openfl-workspace/gandlf_seg_test (#840)

Bumps [onnx](https://github.com/onnx/onnx) from 1.12 to 1.13.0.
- [Release notes](https://github.com/onnx/onnx/releases)
- [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md)
- [Commits](https://github.com/onnx/onnx/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: onnx
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Merged changes of remove-torch-dependency branch

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update setup.py

Upgrading protobuf to 3.20.3 as per tensorboard requirement

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolving merge conflicts

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Accessibility updates (#861)

* Fix incorrectly formatted link in docs

Signed-off-by: Francis Storr <francis.storr@intel.com>

* Font styling, color contrast, other accessibility updates

This update:
1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples.
2. Adds colors (in `colors.css`)
3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future.

Closes #848

Signed-off-by: Francis Storr <francis.storr@intel.com>

---------

Signed-off-by: Francis Storr <francis.storr@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp (#773)

* build(deps): bump tensorflow in /openfl-workspace/keras_nlp

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update RMSProp optimizer import

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* flake8

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_mnist (#771)

* build(deps): bump tensorflow in /openfl-workspace/keras_cnn_mnist

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* revert experimental Adam to legacy (#863)

Signed-off-by: kta-intel <kevin.ta@intel.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: kta-intel <kevin.ta@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_with_compression (#770)

* build(deps): bump tensorflow

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update Adam Optimizer import

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorials/interactive_api/Flax_CNN_CIFAR (#775)

* build(deps): bump tensorflow

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fixed breaking backages

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add quiet flag back to pip install

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp_gramine_ready (#769)

* build(deps): bump tensorflow-cpu

Bumps [tensorflow-cpu](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow-cpu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Use legacy RMSprop optimizer

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Patrick Foley <psfoley@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Accessibility color contrast fixes (#864)

* Fix incorrectly formatted link in docs

Signed-off-by: Francis Storr <francis.storr@intel.com>

* Font styling, color contrast, other accessibility updates

This update:
1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples.
2. Adds colors (in `colors.css`)
3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future.

Closes #848

Signed-off-by: Francis Storr <francis.storr@intel.com>

* Color contrast updates for accessibility

Color contrast updates for accessibility
- update generic `a` element
- update color of links in the toggle-able read-the-docs panel
- update the color of the text in search results
- update the color of notes headers

Signed-off-by: Francis Storr <francis.storr@intel.com>

---------

Signed-off-by: Francis Storr <francis.storr@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Tweak link color so it’s not so aggressive (#865)

Signed-off-by: Francis Storr <francis.storr@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/interactive_api_director/experiments/tensorflow_mnist/envoy (#772)

* build(deps): bump tensorflow

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update sd_requirements.txt

* revert to legacy SGD and install tensorflow==2.11 for workflow

Signed-off-by: kta-intel <kevin.ta@intel.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: kta-intel <kevin.ta@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Foley <patrick.foley@intel.com>
Co-authored-by: kta-intel <kevin.ta@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/tf_2dunet (#774)

* build(deps): bump tensorflow in /openfl-workspace/tf_2dunet

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update requirements.txt

to retrigger CI

* Update requirements.txt

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update Tensorflow, gRPC, Protobuf dependencies (#868)

* Update Tensorflow to latest, finally update grpcio/protobuf

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Lint issue fix and missing tf reference

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* pyzmq version fixed

* fix taskrunner tests for windows

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* fix taskrunner test syntax for windows

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* adding user option to workspace pip install requirements for windows

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* fix windows CI test

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* testing virtual env for windows github actions

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* testing virtual env for windows github actions

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* testing virtual env for windows github actions

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* testing venv for windows

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* test venv for windows

* test venv for windows

* Added new KerasSerializer. Fixed other Interactive API experiments

* Update taskrunner.yml

* Update taskrunner.yml

* Update workspace.py

* Update workspace.py

* Update taskrunner.yml

* Remove get_model import from global namespace so dependencies are not loaded into memory unnecessarily (breaking windows build)

* Refactoring and cleaning up imports to support Windows install

* Fixed logger import paths

* Fix missing imports

* Fix native import

* Fix lint errors

* Fix keras optimizer patch. Remove irrelevant unit test

* Format logs in UTF-8 for windows

* Update interactive-kvasir.yml

* Consolidate github actions python versions to single file

* Update python versions

* Update python versions

* Update python versions

* Reduce # of DataLoader workers for Pytorch Kvasir CI test

* Fix Windows encoding

* Fix Windows encoding and limit rounds so Github Actions CI doesn't run out of memory

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix windows encoding

* Fix Windows encoding

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>
Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com>
Co-authored-by: Mansi Sharma <mansi.sharma@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Add FL plan description to documentation (#872)

* Add plan description to documentation

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* fix indentation

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>

* Apply suggestions from code review

Co-authored-by: Patrick Foley <patrick.foley@intel.com>

---------

Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>
Co-authored-by: Patrick Foley <patrick.foley@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolved flake8 issues

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* GPU Added for aggregator
Fixed issue in 103 Cyclic Institutional Incremental Learning tutorial

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolve Coverity Issues (#874)

* Fix coverity issues

* Resolve remaining coverity issues

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Migrate to Ubuntu 22.04 LTS release (supported through 2027) (#875)

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated documentation:
docs/workflow_interface.rst

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Updated documentation

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update workflow_interface.rst

Fixing typo

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added best model and last model extraction technique in docs/workflow_interface.rst

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added GPU for aggregator

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolving merge conflicts in 103 cyclic tutorial notebook

fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833)

* fixing FedAvg averaging in order to be compatible with numpy v1.24+

Signed-off-by: kta-intel <kevin.ta@intel.com>

* uncommenting installations for consistency with other tutorials

Signed-off-by: kta-intel <kevin.ta@intel.com>

* fixing 301_MNIST_Watermarking tutorial FedAvg

Signed-off-by: kta-intel <kevin.ta@intel.com>

* fixing 301_MNIST_Watermarki
ng tutorial FedAvg

Signed-off-by: kta-intel <kevin.ta@intel.com>

* Switching to py38 kernel and clearing cell outputs

Signed-off-by: kta-intel <kevin.ta@intel.com>

---------

Signed-off-by: kta-intel <kevin.ta@intel.com>

---------

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Resolved merge conflicts in tests/github/experimental/testflow_datastore_cli.py

Testflow for verifying stdout redirection to Metaflow datastore (#758)

* implemented ray.wait

* reverted changes back after testing

* adding datastore cli test case

* removed unused variables

* removed stderr validation

* fixed lint suggestions

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Added weighted_average aggregation function under openfl.experimental.interface.{keras,torch}.aggregation_funtions

Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update EdenPipeline in the documentation (#877)

Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* WIP: CI Scans (#873)

* Initial scans commit for bandit, hadolint, trivy

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Address bandit scan results

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix Trivy action

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Fix linting

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Add Coverity Badge

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update Hadolint threshold to flag errors only

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* Update Hadolint threshold to flag errors only

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* Update ROADMAP.md (#878)

Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>

* initial commit

* add docstrings

* change importlib to import

* remove unnecesary files, replace ray with ray_grouped

* remove max concurency, add number of actors

* Trigger CI

* run tests

* lint changes

* flake

* changed number of actors to num_actors, added docs

* Fixed workflow API tests

Signed-off-by: Patrick Foley <psfoley@gmail.com>

* lint fixes

Signed-off-by: Patrick Foley <psfoley@gmail.com>

---------

Signed-off-by: Patrick Foley <psfoley@gmail.com>
Signed-off-by: ParthM-GitHub <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>
Signed-off-by: sarthakpati <sarthak.pati@hotmail.com>
Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com>
Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Signed-off-by: Parth Mandaliya <Parthx.Mandaliya@intel.com>
Signed-off-by: He, Dan H <dan.h.he@intel.com>
Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com>
Signed-off-by: Grant Baker <grant.baker@intel.com>
Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com>
Signed-off-by: Baker, Grant <grant.baker@intel.com>
Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com>
Signed-off-by: Beverly Klemme <beverly.klemme@intel.com>
Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com>
Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Signed-off-by: Yi CAO <yi2.cao@intel.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Francis Storr <francis.storr@intel.com>
Signed-off-by: kta-intel <kevin.ta@intel.com>
Signed-off-by: Mansi Sharma <mansi.sharma@intel.com>
Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com>
Co-authored-by: Patrick Foley <patrick.foley@intel.com>
Co-authored-by: Olga Perepelkina <olga.perepelkina@intel.com>
Co-authored-by: Joe Devon <138038+joedevon@users.noreply.github.com>
Co-authored-by: Sarthak Pati <sarthak.pati@hotmail.com>
Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com>
Co-authored-by: akantak <aleksander.kantak@intel.com>
Co-authored-by: Parth Mandaliya <parthx.mandaliya@intel.com>
Co-authored-by: Parth Mandaliya <Parthx@Mandaliya.intel.com>
Co-authored-by: Parth Mandaliya <parth.mandaliya.007@gmail.com>
Co-authored-by: Keerti Talwar <keertix.talwar@intel.com>
Co-authored-by: KeertiX <keertix.talwarx@intel.com>
Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com>
Co-authored-by: He, Dan H <dan.h.he@intel.com>
Co-authored-by: xiaoranf <desolatefang@gmail.com>
Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com>
Co-authored-by: Lei5 Chen <lei5.chen@intel.com>
Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com>
Co-authored-by: Grant Baker <grant.baker@intel.com>
Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com>
Co-authored-by: Yi CAO <yi2.cao@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu>
Co-authored-by: Francis Storr <francis.storr@intel.com>
Co-authored-by: Patrick Foley <psfoley@gmail.com>
Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com>
Co-authored-by: kta-intel <kevin.ta@intel.com>
Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com>
Co-authored-by: Mansi Sharma <mansi.sharma@intel.com>
Co-authored-by: Sachin Gupta <sachin.gupta.dsp@gmail.com>
Co-authored-by: Keerti Prakash Talwar <115972088+KeertiX@users.noreply.github.com>
Co-authored-by: Amit Portnoy <1131991+amitport@users.noreply.github.com>
Signed-off-by: nammbash <niroopshankar.ammbashankar@intel.com>
  • Loading branch information
1 parent c766a58 commit 9936202
Show file tree
Hide file tree
Showing 50 changed files with 3,406 additions and 1,282 deletions.
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ venv/*
.idea
*_pb2.py
*_pb2_grpc.py

*.jpg
*.crt
*.key
Expand Down
189 changes: 144 additions & 45 deletions docs/workflow_interface.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,30 +146,60 @@ A :code:`Runtime` defines where the flow will be executed, who the participants

.. code-block:: python
# Setup participants
aggregator = Aggregator()
aggregator.private_attributes = {}
# Setup collaborators with private attributes
collaborator_names = ['Portland', 'Seattle', 'Chandler','Bangalore']
collaborators = [Collaborator(name=name) for name in collaborator_names]
for idx, collaborator in enumerate(collaborators):
local_train = deepcopy(mnist_train)
local_test = deepcopy(mnist_test)
local_train.data = mnist_train.data[idx::len(collaborators)]
local_train.targets = mnist_train.targets[idx::len(collaborators)]
local_test.data = mnist_test.data[idx::len(collaborators)]
local_test.targets = mnist_test.targets[idx::len(collaborators)]
collaborator.private_attributes = {
'train_loader': torch.utils.data.DataLoader(local_train,batch_size=batch_size_train, shuffle=True),
'test_loader': torch.utils.data.DataLoader(local_test,batch_size=batch_size_train, shuffle=True)
# Aggregator
aggregator_ = Aggregator()
collaborator_names = ["Portland", "Seattle", "Chandler", "Bangalore"]
def callable_to_initialize_collaborator_private_attributes(index, n_collaborators, batch_size, train_dataset, test_dataset):
train = deepcopy(train_dataset)
test = deepcopy(test_dataset)
train.data = train_dataset.data[index::n_collaborators]
train.targets = train_dataset.targets[index::n_collaborators]
test.data = test_dataset.data[index::n_collaborators]
test.targets = test_dataset.targets[index::n_collaborators]
return {
"train_loader": torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True),
"test_loader": torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=True),
}
# This is equivalent to:
# local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, backend='single_process')
local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators)
Let's break this down, starting with the :code:`Aggregator` and :code:`Collaborator` placeholders. These placeholders represent the nodes where tasks will be executed. Each participant placeholder has its own set of :code:`private_attributes`; a dictionary where the key is the name of the attribute, and the value is the object. In the above example, each of the four collaborators ('Portland', 'Seattle', 'Chandler', and 'Bangalore'), have a :code:`train_loader` and `test_loader` that they can access. These private attributes can be named anything, and do not necessarily need to be the same across each participant.
# Setup collaborators private attributes via callable function
collaborators = []
for idx, collaborator_name in enumerate(collaborator_names):
collaborators.append(
Collaborator(
name=collaborator_name,
private_attributes_callable=callable_to_initialize_collaborator_private_attributes,
index=idx,
n_collaborators=len(collaborator_names),
train_dataset=mnist_train,
test_dataset=mnist_test,
batch_size=64
)
)
local_runtime = LocalRuntime(aggregator=aggregator_, collaborators=collaborators)
Let's break this down, starting with the :code:`Aggregator` and :code:`Collaborator` components. These components represent the *Participants* in a Federated Learning experiment. Each participant has its own set of *private attributes* that represent the information / data specific to its role or requirements. As the name suggests these *private attributes* are accessible only to the particular participant, and are appropriately inserted into or filtered out of current Flow state when transferring from between Participants. For e.g. Collaborator private attributes are inserted into :code:`flow` when transitioning from Aggregator to Collaborator and are filtered out when transitioning from Collaborator to Aggregator.

In the above :code:`FederatedFlow`, each collaborator accesses train and test datasets via *private attributes* :code:`train_loader` and :code:`test_loader`. These *private attributes* need to be set using a (user defined) callback function while instantiating the participant. Participant *private attributes* are returned by the callback function in form of a dictionary, where the key is the name of the attribute and the value is the object.

In this example callback function :code:`callable_to_initialize_collaborator_private_attributes()` returns the collaborator private attributes :code:`train_loader` and :code:`test_loader` that are accessed by collaborator steps (:code:`aggregated_model_validation`, :code:`train` and :code:`local_model_validation`). Some important points to remember while creating callback function and private attributes are:

- Callback Function needs to be defined by the user and should return the *private attributes* required by the participant in form of a key/value pair
- In above example multiple collaborators have the same callback function. Depending on the Federated Learning requirements, user can specify unique callback functions for each Participant
- If no Callback Function is specified then the Participant shall not have any *private attributes*
- Callback function can be provided with any parameters required as arguments. In this example, parameters essential for the callback function are supplied with corresponding values bearing *same names* during the instantiation of the Collaborator

* :code:`index`: Index of the particular collaborator needed to shard the dataset
* :code:`n_collaborators`: Total number of collaborators in which the dataset is sharded
* :code:`batch_size`: For the train and test loaders
* :code:`train_dataset`: Train Dataset to be sharded between n_collaborators
* :code:`test_dataset`: Test Dataset to be sharded between n_collaborators

- Callback function needs to be specified by user while instantiating the participant. Callback function is invoked by the OpenFL runtime at the time participant is created and once created these attributes cannot be modified
- Private attributes are accessible only in the Participant steps

Now let's see how the runtime for a flow is assigned, and the flow gets run:

Expand All @@ -184,23 +214,43 @@ And that's it! This will run an instance of the :code:`FederatedFlow` on a singl
Runtime Backends
================

The Runtime defines where code will run, but the Runtime has a :code:`Backend` - which defines the underlying implementation of *how* the flow will be executed. :code:`'single_process'` is the default in the :code:`LocalRuntime`: it executes all code sequentially within a single python process, and is well suited to run both on high spec and low spec hardware. For users with large servers or multiple GPUs they wish to take advantage of, we also provide a `Ray <https://github.com/ray-project/ray>` backend. The Ray backend enables parallel task execution for collaborators, and optionally allows users to request dedicated GPUs for collaborator tasks in the placement decorator, as follows:
The Runtime defines where code will run, but the Runtime has a :code:`Backend` - which defines the underlying implementation of *how* the flow will be executed. :code:`single_process` is the default in the :code:`LocalRuntime`: it executes all code sequentially within a single python process, and is well suited to run both on high spec and low spec hardware

For users with large servers or multiple GPUs they wish to take advantage of, we also provide a :code:`ray` `<https://github.com/ray-project/ray>` backend. The Ray backend enables parallel task execution for collaborators, and optionally allows users to request dedicated CPU / GPUs for Participants by using the :code:`num_cpus` and :code:`num_gpus` arguments while instantiating the Participant in following manner:

.. code-block:: python
ExampleDedicatedGPUFlow(FLSpec):
...
# We request one dedicated GPU for this task
@collaborator(num_gpus=1)
def training(self):
print(f'CUDA_VISIBLE_DEVICES: {os.environ["CUDA_VISIBLE_DEVICES"]}'))
self.loss = train_func(self.model, self.train_loader)
self.next(self.validation)
...
# Aggregator
aggregator_ = Aggregator(num_gpus=0.2)
collaborator_names = ["Portland", "Seattle", "Chandler", "Bangalore"]
def callable_to_initialize_collaborator_private_attributes(index, n_collaborators, batch_size, train_dataset, test_dataset):
...
# Setup collaborators private attributes via callable function
collaborators = []
for idx, collaborator_name in enumerate(collaborator_names):
collaborators.append(
Collaborator(
name=collaborator_name,
num_gpus=0.2, # Number of the GPU allocated to Participant
private_attributes_callable=callable_to_initialize_collaborator_private_attributes,
index=idx,
n_collaborators=len(collaborator_names),
train_dataset=mnist_train,
test_dataset=mnist_test,
batch_size=64
)
)
# The Ray Backend will now be used for local execution
local_runtime = LocalRuntime(aggregator=aggregator, collaborators=collaborators, backend='ray')
In the above example, we have used :code:`num_gpus=0.2` while instantiating Aggregator and Collaborator to specify that each participant shall use 1/5th of GPU - this results in one GPU being dedicated for a total of 4 collaborators and 1 Aggregator. Users can tune these arguments based on their Federated Learning requirements and available hardware resources. Configurations where one Participant is shared across GPUs is not supported. For e.g. trying to run 5 participants on 2 GPU hardware with :code:`num_gpus=0.4` will not work since 80% of each GPU is allocated to 4 participants and 5th participant does not have any available GPU remaining for use.

**Note:** It is not necessary to have ALL the participants use GPUs. For e.g. only the Collaborator are allocated to GPUs. In this scenario user should ensure that the artifacts returned by Collaborators to Aggregator (e.g. locally trained model object) should be loaded back to CPU before exiting the collaborator step (i.e. before the join step). As Tensorflow manages the object allocation by default therefore this step is needed only for Pytorch.

Debugging with the Metaflow Client
==================================

Expand All @@ -218,19 +268,38 @@ After the flow has started running, you can use the Metaflow Client to get inter

.. code-block:: python
from metaflow import Flow, Run, Task, Step
from metaflow import Metaflow, Flow, Step, Task
# Initialize Metaflow object and obtain list of executed flows:
m = Metaflow()
list(m)
> [Flow('FederatedFlow'), Flow('AggregatorValidationFlow'), Flow('FederatedFlow_MNIST_Watermarking')]
# The name of the flow is the name of the class
flow = Flow('FederatedFlow')
run = flow.latest_run
# Identify the Flow name
flow_name = 'FederatedFlow'
# List all instances of Federatedflow executed under distinct run IDs
flow = Flow(flow_name)
list(flow)
> [Run('FederatedFlow/1692946840822001'),
Run('FederatedFlow/1692946796234386'),
Run('FederatedFlow/1692902602941163'),
Run('FederatedFlow/1692902559123920'),]
# To Retrieve the latest run of the Federatedflow
run = Flow(flow_name).latest_run
print(run)
> Run('FederatedFlow/1692946840822001')
list(run)
> [Step('FederatedFlow/1671152854447797/end'),
Step('FederatedFlow/1671152854447797/join'),
Step('FederatedFlow/1671152854447797/local_model_validation'),
Step('FederatedFlow/1671152854447797/train'),
Step('FederatedFlow/1671152854447797/aggregated_model_validation'),
Step('FederatedFlow/1671152854447797/start')]
step = Step('FederatedFlow/1671152854447797/aggregated_model_validation')
> [Step('FederatedFlow/1692946840822001/end'),
Step('FederatedFlow/1692946840822001/join'),
Step('FederatedFlow/1692946840822001/local_model_validation'),
Step('FederatedFlow/1692946840822001/train'),
Step('FederatedFlow/1692946840822001/aggregated_model_validation'),
Step('FederatedFlow/1692946840822001/start')]
step = Step('FederatedFlow/1692946840822001/aggregated_model_validation')
for task in step:
if task.data.input == 'Portland':
print(task.data)
Expand Down Expand Up @@ -260,6 +329,37 @@ And if we wanted to get log or error message for that task, you can just run:
print(portland_task.stderr)
> [No output]
Also, If we wanted to get the best model and the last model, you can just run:

.. code-block:: python
# Choose the specific step containing the desired models (e.g., 'join' step):
step = Step('FederatedFlow/1692946840822001/join')
list(step)
> [Task('FederatedFlow/1692946840822001/join/12'),--> Round 3
Task('FederatedFlow/1692946840822001/join/9'), --> Round 2
Task('FederatedFlow/1692946840822001/join/6'), --> Round 1
Task('FederatedFlow/1692946840822001/join/3')] --> Round 0
"""The sequence of tasks represents each round, with the most recent task corresponding to the final round and the preceding tasks indicating the previous rounds
in chronological order.
To determine the best model, analyze the command line logs and model accuracy for each round. Then, provide the corresponding task ID associated with that Task"""
task = Task('FederatedFlow/1692946840822001/join/9')
# Access the best model and its associated data
best_model = task.data.model
best_local_model_accuracy = task.data.local_model_accuracy
best_aggregated_model_accuracy = t.data.aggregated_model_accuracy
# To retrieve the last model, select the most recent Task i.e last round.
task = Task('FederatedFlow/1692946840822001/join/12')
last_model = task.data.model
# Save the chosen models using a suitable framework (e.g., PyTorch in this example):
import torch
torch.save(last_model.state_dict(), PATH)
torch.save(best_model.state_dict(), PATH)
While this information is useful for debugging, depending on your workflow it may require significant disk space. For this reason, `checkpoint` is disabled by default.

Runtimes: Future Plans
Expand All @@ -279,4 +379,3 @@ Our goal is to make it a one line change to configure where and how a flow is ex
federated_runtime = FederatedRuntime(...)
flow.runtime = federated_runtime
flow.run()
Loading

0 comments on commit 9936202

Please sign in to comment.