Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement task_group filtering for FedEval #1226

Merged
merged 1 commit into from
Jan 30, 2025

Conversation

ishaileshpant
Copy link
Collaborator

@ishaileshpant ishaileshpant commented Dec 23, 2024

  • implement a new task_group filtering decorator 'with_selected_task_group' in Assigner class

  • update all the sub-classes that use task_groups to use the decorator

  • update fedeval sample workspace to use default assigner, tasks and aggregator

  • use of federated-evaluation/aggregator.yaml for FedEval specific workspace example to use round_number as 1

  • removed assigner and tasks yaml from defaults/federated-evaluation, superseded by default assigner/tasks

  • added additional checks for assigner sub-classes that might not have task_groups

  • Addressing review comments

  • Updated existing test cases for Assigner sub-classes

  • Remove hard-coded setting in assigner for torch_cnn_mnist ws, refer to default as in other Workspaces

  • Use aggregator supplied --task_group to override the assinger selected_task_group

  • update existing test cases of aggregator cli

  • add test cases for the decorator

  • rebased 25-Jan.1
    Signed-off-by: Shailesh Pant shailesh.pant@intel.com

  • Testcases for task_group_filtering decorator

  • Testing for all existing workspaces

  • we shouldn't be saving any new models if running in evaluation mode.

  • Change all workspaces to use new Assigner [Not needed as defaults ensure all workspace, by default use new assigner]

@ishaileshpant ishaileshpant added this to the 1.8 milestone Dec 23, 2024
@ishaileshpant ishaileshpant force-pushed the poc-mode-assigner branch 19 times, most recently from f1fcdcf to 93248b8 Compare December 24, 2024 11:19
@ishaileshpant ishaileshpant marked this pull request as ready for review January 5, 2025 10:50
@ishaileshpant ishaileshpant changed the title [Please Don't Merge] implement a new ModeBasedAssigner for FedEval Implement a new ModeBasedAssigner for FedEval Jan 5, 2025
@ishaileshpant ishaileshpant force-pushed the poc-mode-assigner branch 4 times, most recently from d63c561 to 2331f75 Compare January 5, 2025 11:30
@ishaileshpant ishaileshpant force-pushed the poc-mode-assigner branch 2 times, most recently from 9aba086 to e8ca274 Compare January 22, 2025 05:51
@ishaileshpant ishaileshpant force-pushed the poc-mode-assigner branch 4 times, most recently from 193ddf7 to d2b46f0 Compare January 22, 2025 09:56
Copy link
Collaborator

@teoparvanov teoparvanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, @ishaileshpant, I think it's looking good at this stage!
I just have a couple of minor remaining comments:

tests/openfl/component/assigner/test_assigner.py Outdated Show resolved Hide resolved
openfl/component/assigner/static_grouped_assigner.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@rahulga1 rahulga1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just modify based on Teo's suggestions.

@ishaileshpant ishaileshpant force-pushed the poc-mode-assigner branch 2 times, most recently from 233feed to db33263 Compare January 25, 2025 05:00
Copy link
Contributor

@psfoley psfoley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, @ishaileshpant. The scenario of multiple user defined task groups needs to be considered before this can be approved. Details are provided in the comments. I can go into more depth on the Assigner design during our standup as well.

openfl-workspace/workspace/plan/defaults/assigner.yaml Outdated Show resolved Hide resolved
openfl/component/assigner/assigner.py Outdated Show resolved Hide resolved
…group' in Assigner class

- update all the sub-classes that use task_groups to use the decorator
- update fedeval sample workspace to use default assigner, tasks and aggregator
- use of federated-evaluation/aggregator.yaml for FedEval specific workspace example to use round_number as 1
- removed assigner and tasks yaml from defaults/federated-evaluation, superseded by default assigner/tasks
- added additional checks for assigner sub-classes that might not have task_groups
- Addressing review comments
- Updated existing test cases for Assigner sub-classes
- Remove hard-coded setting in assigner for torch_cnn_mnist ws, refer to default as in other Workspaces
- Use aggregator supplied --task_group to override the assinger selected_task_group
- update existing test cases of aggregator cli
- add test cases for the decorator
- rebased 25-Jan.1
- implemented the support of multiple task_group without selection
- defaulting of selected_task group 'percentage' to 1.0 post successful filtering
- updated test cases for multiple task group support
Signed-off-by: Shailesh Pant <shailesh.pant@intel.com>
openfl/component/assigner/assigner.py Dismissed Show dismissed Hide dismissed
openfl/component/assigner/assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_random_grouped_assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_random_grouped_assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_random_grouped_assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_random_grouped_assigner.py Dismissed Show dismissed Hide dismissed
tests/openfl/component/assigner/test_random_grouped_assigner.py Dismissed Show dismissed Hide dismissed
@ishaileshpant
Copy link
Collaborator Author

@kta-intel for the last task as discussed in #1258 i've created an issue #1321 and will track that fix via that

@rahulga1 rahulga1 merged commit 0ed1338 into securefederatedai:develop Jan 30, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants