Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update cfg docs w.r.t. evaluation and fl algos. #301

Merged
merged 2 commits into from
Aug 8, 2022
Merged

update cfg docs w.r.t. evaluation and fl algos. #301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
97ae7aa
update cfg docs w.r.t. evaluation and fl algos.
yxdyc Aug 8, 2022
f67acfb
Merge remote-tracking branch 'upstream/master' into feature/cfg_doc
yxdyc Aug 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions federatedscope/core/configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ The configurations related to the data/dataset are defined in `cfg_data.py`.
| `data.root` | (string) 'data' | <font size=1> The folder where the data file located. `data.root` would be used together with `data.type` to load the dataset. </font> | - |
| `data.type` | (string) 'toy' | <font size=1>Dataset name</font> | CV: 'femnist', 'celeba' ; NLP: 'shakespeare', 'subreddit', 'twitter'; Graph: 'cora', 'citeseer', 'pubmed', 'dblp_conf', 'dblp_org', 'csbm', 'epinions', 'ciao', 'fb15k-237', 'wn18', 'fb15k' , 'MUTAG', 'BZR', 'COX2', 'DHFR', 'PTC_MR', 'AIDS', 'NCI1', 'ENZYMES', 'DD', 'PROTEINS', 'COLLAB', 'IMDB-BINARY', 'IMDB-MULTI', 'REDDIT-BINARY', 'IMDB-BINARY', 'IMDB-MULTI', 'HIV', 'ESOL', 'FREESOLV', 'LIPO', 'PCBA', 'MUV', 'BACE', 'BBBP', 'TOX21', 'TOXCAST', 'SIDER', 'CLINTOX', 'graph_multi_domain_mol', 'graph_multi_domain_small', 'graph_multi_domain_mix', 'graph_multi_domain_biochem'; MF: 'vflmovielens1m', 'vflmovielens10m', 'hflmovielens1m', 'hflmovielens10m', 'vflnetflix', 'hflnetflix'; Tabular: 'toy', 'synthetic'; External dataset: 'DNAME@torchvision', 'DNAME@torchtext', 'DNAME@huggingface_datasets', 'DNAME@openml'. |
| `data.args` | (list) [] | <font size=1>Args for the external dataset</font> | Used for external dataset, eg. `[{'download': False}]` |
| `data.save_data` | (bool) False | <font size=1> Whether to save the generated toy data </font> | - |
| `data.splitter` | (string) '' | <font size=1>Splitter name for standalone dataset</font> | Generic splitter: 'lda'; Graph splitter: 'louvain', 'random', 'rel_type', 'graph_type', 'scaffold', 'scaffold_lda', 'rand_chunk' |
| `data.splitter_args` | (list) [] | <font size=1>Args for splitter.</font> | Used for splitter, eg. `[{'alpha': 0.5}]` |
| `data.transform` | (list) [] | <font size=1>Transform for x of data</font> | Used in `get_item` in torch.dataset, eg. `[['ToTensor'], ['Normalize', {'mean': [0.1307], 'std': [0.3081]}]]` |
Expand All @@ -31,6 +32,7 @@ The configurations related to the data/dataset are defined in `cfg_data.py`.
| `cfg.data.quadratic.min_curv` | (float) 0.02 | <font size=1>Min_curve of synthetic quadratic dataset</font> | - |
| `cfg.data.quadratic.max_curv` | (float) 12.5 | <font size=1>Max_cur of synthetic quadratic dataset</font> | - |


### Model

The configurations related to the model are defined in `cfg_model.py`.
Expand Down Expand Up @@ -64,6 +66,60 @@ The configurations related to the model are defined in `cfg_model.py`.
| `regularizer.type` | (string) ' ' | The type of the regularizer | Chosen from [`proximal_regularizer`] |
| `regularizer.mu` | (float) 0 | The factor that controls the loss of the regularization term | - |


### Federated Algorithms
The configurations related to specific federated algorithms, which are
defined in
`cfg_fl_algo.py`.

#### `fedopt`: for FedOpt algorithm
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `fedopt.use` | (bool) False | <font size=1> Whether to run FL courses with FedOpt algorithm. </font> | If False, all the related configurations (cfg.fedopt.xxx) would not take effect. |
| `fedopt.optimizer.type` | (string) 'SGD' | <font size=1> The type of optimizer used for FedOpt algorithm. </font> | Currently we support all optimizers build in PyTorch (The modules under torch.optim). |
| `fedopt.optimizer.lr` | (float) 0.1 | <font size=1> The learning rate used in for FedOpt optimizer. </font> | - |
#### `fedprox`: for FedProx algorithm
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `fedprox.use` | (bool) False | <font size=1> Whether to run FL courses with FedProx algorithm. </font> | If False, all the related configurations (cfg.fedprox.xxx) would not take effect. |
| `fedprox.mu` | (float) 0.0 | <font size=1> The hyper-parameter $\mu$ used in FedProx algorithm. </font> | - |
#### `personalization`: for personalization algorithms
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `personalization.local_param` | (list of str) [] | <font size=1> The client-distinct local param names, e.g., ['pre', 'bn'] </font> | - |
| `personalization.share_non_trainable_para` | (bool) False | <font size=1> Whether transmit non-trainable parameters between FL participants </font> | - |
| `personalization.local_update_steps` | (int) -1 | <font size=1> The local training steps for personalized models </font> | By default, -1 indicates that the local model steps will be set to be the same as the valid `train.local_update_steps` |
| `personalization.regular_weight` | (float) 0.1 | <font size=1> The regularization factor used for model para regularization methods such as Ditto and pFedMe. </font> | The smaller the regular_weight is, the stronger emphasising on personalized model. |
| `personalization.lr` | (float) 0.0 | <font size=1> The personalized learning rate used in personalized FL algorithms. </font> | The default value 0.0 indicates that the value will be set to be the same as `train.optimizer.lr` in case of users have not specify a valid `personalization.lr` |
| `personalization.K` | (int) 5 | <font size=1> The local approximation steps for pFedMe. </font> | - |
| `personalization.beta` | (float) 5 | <font size=1> The average moving parameter for pFedMe. </font> | - |
#### `fedsageplus`: for fedsageplus algorithm
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `fedsageplus.num_pred` | (int) 5 | <font size=1> Number of nodes generated by the generator </font> | - |
| `fedsageplus.gen_hidden` | (int) 128 | <font size=1> Hidden layer dimension of generator </font> | - |
| `fedsageplus.hide_portion` | (float) 0.5 | <font size=1> Hide graph portion </font> | - |
| `fedsageplus.fedgen_epoch` | (int) 200 | <font size=1> Federated training round for generator </font> | - |
| `fedsageplus.loc_epoch` | (int) 1 | <font size=1> Local pre-train round for generator </font> | - |
| `fedsageplus.a` | (float) 1.0 | <font size=1> Coefficient for criterion number of missing node </font> | - |
| `fedsageplus.b` | (float) 1.0 | <font size=1> Coefficient for criterion feature </font> | - |
| `fedsageplus.c` | (float) 1.0 | <font size=1> Coefficient for criterion classification </font> | - |
#### `gcflplus`: for gcflplus algorithm
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `gcflplus.EPS_1` | (float) 0.05 | <font size=1> Bound for mean_norm </font> | - |
| `gcflplus.EPS_2` | (float) 0.1 | <font size=1> Bound for max_norm </font> | - |
| `gcflplus.seq_length` | (int) 5 | <font size=1> Length of the gradient sequence </font> | - |
| `gcflplus.standardize` | (bool) False | <font size=1> Whether standardized dtw_distances </font> | - |
#### `flitplus`: for flitplus algorithm
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `flitplus.tmpFed` | (float) 0.5 | <font size=1> gamma in focal loss (Eq.4) </font> | - |
| `flitplus.lambdavat` | (float) 0.5 | <font size=1> lambda in phi (Eq.10) </font> | - |
| `flitplus.factor_ema` | (float) 0.8 | <font size=1> beta in omega (Eq.12) </font> | - |
| `flitplus.weightReg` | (float) 1.0 | <font size=1> balance lossLocalLabel and lossLocalVAT </font> | - |


### Federated training
The configurations related to federated training are defined in `cfg_training.py`.
Considering it's infeasible to list all the potential arguments for optimizers and schedulers, we allow the users to add new parameters directly under the corresponding namespace.
Expand Down Expand Up @@ -159,6 +215,68 @@ The configurations related to FL settings are defined in `cfg_fl_setting.py`.
| `vertical.dims` | (list of int) [5,10] | The dimensions of the input features for participants. | - |
| `vertical.key_size` | (int) 3072 | The length (bit) of the public keys. | - |






### Evaluation
The configurations related to monitoring and evaluation, which are
defined in
`cfg_evaluation.py`.

| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `eval.freq` | (int) 1 | <font size=1> The frequency we conduct evaluation. </font> | - |
| `eval.metrics` | (list of str) [] | <font size=1> The names of adopted evaluation metrics. </font> | By default, we calculate the ['loss', 'avg_loss', 'total'], all the supported metric can be find in `core/monitors/metric_calculator.py` |
| `eval.split` | (list of str) ['test', 'val'] | <font size=1> The data splits' names we conduct evaluation. </font> | - |
| `eval.report` | (list of str) ['weighted_avg', 'avg', 'fairness', 'raw'] | <font size=1> The results reported forms to loggers </font> | By default, we report comprehensive results, - `weighted_avg` and `avg` indicate the weighted average and uniform average over all evaluated clients; - `fairness` indicates report fairness-related results such as individual performance and std across all evaluated clients; - `raw` indicates that we save and compress all clients' individual results without summarization, and users can flexibly post-process the saved results further.|
| `eval.best_res_update_round_wise_key` | (str) 'val_loss' | <font size=1> The metric name we used to as the primary key to check the performance improvement at each evaluation round. </font> | - |
| `eval.monitoring` | (list of str) [] | <font size=1> Extended monitoring methods or metric, e.g., 'dissim' for B-local dissimilarity </font> | - |
| `eval.count_flops` | (bool) True | <font size=1> Whether to count the flops during the FL courses. </font> | - |
#### `wandb`: for wandb tracking and visualization
| Name | (Type) Default Value | Description | Note |
|:----:|:-----:|:---------- |:---- |
| `wandb.use` | (bool) False | <font size=1> Whether to use wandb to track and visualize the FL dynamics and results. </font> | If `False`, all the related configurations (`wandb.xxx`) would not take effect. |
| `wandb.name_user` | (str) '' | <font size=1> the user name used in wandb management </font> | - |
| `wandb.name_project` | (str) '' | <font size=1> the project name used in wandb management </font> | - |
| `wandb.online_track` | (bool) True | <font size=1> whether to track the results in an online manner, i.e., log results at every evaluation round </font> | - |
| `wandb.client_train_info` | (bool) True | <font size=1> whether to track the training info of clients </font> | - |



#### Fine tuning
The following configurations are related to the fine tuning.

| Name | (Type) Default Value | Description | Note |
|:--------------------------:|:--------------------:|:------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| `finetune.before_eval` | (bool) False | Indicator of fintune before evaluation | If `True`, the clients will fine tune its model before each evaluation. Note the fine tuning is only conducted before evaluation and won't influence the upload weights in each round. |
| `finetune.local_update_steps` | (int) 1 | The number of local fine tune steps | - |
| `finetune.batch_or_epoch` | (string) `batch` | The type of local fine tuning. | Similar with `train.batch_or_epoch`, `finetune.batch_or_epoch` specifies the unit of `finetune.local_update_steps` |
| `finetune.optimizer` | - | - | You can add new parameters under `finetune.optimizer` according to the type of optimizer. All new parameters will be used as arguments for the chosen optimizer. |
| `finetune.optimizer.type` | (string) 'SGD' | The type of the optimizer used in fine tuning. | Currently we support all optimizers build in PyTorch (The modules under `torch.optim`). |
| `finetune.optimizer.lr` | (float) 0.1 | The learning rate used in local fine tuning | - |
| `finetune.scheduler` | - | - | Similar with `train.scheduler`, you can add new parameters as you need, and all new parameters will be used as arguments for the chosen scheduler. |

#### Grad Clipping
The following configurations are related to the grad clipping.

| Name | (Type) Default Value | Description | Note |
|:--------------------------:|:--------------------:|:------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| `grad.grad_clip` | (float) -1.0 | The threshold used in gradient clipping. | `grad.grad_clip < 0` means we don't clip the gradient. |

#### Early Stop

| Name | (Type) Default Value | Description | Note |
|:----------------------------------------:|:--------------------:|:------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| `early_stop.patience` | (int) 5 | How long to wait after last time the monitored metric improved. | Note that the actual_checking_round = `early_step.patience` * `eval.freq`. To disable the early stop, set the `early_stop.patience` <=0 |
| `early_stop.delta` | (float) 0. | Minimum change in the monitored metric to indicate a improvement. | - |
| `early_stop.improve_indicaator_mode` | (string) 'best' | Early stop when there is no improvement within the last `early_step.patience` rounds, in ['mean', 'best'] | Chosen from 'mean' or 'best' |
| `early_step.the_smaller_the_better` | (bool) True | The optimized direction of the chosen metric | - |




### Asynchronous Training Strategies
The configurations related to applying asynchronous training strategies in FL are defined in `cfg_asyn.py`.

Expand Down
1 change: 1 addition & 0 deletions federatedscope/core/configs/cfg_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ def extend_data_cfg(cfg):

cfg.data.root = 'data'
cfg.data.type = 'toy'
cfg.data.save_data = False # whether to save the generated toy data
cfg.data.args = [] # args for external dataset, eg. [{'download': True}]
cfg.data.splitter = ''
cfg.data.splitter_args = [] # args for splitter, eg. [{'alpha': 0.5}]
Expand Down
1 change: 0 additions & 1 deletion federatedscope/core/configs/cfg_evaluation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ def extend_evaluation_cfg(cfg):
cfg.eval = CN(
new_allowed=True) # allow user to add their settings under `cfg.eval`

cfg.eval.save_data = False
cfg.eval.freq = 1
cfg.eval.metrics = []
cfg.eval.split = ['test', 'val']
Expand Down
12 changes: 12 additions & 0 deletions federatedscope/core/configs/cfg_fl_algo.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,23 +54,35 @@ def extend_fl_algo_cfg(cfg):
# ---------------------------------------------------------------------- #
cfg.fedsageplus = CN()

# Number of nodes generated by the generator
cfg.fedsageplus.num_pred = 5
# Hidden layer dimension of generator
cfg.fedsageplus.gen_hidden = 128
# Hide graph portion
cfg.fedsageplus.hide_portion = 0.5
# Federated training round for generator
cfg.fedsageplus.fedgen_epoch = 200
# Local pre-train round for generator
cfg.fedsageplus.loc_epoch = 1
# Coefficient for criterion number of missing node
cfg.fedsageplus.a = 1.0
# Coefficient for criterion feature
cfg.fedsageplus.b = 1.0
# Coefficient for criterion classification
cfg.fedsageplus.c = 1.0

# ---------------------------------------------------------------------- #
# GCFL+ related options, gfl
# ---------------------------------------------------------------------- #
cfg.gcflplus = CN()

# Bound for mean_norm
cfg.gcflplus.EPS_1 = 0.05
# Bound for max_norm
cfg.gcflplus.EPS_2 = 0.1
# Length of the gradient sequence
cfg.gcflplus.seq_length = 5
# Whether standardized dtw_distances
cfg.gcflplus.standardize = False

# ---------------------------------------------------------------------- #
Expand Down