Homomorphic Encryption Model Layer Selection

[wzjin2017]

Hi,

Thanks for the amazing work! I have been using your framework, specifically on the HE functionality. As you mentioned in the doc, to further reduce impacts from costly HE operations, it is a good idea to only do HE for certain model layers. I was wondering what is the best way to select layers regarding privacy guarantee (since we are not working on the fully encrypted data, still some layers of local models are exposed to the server) and performance (both overhead and model accuracy etc)? Thanks!

[YuanTingHsieh]

@wzjin2017 thanks for the interesting and discussion.
@holgerroth could you share some insight on HE?

[holgerroth]

Hi @wzjin2017, interesting question. To address this question, it would be great to do a study to quantify the data leakage with respect to different layers using our new filters to quantify data leakage based on gradient inversion https://github.com/NVIDIA/NVFlare/tree/dev/research/quantifying-data-leakage.

There is also some work we did on gradient inversion with vision transformers. In that paper, there's a figure that includes an analysis with respect to the layers of the network. https://openaccess.thecvf.com/content/CVPR2022/html/Hatamizadeh_GradViT_Gradient_Inversion_of_Vision_Transformers_CVPR_2022_paper.html

I think the behavior of CNNs will be similar but would be interesting to confirm.

[wzjin2017]

@holgerroth Thanks for the quick reply! I am really interested in the results of the CVPR paper on the layer importance of ViT. However, the link to the code in that paper is invalid. Could you direct me to the right repo so I can reproduce the results?

[holgerroth]

I think the code will be released here in the near future. Please keep checking that site. In the meantime, you can achieve the same using the provided filter in our example for CNNs: https://github.com/NVIDIA/NVFlare/tree/dev/research/quantifying-data-leakage