Privatar is the first that leverages both local and untrusted cloud to concurrently achieve privacy-preserving multi-user avatar reconstruction. The entire post-split flow is illustrated as the figure below.
- multiface_partition: Adopt BDCT filter with block=8, which gives 64 different frequency blocks, this does not have having input upsampling and remove three convolution layers.
- multiface_partition_bdct4x4: Decompose unwrapped texture into multiple frequency components -- Adopt BDCT filter with block=4, which gives 16 different frequency blocks, this remove 2 convolution layers, no outsourcing.
- multiface_partition_bdct4x4_hp: Decompose unwrapped texture into multiple frequency components AND horizontally partition all components into local and cloud. Adopt BDCT filter with block=4, which gives 16 different frequency blocks, this remove 2 convolution layers. The number of outsourced frequency components is controlled via
num_freq_comp_outsourced. - multiface_sparse: add sparsity to only decoder of the original VAE model.
- multiface_quantization: change the bitprecision of data into 8-/16-/32-bit integer for the decoder only.
We list two flows to install, run and test Privatar because different environments have different package in support. E.g. RTX 3090 supports RasterizeGLContext but GH200 only supports RasterizeCudaContext. To reduce the effort for setting up configurations for different platforms, we directly prepare two sets of scripts, separately.
- For typical desktop-class GPU such as RTX 3090, we recommend using the conda virtual environment.
conda create -n <your_favoriate_name> python=3.7
conda activate <your_favoriate_name>- For cloud-class GPU such as GH200 We recommand using NVIDIA built-in docker.
Command 1: Download the docker. If you have Docker 19.03 or later, a typical command to launch the container is:
docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:xx.xx-py3Note: our experiments base on nvcr.io/nvidia/pytorch:24.01-py3
Command 2: Launch the docker.
docker run --gpus all -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --memory 51200m --rm <docker_name>where <docker_name> refers to the name of your downloaded docker "nvcr.io/nvidia/pytorch:xx.xx-py3"
- Install OS-level dependencies
$ sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
$ sudo apt-get install mesa-utils
$ glxinfo | grep -i opengl- Install python packages
pip3 install torch==1.13.0 -f https://download.pytorch.org/whl/cu121/torch_stable.html
pip3 install Pillow ninja imageio imageio_ffmpeg six tensorboard opencv-python==4.8.0.74 wandb torchjpeg
Note: we use "wandb" to track the training, testing progress and record the final results.
By default, wandb is turned off, u could change wandb_enable from each training/testing script to enable wandb.
- Download and install NVDiffrast
cd <path_to_privatar>
git clone https://github.com/NVlabs/nvdiffrast
cd nvdiffrast
python3 setup.py install- Download and install torchjpeg
git clone https://github.com/Queuecumber/torchjpeg
cd torchjpeg
pip3 setup.py installWe don't recommend installing torchjpeg using pip3 install because it might break other dependencies.
cd <path_to_privatar>
python3 ./multiface/download_dataset.py --dest "<path_to_Privatar>/dataset" --download_config "./mini_download_config.json"The pretrained weights for different users in the provided datasets are collected at this facial_pretrained_datasets. We use 6795937 base model as the evaluation target. Other structure would work as well. The link for the pretrained model weights of 6795937 is 6795937_base
cd <path_to_privatar>
mkdir pretrain_model
wget https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/PretrainedModel/6795937--GHS-base_nosl/best_model.pth -O 6795937_best_model.pth- Original: multiface baseline using the pretrained model weights
cd <path_to_privatar>/multiface
python3 launch_train_job_serial_GH200.py- Design Choice 1: directly split mesh and unwrapped texture into two separate path (outsource entire unwrapped texture)
cd <path_to_privatar>/multiface_direct_split
python3 launch_train_job_serial_GH200.py- Design Choice 2: quantize model to be low precision
cd <path_to_privatar>/multiface_quantization
python3 launch_train_job_serial_GH200.py- Design Choice 3: prune channels from the decoder to reduce the local computation
cd <path_to_privatar>/multiface_sparse
python3 launch_train_job_serial_GH200.py- Design Choice 4: decompose "unwrapped texture" into 16 frequency components, but keep all of them run local. This requires training to be completed.
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct
python3 launch_train_job_serial_GH200.py- Design Choice 5: decompose "unwrapped texture" into 16 frequency components, configurable components outsourcing. The number of frequency components are controlled by
num_freq_comp_outsourced. This requires training to be completed.
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct_hp
python3 launch_train_job_serial_GH200.pyNote: all training results locate at the <path_to_privatar>/training_results folder.
- Original: multiface baseline using the pretrained model weights
cd <path_to_privatar>/multiface
python3 launch_test_job_serial_GH200.py- Design Choice 1: directly split mesh and unwrapped texture into two separate path (outsource entire unwrapped texture)
cd <path_to_privatar>/multiface_direct_split
python3 launch_test_job_serial_GH200.py- Design Choice 2: quantize model to be low precision
cd <path_to_privatar>/multiface_quantization
python3 launch_test_job_serial_GH200.py- Design Choice 3: prune channels from the decoder to reduce the local computation
cd <path_to_privatar>/multiface_sparse
python3 launch_test_job_serial_GH200.py- Design Choice 4: decompose "unwrapped texture" into 16 frequency components, but keep all of them run local. This requires training to be completed.
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct
python3 launch_test_job_serial_GH200.py- Design Choice 5: decompose "unwrapped texture" into 16 frequency components, configurable components outsourcing. The number of frequency components are controlled by
num_freq_comp_outsourced. This requires training to be completed.
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct_hp
python3 launch_test_job_serial_GH200.pyNote: all testing results locate at the <path_to_privatar>/testing_results folder.
Before calculating noise, two following steps need to be completed.
-
Complete Training: Train the model to generate the final weights required for testing.
-
Perform Testing: Use the trained model weights to evaluate. During testing, the latent codes are stored in the following directory:
./testing_results/<project_name>/latent_code/z_<id>.pth: Latent code for the local path.z_outsource_<id>.pth: Latent code for the outsourced path (runs on the cloud). Note: thesave_latent_codecommand in the scriptlaunch_test_job_serial_GH200.pycontrols whether to explicitlys store latent code to the external drive.
Differential Privacy (DP) noise calculation follows paper. DP requires the prior knowledge of the L2 norm of all outsourced latent codes. Detailed procedures are written as the comments in the code.
python3 <path_to_privatar>/Privatar/experiment_scripts/dp_analysis/dp_noise_generation.pyDetailed procedures in calculating DP noise are detailed in the function dp_noise_calculation.
After this script, the generated noise will show up in the path
<path_to_privatar>/experiment_scripts/dp_analysis/generated_dp_noise/dp_noise_ibdct_decoder_<num_outsourced_freq_components>_<mutual_information_bound>.npy
Noise calculation following PAC privacy requires the prior knowledge of the L2 norm of the covariance of all outsourced latent codes. So that PAC privacy could leverage the dimensional differences to generate non-uniform noise for minimizing the overall noise intensity.
python3 <path_to_privatar>/Privatar/experiment_scripts/pac_analysis/pac_noise_covariance_calculation_GH200.pyAfter this script, the generated noise will show up in the path
<path_to_privatar>/experiment_scripts/pac_analysis/noise_covariance/pac_noise_outsourced_ibdct_decoder_<num_outsourced_freq_components>_<mutual_information_bound>.npy
After noise calculation, now we could start testing the actual accuracy and latency of noisy inference! Specifically, in horizontally partitioned avatar reconstruction flow, generated noise is only injected to the outsourced latent codes.
Noisy inference
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct_hp
python3 launch_noisy_test_job_serial_GH200.pyThis generates detailed accuracy of noisy horizontal partitioned avatar reconstruction under various partition configurations.
The attacker guesses the expression with the minimal difference through comparing the estimated frequency component to the reference expression, as shown by the Fig. 8 of the paper. Therefore, we need to first define the frequency components of reference expressions.
We provide three different ways of generating reference components, under all of which the attacker shows similar empirical successful rate.
- Method 1:
accumulate_channelmode, where only high frequency components are used as reference. - Method 2:
attack_from_high_frequency_channelmode, where only high frequency components are used as reference. - Method 3: when both modes are set as False, all 16 frequency components are sorted based on the amibuigity, and the frequency components coming with similar amount of ambuiguity will be merged as one
if model.normalize_list[freq_pair[0]] + model.normalize_list[freq_pair[1]] < 2:, where 2 is a random choice that could be changed into different values for obtaining different merging strategies.
Specify the configuration of accumulate_channel and attack_from_high_frequency_channel, then run the following script to launch attack.
python3 launch_empirical_attack_GH200.pyThe final accuracy will be printed out as attack_accuracy_mean <final_PSR>
Note that: using_pac_noise is the command to control using PAC privacy based noise or Differential Privacy based noise.
We also offer few scripts for researchers, who are interested in exploring the model more deeply, to see different statistics and configurations.
<path_to_privatar>/experiment_scripts/bdct_reconstruction/bdct_4x4_reconstruction_dataloader.ipynb includes the script to decomposes given unwrapped texture into different frequency components, and list few different configurations of decomposition.
For each configuration, all frequency components will be written to the folder <path_to_privatar>/experiment_scripts/bdct_reconstruction for visualization.
To help understand the covariance of different frequency components under differnet datasets, we provide the script to perform running profiling of all decomposed frequency components under designated datasets.
To run it,
cd <path_to_privatar>/multiface_partition_bdct4x4_ibdct
python3 launch_l2norm_freq_cov_analysis_GH200.pyIt will shows results like following
trace of covariance = 11308.309006199575 for freq component = 0
trace of covariance = 199.41605765586263 for freq component = 1
trace of covariance = 77.53071010274161 for freq component = 2
trace of covariance = 41.89610293635083 for freq component = 3
trace of covariance = 152.33769320529836 for freq component = 4
trace of covariance = 33.72750777014488 for freq component = 5
trace of covariance = 26.97710811919577 for freq component = 6
trace of covariance = 19.355714181792997 for freq component = 7
trace of covariance = 38.930325425557726 for freq component = 8
trace of covariance = 25.547661178709628 for freq component = 9
trace of covariance = 23.260128349715654 for freq component = 10
trace of covariance = 18.580644217635967 for freq component = 11
trace of covariance = 23.134909910006407 for freq component = 12
trace of covariance = 16.6648382626215 for freq component = 13
trace of covariance = 16.267422970259233 for freq component = 14
trace of covariance = 12.812623106426202 for freq component = 15Note, custom_scripts contains scripts for development, maintainence and verification purpose.
Have Fun! Enjoy! :D