From 9bdb9c4e501243deac917f3ffa831b24e2a3fc5a Mon Sep 17 00:00:00 2001 From: taojiashu Date: Mon, 4 Nov 2024 15:05:29 +0800 Subject: [PATCH 1/2] update README to include conda installation speed up and information about the demo configs --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a10467b..76877ac 100644 --- a/README.md +++ b/README.md @@ -45,20 +45,21 @@ To install the dependencies, run the following command: ``` conda env create -f env.yaml ``` -This should create a conda environment named `privacy_meter` and install all necessary libraries in it. +This should create a conda environment named `privacy_meter` and install all necessary libraries in it. If conda takes too much time (more than a few minutes) to solve the environment, we suggest updating the conda default solver by following this official [article](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community). + To run our demo, you can use the following command ``` python main.py --cf configs/config.yaml ``` -The `.yaml` file allows you to specify the hyperparameters for training the model, and the details of the membership inference attack. +The `.yaml` file allows you to specify the hyperparameters for training the model, and the details of the membership inference attack. To shorten the time to run the demo, we set the number of epochs to 10. To properly audit the privacy risk, we suggest change the number of epochs to 100 or whatever is appropriate for your use case. For a comprehensive explanation of each parameter, please refer to each `.yaml` file and the explanation [here](configs/README.md). You can also refer to the [demo notebook](demo.ipynb) for a step-by-step walkthrough. Upon audit completion, you will find the results in the `demo` folder, with the attack results saved in `demo/report`. Furthermore, we also offer a timing log for each run, which can be found in the file `log_time_analysis.log`. We recommend running each new set of experiments with different hyperparameters under a different `log_dir` to avoid misusing old trained models or losing previous results. ### Supported dataset and models by default -By default, Privacy Meter supports various datasets widely used in the MIA literature, including CIFAR10 (`cifar10`), CIFAR100 (`cifar100`), Purchase (`purchase100`), Texas (`texas100`), and AG News (`agnews`). In terms of models, we provide support for CNN (`cnn`), AlexNet (`alexnet`), WideResNet (`wrn28-1`, `wrn28-2`, `wrn28-10`), MLP (`mlp`), and GPT-2 (`gpt2`) models. To specify the dataset and model, you can use the `dataset` and `model_name` parameters in the configuration file. Sample configurations have been provided in the `configs` folder for Purchase-100, CIFAR-10 and AG News dtasets. +By default, Privacy Meter supports various datasets widely used in the MIA literature, including CIFAR10 (`cifar10`), CIFAR100 (`cifar100`), Purchase (`purchase100`), Texas (`texas100`), and AG News (`agnews`). In terms of models, we provide support for CNN (`cnn`), AlexNet (`alexnet`), WideResNet (`wrn28-1`, `wrn28-2`, `wrn28-10`), MLP (`mlp`), and GPT-2 (`gpt2`) models. To specify the dataset and model, you can use the `dataset` and `model_name` parameters in the configuration file. Sample configurations have been provided in the `configs` folder for Purchase-100, CIFAR-10 and AG News datasets. ## Extending to Other Datasets and Models ### Attacking LLMs with other datasets @@ -121,7 +122,6 @@ The Privacy Meter is built upon the following research papers ([bib file](CITATI 3. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Reza Shokri. [Enhanced Membership Inference Attacks against Machine Learning Models](https://arxiv.org/pdf/2111.09679.pdf) in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022. - 4. Milad Nasr, Reza Shokri, and Amir Houmansadr. [Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks](https://www.comp.nus.edu.sg/~reza/files/Shokri-SP2019.pdf) in IEEE Symposium on Security and Privacy, 2019. 5. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. [Membership Inference Attacks against Machine Learning Models](https://www.comp.nus.edu.sg/~reza/files/Shokri-SP2017.pdf) in IEEE Symposium on Security and Privacy, 2017. From 2e78664210f7a61c1b0f69e8a307e9dd05c84458 Mon Sep 17 00:00:00 2001 From: taojiashu Date: Tue, 5 Nov 2024 15:40:35 +0800 Subject: [PATCH 2/2] provide a pip requirement file --- README.md | 4 ++ requirements.txt | 151 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 155 insertions(+) create mode 100644 requirements.txt diff --git a/README.md b/README.md index 76877ac..164241e 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,10 @@ flowchart LR ### Getting started To install the dependencies, run the following command: ``` +pip install -r requirements.txt +``` +Alternatively, if you prefer using conda, you can create a new environment using the provided `env.yaml` file: +``` conda env create -f env.yaml ``` This should create a conda environment named `privacy_meter` and install all necessary libraries in it. If conda takes too much time (more than a few minutes) to solve the environment, we suggest updating the conda default solver by following this official [article](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community). diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..e549424 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,151 @@ +accelerate==1.1.0 +aiohappyeyeballs==2.4.3 +aiohttp==3.10.10 +aiosignal==1.3.1 +anyio==4.6.2.post1 +argon2-cffi==23.1.0 +argon2-cffi-bindings==21.2.0 +arrow==1.3.0 +asttokens==2.4.1 +async-lru==2.0.4 +attrs==24.2.0 +babel==2.16.0 +beautifulsoup4==4.12.3 +bleach==6.2.0 +certifi==2024.8.30 +cffi==1.17.1 +charset-normalizer==3.4.0 +comm==0.2.2 +contourpy==1.3.0 +cycler==0.12.1 +datasets==2.21.0 +debugpy==1.8.7 +decorator==5.1.1 +defusedxml==0.7.1 +dill==0.3.8 +executing==2.1.0 +fastjsonschema==2.20.0 +filelock==3.13.1 +fonttools==4.54.1 +fqdn==1.5.1 +frozenlist==1.5.0 +fsspec==2024.2.0 +h11==0.14.0 +httpcore==1.0.6 +httpx==0.27.2 +huggingface-hub==0.26.2 +idna==3.10 +ipykernel==6.29.5 +ipython==8.29.0 +ipywidgets==8.1.5 +isoduration==20.11.0 +jedi==0.19.1 +Jinja2==3.1.3 +joblib==1.4.2 +json5==0.9.25 +jsonpointer==3.0.0 +jsonschema==4.23.0 +jsonschema-specifications==2024.10.1 +jupyter==1.1.1 +jupyter-console==6.6.3 +jupyter-events==0.10.0 +jupyter-lsp==2.2.5 +jupyter_client==8.6.3 +jupyter_core==5.7.2 +jupyter_server==2.14.2 +jupyter_server_terminals==0.5.3 +jupyterlab==4.2.5 +jupyterlab_pygments==0.3.0 +jupyterlab_server==2.27.3 +jupyterlab_widgets==3.0.13 +kiwisolver==1.4.7 +MarkupSafe==2.1.5 +matplotlib==3.9.2 +matplotlib-inline==0.1.7 +mistune==3.0.2 +mpmath==1.3.0 +multidict==6.1.0 +multiprocess==0.70.16 +nbclient==0.10.0 +nbconvert==7.16.4 +nbformat==5.10.4 +nest-asyncio==1.6.0 +networkx==3.2.1 +notebook==7.2.2 +notebook_shim==0.2.4 +numpy==1.26.3 +nvidia-cublas-cu11==11.11.3.6 +nvidia-cuda-cupti-cu11==11.8.87 +nvidia-cuda-nvrtc-cu11==11.8.89 +nvidia-cuda-runtime-cu11==11.8.89 +nvidia-cudnn-cu11==9.1.0.70 +nvidia-cufft-cu11==10.9.0.58 +nvidia-curand-cu11==10.3.0.86 +nvidia-cusolver-cu11==11.4.1.48 +nvidia-cusparse-cu11==11.7.5.86 +nvidia-nccl-cu11==2.20.5 +nvidia-nvtx-cu11==11.8.86 +overrides==7.7.0 +packaging==24.1 +pandas==2.2.3 +pandocfilters==1.5.1 +parso==0.8.4 +peft==0.13.1 +pexpect==4.9.0 +pillow==10.2.0 +platformdirs==4.3.6 +prometheus_client==0.21.0 +prompt_toolkit==3.0.48 +propcache==0.2.0 +psutil==6.1.0 +ptyprocess==0.7.0 +pure_eval==0.2.3 +pyarrow==18.0.0 +pycparser==2.22 +Pygments==2.18.0 +pyparsing==3.2.0 +python-dateutil==2.9.0.post0 +python-json-logger==2.0.7 +pytz==2024.2 +PyYAML==6.0.2 +pyzmq==26.2.0 +referencing==0.35.1 +regex==2024.9.11 +requests==2.32.3 +rfc3339-validator==0.1.4 +rfc3986-validator==0.1.1 +rpds-py==0.20.1 +safetensors==0.4.5 +scikit-learn==1.5.2 +scipy==1.14.1 +Send2Trash==1.8.3 +setuptools==70.0.0 +six==1.16.0 +sniffio==1.3.1 +soupsieve==2.6 +stack-data==0.6.3 +sympy==1.13.1 +terminado==0.18.1 +threadpoolctl==3.5.0 +tinycss2==1.4.0 +tokenizers==0.19.1 +torch==2.4.1+cu118 +torchaudio==2.4.1+cu118 +torchvision==0.19.1+cu118 +tornado==6.4.1 +tqdm==4.66.6 +traitlets==5.14.3 +transformers==4.44.2 +triton==3.0.0 +types-python-dateutil==2.9.0.20241003 +typing_extensions==4.9.0 +tzdata==2024.2 +uri-template==1.3.0 +urllib3==2.2.3 +wcwidth==0.2.13 +webcolors==24.8.0 +webencodings==0.5.1 +websocket-client==1.8.0 +widgetsnbextension==4.0.13 +xxhash==3.5.0 +yarl==1.17.1