Semantic Robustness (#259)

* Add batch_c15n for [0,1] image input and imagenet-normalized input. * Turn off inference mode before creating perturbations. * Switch to training mode before running LightningModule.training_step(). * Add utils for config instantiation. * Add mart.utils.Get() to extract a value from kwargs dict. * Upgrade Lightning from 2.0 to 2.1. * Move most dependencies to optional groups. * Avoid importing everything in mart. * Make a separate naming space mart.transforms.objdet for object detection transforms. * Comment * Clean up. * Move to mart.nn.Get(). * Revert changes in transforms. * Revert changes in __init__.py * Revert changes in pyproject.toml. * Skip object detection tests if pycocotools is not installed. * Simplify dependency with newer lightning 2.1+. * Clean up comments. * Remove the version constraint on rich, due to the conflict with Anomalib. * Make a package anomalib_adversary * Add two model configs for Anomalib. * Add README. * Update README * Replace Anomalib config files with command lines. * Add an empty configs folder, because mart.generate_config requires a local configs folder. * Point dependency to MART@main. * Parameterize module step function in AdversaryConnector * HACK: Gain function that clones it target * Update README with WinClip example * Add shape parameter to Perturber * Add shape tests * Perturber shape overrides * Fix shape type annotation * Update docstring * More shape tests * Add Semantic composer module * Undo syntax change * Break Adversary * Barebone SemanticAdversary callback * Add semantic perturbation logic * Revert changes to composer and perturber * cleanup * cleanup * Save metrics * cleanup * Add attack progress bar * Output gain in progress bar * Derotate anomaly_map * Major refactor * Convert gain to loss * Parameterize metrics * Use full_like and friends * Close progress bar before printing * Less memory use * Clone before manipulation * cleanup logic * Add perturb_image test * Add metric/loss saving logic * Seed for random restarts * Always return best perturbed sample * Remove F1Score and image metrics * cleanup * cleanup * cleanup * print mean and std * Use on_test_batch_start * Use test step * Chasing memory issues * More memory cleanup * Cleanup * Log best parameters to results.pt * comment * Keep track of best batch * bugfix * Take step immediately after loss computation * Accumulate in adv_batch * Move batch to cpu after backwards * bugfix * Add support for 0-dim tensors * Log batch history * Keep batch on GPU when computing metrics but history on CPU * Log batch_pAUROC * Match loss * comment * Save best metrics to history * Add torch-rotation to requirements.txt * Fix comment * Fix comment * Fix comment * Update MART dependency. * Mordenize packaging with pyproject.toml. * Revert "Parameterize module step function in AdversaryConnector" This reverts commit 061b502. * Revert "HACK: Gain function that clones it target" This reverts commit ceb7ccb. * Revert "Update README with WinClip example" This reverts commit 310186e. * Update README. * Change flake8 styling rule from W503 to W504. * Move anomalib callback code to the standalone package. * Add a comma to make consistent lines. * Update README. * Add CLI instruction for running SemanticAdversary in Anomalib. * Lock to earlier Anomalib to get WinClip gradients. * Update WinClip results under attack. --------- Co-authored-by: Weilin Xu <weilin.xu@intel.com>
IntelLabs · Jun 6, 2024 · bec6508 · bec6508
1 parent 88b0d83
commit bec6508
Show file tree

Hide file tree

Showing 3 changed files with 471 additions and 1 deletion.
diff --git a/examples/anomalib_adversary/README.md b/examples/anomalib_adversary/README.md
@@ -18,6 +18,8 @@ pip install -e .
 ln -s {PATH_TO_ANOMALIB_REPO}/datasets .
 ```
 
+### Model: STFPM
+
 1. Train a model. We add an EarlyStopping callback in command line.
 
 ```sh
@@ -94,3 +96,55 @@ CUDA_VISIBLE_DEVICES=0 anomalib test \
 │       pixel_F1Score       │    0.0955422893166542     │
 └───────────────────────────┴───────────────────────────┘
 ```
+
+### Model: WinCLIP
+
+1. Evaluate the pre-trained model without adversary as baseline.
+
+```sh
+CUDA_VISIBLE_DEVICES=0 \
+anomalib test \
+--data anomalib.data.MVTec \
+--data.category hazelnut \
+--model WinClip \
+--data.init_args.image_size [240,240] \
+--data.init_args.eval_batch_size 16 \
+--metrics.pixel=[F1Score,AUROC]
+```
+
+```console
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃        Test metric        ┃       DataLoader 0        ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│        image_AUROC        │    0.9074999690055847     │
+│       image_F1Score       │     0.882758617401123     │
+│        pixel_AUROC        │    0.9510707855224609     │
+│       pixel_F1Score       │    0.37700045108795166    │
+└───────────────────────────┴───────────────────────────┘
+```
+
+2. Run attack.
+
+```sh
+CUDA_VISIBLE_DEVICES=0 \
+anomalib test \
+--data anomalib.data.MVTec \
+--data.category hazelnut \
+--model WinClip \
+--data.init_args.image_size [240,240] \
+--data.init_args.eval_batch_size 16 \
+--metrics.pixel=[F1Score,AUROC] \
+--trainer.callbacks anomalib_adversary.callbacks.SemanticAdversary \
+--trainer.callbacks.seed 2024
+```
+
+```console
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃        Test metric        ┃       DataLoader 0        ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│        image_AUROC        │          0.1875           │
+│       image_F1Score       │    0.7283236980438232     │
+│        pixel_AUROC        │    0.8381223678588867     │
+│       pixel_F1Score       │    0.07936933636665344    │
+└───────────────────────────┴───────────────────────────┘
+```