diff --git a/README.md b/README.md index 7d27448e..cc71d562 100644 --- a/README.md +++ b/README.md @@ -98,11 +98,13 @@ python3 -m pip install git+https://github.com/dianna-ai/dianna.git If you get an error related to OpenMP when importing dianna, have a look at [this issue](https://github.com/dianna-ai/dianna/issues/376) for possible workarounds. -### Pre-requisites only for Macbook Pro with M1 Pro chip users - +
Pre-requisites only for Macbook Pro with M1 Pro chip users +

+ - To install TensorFlow you can follow this [tutorial](https://betterdatascience.com/install-tensorflow-2-7-on-macbook-pro-m1-pro/). - To install TensorFlow Addons you can follow these [steps](https://github.com/tensorflow/addons/pull/2504). For further reading see this [issue](https://github.com/tensorflow/addons/issues/2503). Note that this temporary solution works only for macOS versions >= 12.0. Note that this step may have changed already, see https://github.com/dianna-ai/dianna/issues/245. - Before installing DIANNA, comment `tensorflow` requirement in `setup.cfg` file (tensorflow package for M1 is called `tensorflow-macos`). +

## Getting started @@ -141,7 +143,9 @@ explanation = dianna.(model_path, data_item, explainer) dianna.visualization.(explanation[explained_class_index], data_item) ``` -### Text and image usage examples +### Text and image usage +
Examples +

Lets illustrate the template above with *textual* data. The data item of interest is a sentence being (a part of) a movie review and the model has been trained to classify reviews into positive and negative sentiment classes. We are intersted which words are contributing positively (red) and which - negatively (blue) towards the model's desicion to classify the review as positive and we would like to use the *LIME* explainer: @@ -182,32 +186,35 @@ plt.show() +

+ ### Overview tutorial There are **full working examples** on how to use the supported explainers and how to use dianna for **all supported data modalities** in our [overview tutorial](./tutorials/overview.ipynb). -#### Demo movie (update planned): -[![Watch the video on YouTube](https://img.youtube.com/vi/u9_c5DJewLU/default.jpg)](https://youtu.be/u9_c5DJewLU) - ### IMPORTANT: Sensitivity to hyperparameters The explainers are sensitive to the choice of their hyperparameters! In this [work](https://staff.fnwi.uva.nl/a.s.z.belloum/MSctheses/MScthesis_Willem_van_der_Spec.pdf), this sensitivity to hyperparameters is researched and useful conclusions are drawn. The default hyperparameters used in DIANNA for each explainer as well as the values for our tutorial examples are given in the Tutorials [README](./tutorials/README.md#important-hyperparameters). +### Introductory video +This video shows the main functionality of DIANNA and shows you how to use DIANNA also from its dashboard. + +![Watch the video on YouTube](https://youtu.be/9VM5acip2s8) + ## Dashboard -Explore the explanations of your trained model using the DIANNA dashboard (for now images, text and time series classification is supported). +![image](https://github.com/user-attachments/assets/27d621d0-eaab-4640-9353-691ae7adb12b) + + +Explore the explanations of your trained model using the DIANNA dashboard. [Click here](https://github.com/dianna-ai/dianna/tree/main/dianna/dashboard) for more information. -_Dianna dashboard screenshot here_ - + ## Datasets DIANNA comes with simple datasets. Their main goal is to provide intuitive insight into the working of the XAI methods. They can be used as benchmarks for evaluation and comparison of existing and new XAI methods. -### Images +
Images | Dataset | Description | Examples | Generation | | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -215,29 +222,38 @@ DIANNA comes with simple datasets. Their main goal is to provide intuitive insig | [Simple Geometric (circles and triangles)](https://doi.org/10.5281/zenodo.5012824) Simple Geometric Logo | Images of circles and triangles for 2-class geometric shape classificaiton. The shapes of varying size and orientation and the background have varying uniform gray levels. | SimpleGeometric | [Simple geometric shapes dataset generation](https://github.com/dianna-ai/dianna-exploration/tree/main/example_data/dataset_preparation/geometric_shapes) | | [Simple Scientific (LeafSnap30)](https://zenodo.org/record/5061353/) LeafSnap30 Logo | Color images of tree leaves - a 30-class post-processed subset from the LeafSnap dataset for automatic identification of North American tree species. | LeafSnap | [LeafSnap30 dataset generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/dataset_preparation/LeafSnap/) | -### Text +
+ +
Text +

| Dataset | Description | Examples | Generation | | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------- | :--------------------------------------------------------------- | :------------------------------------------------------------------ | | [Stanford sentiment treebank](https://nlp.stanford.edu/sentiment/index.html) nlp-logo_half_size | Dataset for predicting the sentiment, positive or negative, of movie reviews. | _This movie was actually neither that funny, nor super witty._ | [Sentiment treebank](https://nlp.stanford.edu/sentiment/treebank.html) | +| [EU-law statements](https://zenodo.org/records/8200000) nlp-logo_half_size | Reproducibility data for a quantitative study on EU legislation. | _A Member State wishing to grant exemptions referred to in paragraph 6 shall notify the Council in writing_ | [EU legislation strictness analysis](https://github.com/nature-of-eu-rules/eu-legislation-strictness-analysis) | +

-### Time series +
Time series +

| Dataset | Description | Examples | Generation | | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------ | | [Coffee dataset](https://www.timeseriesclassification.com/description.php?Dataset=Coffee) Coffe Logo | Food spectographs time series dataset for a two class problem to distinguish between Robusta and Arabica coffee beans. | example image | [data source](https://github.com/QIBChemometrics/Benchtop-NMR-Coffee-Survey) | | [Weather dataset](https://zenodo.org/record/7525955) Weather Logo | The light version of the weather prediciton dataset, which contains daily observations (89 features) for 11 European locations through the years 2000 to 2010. | example image | [data source](https://github.com/florian-huber/weather_prediction_dataset) | -### Tabular +

+ +
Tabular +

| Dataset | Description | Examples | Generation | | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------ | | [Pengiun dataset](https://www.kaggle.com/code/parulpandey/penguin-dataset-the-new-iris) Penguins Logo | Palmer Archipelago (Antarctica) penguin dataset is a great intro dataset for data exploration & visualization similar to the famous Iris dataset. | example image | [data source](https://github.com/allisonhorst/palmerpenguins) | | [Weather dataset](https://zenodo.org/record/7525955) Weather Logo | The light version of the weather prediciton dataset, which contains daily observations (89 features) for 11 European locations through the years 2000 to 2010. | example image | [data source](https://github.com/florian-huber/weather_prediction_dataset) | | [Land atmosphere dataset](https://zenodo.org/records/12623257) Atmosphere Logo | It contains land-atmosphere variables and latent heat flux (LEtot) simulated by STEMMUS-SCOPE (soil-plant model), version 1.5.0, over 19 Fluxnet sites and for the year 2014 with hourly intervals. | example image | [data source](https://zenodo.org/records/12623257) | -## ONNX model +

- +## Models **We work with ONNX!** ONNX is a great unified neural network standard which can be used to boost reproducible science. Using ONNX for your model also gives you a boost in performance! In case your models are still in another popular DNN (deep neural network) format, here are some simple recipes to convert them: @@ -250,22 +266,25 @@ More converters with examples and tutorials can be found on the [ONNX tutorial p And here are links to notebooks showing how we created our models on the benchmark datasets: -### Images +
Images | Models | Generation | | :-------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Binary MNIST model](https://zenodo.org/record/5907177) | [Binary MNIST model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/MNIST/generate_model_binary.ipynb) | | [Simple Geometric model](https://zenodo.org/deposit/5907059) | [Simple geometric shapes model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/geometric_shapes/generate_model.ipynb) | | [Simple Scientific model](https://zenodo.org/record/5907196) | [LeafSnap30 model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/LeafSnap/generate_model.ipynb) | +
-### Text +
Text | Models | Generation | |:---------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Movie reviews model](https://zenodo.org/record/5910598) | [Stanford sentiment treebank model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/movie_reviews/generate_model.ipynb) | | [Regalatory statement classifier](https://zenodo.org/record/8200001) | [EU-law regulatory-statement-classification](https://github.com/nature-of-eu-rules/regulatory-statement-classification) | -### Time series +
+ +
Time series | Models | Generation | | :-------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -273,7 +292,9 @@ And here are links to notebooks showing how we created our models on the benchma | [Season prediction model](https://zenodo.org/record/7543883) | [Season prediction model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/season_prediction/generate_model.ipynb) | | [Fast Radio Burst classification model](https://zenodo.org/records/10656614) | [Fast Radio Burst classification model generation](https://doi.org/10.3847/1538-3881/aae649) | -### Tabular +
+ +
Tabular | Models | Generation | | :-------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -281,6 +302,7 @@ And here are links to notebooks showing how we created our models on the benchma | [Sunshine hours prediction model (regression)](https://zenodo.org/records/10580833) | [Sunshine hours prediction model generation](https://github.com/dianna-ai/dianna-exploration/blob/main/example_data/model_generation/sunshine_prediction/generate_model.ipynb) | | [Latent heat flux prediction model (regression)](https://zenodo.org/records/12623257) | [Latent heat flux prediction model](doi:10.5281/zenodo.12623256/stemmus_scope_emulator_model_LEtot.onnx) | +
**_We envision the birth of the ONNX Scientific models zoo soon..._** diff --git a/dianna/dashboard/Home.py b/dianna/dashboard/Home.py index 8cb1cfe5..cefd0673 100644 --- a/dianna/dashboard/Home.py +++ b/dianna/dashboard/Home.py @@ -1,5 +1,6 @@ import importlib import streamlit as st +from _shared import add_sidebar_goddess_logo from _shared import data_directory from streamlit_option_menu import option_menu @@ -38,7 +39,7 @@ # Display the content of the selected page if selected == "Home": - + add_sidebar_goddess_logo() _, col, _ = st.columns([1, 3, 1]) with col: st.markdown("""#""") @@ -90,10 +91,8 @@ You can then select the explainer you want to use and set its hyperparameters. ### More information - - - [Source code](https://github.com/dianna-ai/dianna) + - [Research Software directory page](https://research-software-directory.org/software/dianna) - [Documentation](https://dianna.readthedocs.io/) - - [XAI choice](https://blog.esciencecenter.nl/how-to-find-your-artificial-intelligence-explainer-dbb1ac608009) """, unsafe_allow_html=True) diff --git a/dianna/dashboard/_shared.py b/dianna/dashboard/_shared.py index 3db50cab..45fac24b 100644 --- a/dianna/dashboard/_shared.py +++ b/dianna/dashboard/_shared.py @@ -47,6 +47,9 @@ def add_sidebar_logo(): """Upload DIANNA logo to sidebar element.""" st.sidebar.image(str(data_directory / 'logo.png')) +def add_sidebar_goddess_logo(): + """Upload DIANNA goddess logo to sidebar element.""" + st.sidebar.image(str(data_directory / 'Diana_goddess.png')) def _methods_checkboxes(*, choices: Sequence, key): """Get methods from a horizontal row of checkboxes and the corresponding parameters.""" diff --git a/dianna/dashboard/readme.md b/dianna/dashboard/readme.md index 46d7d164..c5fb2c5c 100644 --- a/dianna/dashboard/readme.md +++ b/dianna/dashboard/readme.md @@ -1,9 +1,7 @@ # DIANNA dashboard - -_Dianna dashboard screenshot here_ - -The DIANNA dashboard can be used for simple exploration of your trained model explained by DIANNA. The dashboard produces the visual explanation of your selected XAI method. Additionally it allows you to compare the results of different XAI methods, as well as explanations of the top ranked predicted labels. The dashboard was created using [streamlit](https://streamlit.io/). +The DIANNA dashboard (powered by [streamlit](https://streamlit.io/) Streamlit Logo) can be used for explanation of ONNX models trained for the tasks and datasets presented in several (marked with Streamlit Logo) of the DIANNA [tutorials](../../tutorials/README.md). +The dashboard shows the visual explanation of a models' decision on a selected data item by a selected XAI method (explainer). Results of different explainers can be compared, as well as explanations of the top predicted labels. To open the dashboard, you can install dianna via `pip install -e '.[dashboard]'` and run: @@ -21,26 +19,8 @@ Open the link on which the app is running. Note that you are running the dashboa ## How to use the dashboard -The dashboard will automatically open in the welcome page tab. In the sidebar you can open the image or text pages. - -### Images - -- Click on select image and select an image saved on your computer for which you want the DIANNA explanation. -- Click on select model and select your trained model used for the prediction and explanation. -- Click on select labels and select a text file containing the labels of your trained model. Labels should be separated by line breaks in the file, no other separators should be used. Make sure that both the labels and ordering of the labels is correct in correspondence to your trained model. -- Check the XAI methods you want to use, multiple methods can be used at the same time for comparison. -- Your image explanation will start loading. - -Additionally, you can: -- Select the number of top results you which to show the explanation for, i.e. if you use 2, the dashboard will show not only the explanation for the main prediction for the selected image, but also for the second most likely prediction. -- Set the method specific settings to the selected XAI methods. If you don't change these, the default values are used. - -### Text - -- Input the text you want to use in the "Input string" box. Press enter to update the explanation. -- Click on select model and select your trained model used for the prediction and explanation. -- Click on select labels and select a text file containing the labels of your trained model. Labels should be separated by line breaks in the file, no other separators should be used. Make sure that both the labels and ordering of the labels is correct in correspondence to your trained model. -- Check the XAI methods you want to use, multiple methods can be used at the same time for comparison. -- Your image explanation will start loading. +This [video](https://youtu.be/9VM5acip2s8) shows you how to use the DIANNA dashboard for some of the Streamlit Logo [tutorials](../../tutorials/README.md#summary-of-all-tutorials) to interactively explore the use cases from some of DIANNA's . +Similarly, you can use it with your own data, models and lables. -Additionally, you can set the method specific settings to the selected XAI methods. If you don't change these, the default values are used. +-------------------------------------------------------------------------- +![image](https://github.com/user-attachments/assets/1a98920e-f75e-468c-bf1f-f6e8bd2273ad) diff --git a/dianna/data/Diana_goddess.png b/dianna/data/Diana_goddess.png new file mode 100644 index 00000000..37329890 Binary files /dev/null and b/dianna/data/Diana_goddess.png differ diff --git a/setup.cfg b/setup.cfg index 81798c4e..215c344d 100644 --- a/setup.cfg +++ b/setup.cfg @@ -115,7 +115,7 @@ notebooks = torchvision ipywidgets freetype-py - transformers + transformers<4.46.0 xgboost [options.entry_points] diff --git a/tutorials/README.md b/tutorials/README.md index 9890ce6c..35c08940 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -16,7 +16,7 @@ pip install .[notebooks] ### Datasets and Tasks -#### Illustrative (Simple) +
Illustrative (Simple) |*Data modality*|Dataset| *Task* |Logo| |:------------|:------|:----------------------------------------------------------------------|:----| @@ -28,46 +28,57 @@ pip install .[notebooks] | | [Weather dataset](https://zenodo.org/record/7525955) | Binary *classification* (warm/cold season) of temperature time-series |Weather Logo| |*Tabular*| [Penguin dataset](https://www.kaggle.com/code/parulpandey/penguin-dataset-the-new-iris)| $3$ penguin spicies (Adele, Chinstrap, Gentoo) *classificaiton* | Penguin Logo | | | | [Weather dataset](https://zenodo.org/record/7525955) | Next day sunshine hours prediction (*regression*) | Weather Logo| +
-#### Scientific use-cases - +
Scientific use-cases + |*Data modality*|Dataset|*Task*|Logo| |:------------|:------|:---|:----| |*Images*|[Simple Scientific (LeafSnap30)](https://zenodo.org/record/5061353/)| $30$ tree species leaves *classification* | LeafSnap30 Logo | -|*Text*| [EU-law statements](https://zenodo.org/records/8200001) | Regulatory or non-regulatory *classification* | nlp-logo_half_size| +|*Text*| [EU-law statements](https://zenodo.org/records/8200000) | Regulatory or non-regulatory *classification* | nlp-logo_half_size| |*Timeseries* | Fast Radio Burst (FRB) dataset (not publicly available) | Binary *classificaiton* of Fast Radio Burst (FRB) timeseries data : noise or a real FRB. | FRB logo| |*Tabular*| [Land atmosphere dataset](https://zenodo.org/records/12623257)| Prediction of "latent heat flux" (*regression*). The random forest model is used as an [emulator](https://github.com/EcoExtreML/Emulator) to replace the physical model [STEMMUS_SCOPE](https://github.com/EcoExtreML/STEMMUS_SCOPE) to predict global maps of latent heat flux. | Atmosphere Logo | +
### Models The ONNX models used in the tutorials are available at [dianna/models](https://github.com/dianna-ai/dianna/tree/main/dianna/models), or linked from their respective tutorial notebooks. ### Summary of all Tutorials +All tutorials can be accessed by clicking on the dataset & task logo in the tables below. + +The explainers' output for the models trained on the datasets & tasks which are included in the [dashboard](../dianna/dashboard/readme.md) are marked with Streamlit Logo. -#### Illustrative (Simple) +
Illustrative (Simple) + |*Modality* \ Method|RISE|[LIME](https://youtu.be/d6j6bofhj2M)|Kernel[SHAP](https://youtu.be/9haIOplEIGM)| |:-----|:---|:---|:---| -|*Images*|[mnist_zero_and_one_half_size](./explainers/RISE/rise_mnist.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_mnist.ipynb) | | [mnist_zero_and_one_half_size](./explainers/KernelSHAP/kernelshap_mnist.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_mnist.ipynb) | +|*Images*|[mnist_zero_and_one_half_size](./explainers/RISE/rise_mnist.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_mnist.ipynb) Streamlit Logo|Streamlit Logo | [mnist_zero_and_one_half_size](./explainers/KernelSHAP/kernelshap_mnist.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_mnist.ipynb) Streamlit Logo | | | [ImageNet_autocrop](./explainers/RISE/rise_imagenet.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_imagenet.ipynb) | | [SimpleGeometric Logo](./explainers/KernelSHAP/kernelshap_geometric_shapes.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_geometric_shapes.ipynb)| -|*Text* |[nlp-logo_half_size](./explainers/RISE/rise_text.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_text.ipynb) |[nlp-logo_half_size](./explainers/LIME/lime_text.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_text.ipynb) |[]()| -| *Time series*| [Weather Logo](./explainers/RISE/rise_timeseries_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_timeseries_weather.ipynb)| [Weather Logo](./explainers/LIME/lime_timeseries_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_timeseries_weather.ipynb)| | +|*Text* |[nlp-logo_half_size](./explainers/RISE/rise_text.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_text.ipynb) Streamlit Logo |[nlp-logo_half_size](./explainers/LIME/lime_text.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_text.ipynb) Streamlit Logo |[]()| +| *Time series*| [Weather Logo](./explainers/RISE/rise_timeseries_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_timeseries_weather.ipynb) Streamlit Logo| [Weather Logo](./explainers/LIME/lime_timeseries_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_timeseries_weather.ipynb) Streamlit Logo| | | | | [Coffee Logo](./explainers/LIME/lime_timeseries_coffee.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_timeseries_coffee.ipynb) | | -| *Tabular* | [Penguin Logo](./explainers/RISE/rise_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_tabular_penguin.ipynb) | [Penguin Logo](./explainers/LIME/lime_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_tabular_penguin.ipynb) |[Penguin Logo](./explainers/KernelSHAP/kernelshap_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_penguin.ipynb) | -| | | [Weather Logo](./explainers/LIME/lime_tabular_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_weather.ipynb)|[Weather Logo](./explainers/KernelSHAP/kernelshap_tabular_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_weather.ipynb) | +| *Tabular* | [Penguin Logo](./explainers/RISE/rise_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_tabular_penguin.ipynb) Streamlit Logo| [Penguin Logo](./explainers/LIME/lime_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_tabular_penguin.ipynb) Streamlit Logo |[Penguin Logo](./explainers/KernelSHAP/kernelshap_tabular_penguin.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_penguin.ipynb) Streamlit Logo| +| |Streamlit Logo | [Weather Logo](./explainers/LIME/lime_tabular_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_weather.ipynb) Streamlit Logo|[Weather Logo](./explainers/KernelSHAP/kernelshap_tabular_weather.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_weather.ipynb) Streamlit Logo| To learn more about how we aproach the masking for time-series data, please read our [Masking time-series for XAI](https://blog.esciencecenter.nl/masking-time-series-for-explainable-ai-90247ac252b4) blog-post. +
-#### Scientific use-cases +
Scientific use-cases | *Modality* \ Method |RISE| [LIME](https://youtu.be/d6j6bofhj2M) |Kernel[SHAP](https://youtu.be/9haIOplEIGM)| |:--------------------|:---|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---| | *Images* | | [LeafSnap30 Logo](./explainers/LIME/lime_images.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_images.ipynb) || -| | | [FRB logo](./explainers/RISE/rise_timeseries_frb.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_timeseries_frb.ipynb) | | -| *Text* | | [nlp-logo_half_size](./explainers/LIME/lime_text_eulaw.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_text_eulaw.ipynb) | | -| *Time series* | [FRB logo](./explainers/RISE/rise_timeseries_frb.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_timeseries_frb.ipynb) | | +| *Text* | | [nlp-logo_half_size](./explainers/LIME/lime_text_eulaw.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/LIME/lime_text_eulaw.ipynb) Streamlit Logo| | +| *Time series* | [FRB logo](./explainers/RISE/rise_timeseries_frb.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/RISE/rise_timeseries_frb.ipynb) Streamlit Logo | | | *Tabular* | | |[Atmosphere Logo](./explainers/KernelSHAP/kernelshap_tabular_land_atmosphere.ipynb) or [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dianna-ai/dianna/blob/main/tutorials/explainers/KernelSHAP/kernelshap_tabular_land_atmosphere.ipynb)| +
+ ### IMPORTANT: Hyperparameters + +
Settings per explainer + The XAI methods (explainers) are sensitive to the choice of their hyperparameters! In this [master Thesis](https://staff.fnwi.uva.nl/a.s.z.belloum/MSctheses/MScthesis_Willem_van_der_Spec.pdf), this sensitivity is researched and useful conclusions are drawn. The default hyperparameters used in DIANNA for each explainer as well as the choices for some tutorials and their data modality (*i* - images, *txt* - text, *ts* - time series and *tab* - tabular) are given in the tables below. Also the main conclusions (🠊) from the thesis (on images and text) about the hyperparameters effect are listed. @@ -101,6 +112,9 @@ Also the main conclusions (🠊) from the thesis (on images and text) about the | $n_{segments}$ | **$100$** |$200$ |$200$ |default | | $sigma$ | **$0$** | default | default | default| + 🠊 The most crucial parameter is the nubmer of super-pixels $n_{segments}$. Higher values led to higher sensitivity, however that observaiton was dependant on the evaluaiton metric. 🠊 Regularization had only a marginal detrimental effect, the best results were obtained using no regularization (no smoothing, $sigma = 0$) or least squares regression. + +