diff --git a/README.md b/README.md index 3635ec0..724459c 100644 --- a/README.md +++ b/README.md @@ -158,6 +158,7 @@ Options: - [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? - [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? - [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? + - [ ] **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? ## B. Data Storage - [ ] **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? diff --git a/deon/assets/checklist.yml b/deon/assets/checklist.yml index 91a047c..814eea6 100644 --- a/deon/assets/checklist.yml +++ b/deon/assets/checklist.yml @@ -1,5 +1,5 @@ title: Data Science Ethics Checklist -sections: +sections: - title: Data Collection section_id: A lines: @@ -12,6 +12,9 @@ sections: - line_id: A.3 line_summary: Limit PII exposure line: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? + - line_id: A.4 + line_summary: Downstream bias mitigation + line: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? - title: Data Storage section_id: B lines: diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml index 9a8858e..f114f63 100644 --- a/deon/assets/examples_of_ethical_issues.yml +++ b/deon/assets/examples_of_ethical_issues.yml @@ -16,6 +16,14 @@ url: https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn - text: Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets. url: https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/ +- line_id: A.4 + links: + - text: In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods. + url: https://www.bloomberg.com/graphics/2016-amazon-same-day/ + - text: Facial recognition software is significanty worse at identifying people with darker skin. + url: https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/ + - text: -- Related academic study. + url: http://proceedings.mlr.press/v81/buolamwini18a.html - line_id: B.1 links: - text: Personal and financial data for more than 146 million people was stolen in Equifax data breach. @@ -52,6 +60,8 @@ links: - text: Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services. url: https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/ + - text: Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically. + url: https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening - line_id: C.4 links: - text: Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts. @@ -62,8 +72,6 @@ url: https://www.bbc.com/news/magazine-22223190 - line_id: D.1 links: - - text: In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods. - url: https://www.bloomberg.com/graphics/2016-amazon-same-day/ - text: Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny. url: https://www.wired.com/story/excerpt-from-automating-inequality/ - text: Amazon scraps AI recruiting tool that showed bias against women. @@ -74,6 +82,8 @@ url: https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know - line_id: D.2 links: + - text: Apple credit card offers smaller lines of credit to women than men. + url: https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/ - text: Google Photos tags two African-Americans as gorillas. url: https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d - text: With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend. @@ -84,10 +94,6 @@ url: https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047 - text: Google's speech recognition software doesn't recognize women's voices as well as men's. url: https://www.dailydot.com/debug/google-voice-recognition-gender-bias/ - - text: Facial recognition software is significanty worse at identifying people with darker skin. - url: https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/ - - text: -- Related academic study. - url: http://proceedings.mlr.press/v81/buolamwini18a.html - text: Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names. url: https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/ - text: -- Related academic study. diff --git a/docs/docs/examples.md b/docs/docs/examples.md index 6dbbecf..bf19c52 100644 --- a/docs/docs/examples.md +++ b/docs/docs/examples.md @@ -10,6 +10,7 @@ To make the ideas contained in the checklist more concrete, we've compiled examp **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | +**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | |
**Data Storage**
**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | **B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | @@ -17,12 +18,12 @@ To make the ideas contained in the checklist more concrete, we've compiled examp |
**Analysis**
**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | **C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | -**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | +**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | **C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | |
**Modeling**
-**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | -**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | +**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | +**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | **D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | **D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | **D.5 Communicate bias**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | diff --git a/docs/docs/index.md b/docs/docs/index.md index ede260d..d809372 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -151,6 +151,7 @@ Options: - [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? - [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? - [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? + - [ ] **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? ## B. Data Storage - [ ] **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? diff --git a/examples/ethics.html b/examples/ethics.html index 565a12f..2ed83bd 100644 --- a/examples/ethics.html +++ b/examples/ethics.html @@ -36,6 +36,13 @@

Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? +
  • + + + A.4 Downstream bias mitigation: + + Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? +

    • diff --git a/examples/ethics.ipynb b/examples/ethics.ipynb index 26519b9..2d5dc17 100644 --- a/examples/ethics.ipynb +++ b/examples/ethics.ipynb @@ -1 +1 @@ -{"nbformat": 4, "nbformat_minor": 2, "metadata": {}, "cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Data Science Ethics Checklist\n", "\n", "[![Deon badge](https://img.shields.io/badge/ethics%20checklist-deon-brightgreen.svg?style=popout-square)](http://deon.drivendata.org/)\n", "\n", "## A. Data Collection\n", " - [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent?\n", " - [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those?\n", " - [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis?\n", "\n", "## B. Data Storage\n", " - [ ] **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)?\n", " - [ ] **B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed?\n", " - [ ] **B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed?\n", "\n", "## C. Analysis\n", " - [ ] **C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)?\n", " - [ ] **C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)?\n", " - [ ] **C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data?\n", " - [ ] **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis?\n", " - [ ] **C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future?\n", "\n", "## D. Modeling\n", " - [ ] **D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory?\n", " - [ ] **D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)?\n", " - [ ] **D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics?\n", " - [ ] **D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed?\n", " - [ ] **D.5 Communicate bias**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood?\n", "\n", "## E. Deployment\n", " - [ ] **E.1 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)?\n", " - [ ] **E.2 Roll back**: Is there a way to turn off or roll back the model in production if necessary?\n", " - [ ] **E.3 Concept drift**: Do we test and monitor for concept drift to ensure the model remains fair over time?\n", " - [ ] **E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed?\n", "\n", "*Data Science Ethics Checklist generated with [deon](http://deon.drivendata.org).*\n"]}]} \ No newline at end of file +{"nbformat": 4, "nbformat_minor": 2, "metadata": {}, "cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Data Science Ethics Checklist\n", "\n", "[![Deon badge](https://img.shields.io/badge/ethics%20checklist-deon-brightgreen.svg?style=popout-square)](http://deon.drivendata.org/)\n", "\n", "## A. Data Collection\n", " - [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent?\n", " - [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those?\n", " - [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis?\n", " - [ ] **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)?\n", "\n", "## B. Data Storage\n", " - [ ] **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)?\n", " - [ ] **B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed?\n", " - [ ] **B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed?\n", "\n", "## C. Analysis\n", " - [ ] **C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)?\n", " - [ ] **C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)?\n", " - [ ] **C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data?\n", " - [ ] **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis?\n", " - [ ] **C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future?\n", "\n", "## D. Modeling\n", " - [ ] **D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory?\n", " - [ ] **D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)?\n", " - [ ] **D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics?\n", " - [ ] **D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed?\n", " - [ ] **D.5 Communicate bias**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood?\n", "\n", "## E. Deployment\n", " - [ ] **E.1 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)?\n", " - [ ] **E.2 Roll back**: Is there a way to turn off or roll back the model in production if necessary?\n", " - [ ] **E.3 Concept drift**: Do we test and monitor for concept drift to ensure the model remains fair over time?\n", " - [ ] **E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed?\n", "\n", "*Data Science Ethics Checklist generated with [deon](http://deon.drivendata.org).*\n"]}]} \ No newline at end of file diff --git a/examples/ethics.md b/examples/ethics.md index e65e08b..4227504 100644 --- a/examples/ethics.md +++ b/examples/ethics.md @@ -6,6 +6,7 @@ - [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? - [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? - [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? + - [ ] **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? ## B. Data Storage - [ ] **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? diff --git a/examples/ethics.rst b/examples/ethics.rst index 5c6731a..df92eec 100644 --- a/examples/ethics.rst +++ b/examples/ethics.rst @@ -10,6 +10,7 @@ A. Data Collection * [ ] **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? * [ ] **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? * [ ] **A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? +* [ ] **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? B. Data Storage --------- diff --git a/examples/ethics.txt b/examples/ethics.txt index a7ad6bc..7d4344e 100644 --- a/examples/ethics.txt +++ b/examples/ethics.txt @@ -4,6 +4,7 @@ A. Data Collection * A.1 Informed consent: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? * A.2 Collection bias: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? * A.3 Limit PII exposure: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? +* A.4 Downstream bias mitigation: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? B. Data Storage * B.1 Data security: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)?