From aacc6eb0173902dc24694c02b80dcc76c35e8eee Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Mon, 10 Oct 2022 18:44:47 -0700 Subject: [PATCH 1/6] Create user test resources and test 1 outlines --- .../0-support/outreach-email-template.md | 34 ++++ user-tests/1-navigation/results.md | 27 ++++ user-tests/1-navigation/test-script.md | 151 ++++++++++++++++++ user-tests/README.md | 19 +++ 4 files changed, 231 insertions(+) create mode 100644 user-tests/0-support/outreach-email-template.md create mode 100644 user-tests/1-navigation/results.md create mode 100644 user-tests/1-navigation/test-script.md create mode 100644 user-tests/README.md diff --git a/user-tests/0-support/outreach-email-template.md b/user-tests/0-support/outreach-email-template.md new file mode 100644 index 00000000..a74d0d4f --- /dev/null +++ b/user-tests/0-support/outreach-email-template.md @@ -0,0 +1,34 @@ +# User testing outreach email templates + +Subject line: Paid Notebook Accessibility Feedback Session + +## Version 2 + +Hi {{name}}, + +I’m {{sender}}, and I’m reaching out on behalf of the Space Telescope Science Institute and the Project Jupyter community. We’d like to invite you to join us as a paid participant in our upcoming user tests to learn more about Jupyter notebooks and their accessibility needs in {{month}}. We got your contact information from {{referral}}. + +Our team has been working on improving the inclusivity of our large collection of notebooks and the notebooks Space Telescope scientists will be making in the future. One of our main goals is to evaluate how easy it is to navigate and understand our public non-editable notebooks, and we want your feedback. During the one-hour feedback sessions we’d be asking you to complete some tasks while navigating a preset notebook. We can offer $100 per hour-long session. If you are interested, let me know and I will follow up with scheduling information. + +Feel free to reach out if you have any other questions. If you are interested in joining us and can’t make that timeline, please let me know. We are planning multiple rounds of user testing and may be able to schedule one that works for you. + +Best, +{{sender}} + +## Version 2 + +Hi {{name}}, + +This is {{sender}} from {{affiliation}}. It’s been awhile since we’ve spoken, I hope you’re doing well! + +Are you interested in being a paid participant in upcoming user tests about Jupyter notebooks and their accessibility. We can offer $100 per hour-long session. We are aiming to schedule the first round of tests roughly between {{date 1}} and {{date 2}}. + +If you are interested, please pick a time using this calendy link: + +If you have trouble with calendy, here’s a link to some accessibility help: +or feel free to respond and tell us what date/times work best for you and we can schedule that way. + +Please email me with any other questions. If you are interested in joining us and can’t make that timeline, let me know—we are planning multiple rounds of user testing and may be able to schedule one that works for you. + +Best, +{{sender}} \ No newline at end of file diff --git a/user-tests/1-navigation/results.md b/user-tests/1-navigation/results.md new file mode 100644 index 00000000..198a89ca --- /dev/null +++ b/user-tests/1-navigation/results.md @@ -0,0 +1,27 @@ +# Results: Structure & Navigation in Rendered Notebooks + +These results are from user interviews conducted in August 2022 with [the navigation test script](test-script.md) on [the STScI tutorial sample notebook](). + +## What we tested + +Operating systems: +Browsers: +Assistive tech: +[Notebook (hosted at via GitHub pages)] () +Sample size: +Method: + + +## How users navigated + +### By headings + +### By preset keys + +### By zooming and skimming + +### With `find` controls + +## Major takeaways + +## What we would do differently next time \ No newline at end of file diff --git a/user-tests/1-navigation/test-script.md b/user-tests/1-navigation/test-script.md new file mode 100644 index 00000000..aa434eea --- /dev/null +++ b/user-tests/1-navigation/test-script.md @@ -0,0 +1,151 @@ +# Structure & Navigation in Rendered Notebooks + +## Introduction + +Hi, I’m ________ I‘ll be running this meeting today. This is ________, they are here to take notes. + +Thank you for taking the time to participate in this study. Before we begin, I’ll give you a brief overview of how this will work. Throughout the hour, I will be reading from a script. + +This is completely voluntary. Please let me know at any point if you wish to stop participating. + +There’s a few things I want to remind you about before we start. + +- This session will be recorded, and we will be taking notes +- Recordings and notes will be saved online in a private folder that only our team members can access. Your private info is only available to the people working directly on this project +- Some anonymous information will be publicly available on our GitHub repo. https://github.com/Iota-School/notebooks-for-all + +During this study, we will be exploring a read only webpage that has been exported from a notebook. You can’t edit it like a traditional notebook. I will be giving you a broad task to complete. I will set context for each task such as why you might be doing it and what you hope to achieve. Then I will ask some follow up questions. + +As you work through the task, please think out loud. Speak your thoughts as often as you can. Please tell us if something unexpected happens, if something works well, if you think something that could be improved, or if you’re confused. We do not expect all the tasks to be easy to complete with assistive tech and we invite you to complain! + +It’s important to know that we are not testing you, we are testing how Jupyter Notebooks behave with assistive tech. There are no correct or incorrect answers. You cannot do anything wrong. We are expecting them to break. + +If at any point you have questions, please don’t hesitate to ask. Before we start, do you have any questions? + +## Participant Introduction + +First, I want to ask a few things about you and your prior experience with Jupyter notebooks. + +- What is the best way to send you links during this discussion? (in chat, spoken, email?) +- Would us sending messages in the chat to timestamp certain sections be disruptive for you? (Perhaps if you’re using a screen reader) +- What operating system and browser are you using today? (We use this info to reproduce errors and support you if needed.) +- What assistive tech are you using during this session? +- How have you interacted with notebooks in the past? Tell me about the most common way you interact with a notebook? + +## Notebook Tasks + +Now that we know a little bit about you, I’m going to ask you to complete some tasks using an uneditable form of a Jupyter notebook hosted by Space Telescope Science Institute (STScI). Once again, we are asking for honest feedback on the software—please complain. + +First, can you please share your screen. When sharing, can you set it so that you either share the sound or share the text with a “speech viewer” ex: In NVDA you can set it to hold the NVDA button (insert or caps) and hit N and then you get options, go down to tools, and select speech viewer. + +### Task 1 + +Today we want to explore a platform that provides STScI notebooks in a non-editable form with all cell outputs shown. That means all the code has been run, we’re just exploring the document without it changing, like reading a book. + +Now I’d like you to open the style guide template notebook. I’ll tell you how to do this part. + +1. Open your web browser of choice +2. Paste this link: https://eteq.github.io/notebooks-for-all/14jun22_stsci_example_notebook.html +https://bit.ly/3pgkFSl + +**Can you tell me what the title of this notebook is?** +- Tutorial Title [Y][N] + +**How easy or difficult was it for you to open the notebook?** + +**If you were to magically make opening this notebook easier, what would you change?** + +### Task 2 + +When you first open a notebook, there’s some information to help you orient yourself. + +**Can you tell me who the author of this notebook is?** +- Jessie Blogs, Archive Scientist [Y][N] + +**What other [descriptive] information would help you orient yourself in the notebook?** + +**Are you satisfied with the organization of this information?** + +**Do you feel that you are missing any information?** +- **What kinds of feedback do you search for to feel confident that you have all the information?** + +### Task 3 + +Next, we’re going to explore the File Information Section. + +**Show me how you would navigate to File Information.** +1. Were they able to navigate to File Information [Y][N] +2. Notes on how they did it: + +**There is a table in this section. Can you tell me the headers of the first two columns on the table?** +1. obsID [Y][N] +2. Obs_collection [Y][N] + +**How easy or difficult was it for you to navigate to and through the table?** + +**If you were able to magically make accessing this section easier, what would be the same? What would be different?** + +### Task 4 + +There’s more information at the bottom of the notebook, so let’s go there next. + +**I’d like you to navigate to the final cell of the notebook.** +1. Were they able to navigate to the footer and/or citations? [Y][N] +2. Notes on how they did it: + +**Can you tell me what information the footer (the section at the very bottom) includes?** +- Citations [Y][N] +- Link to top of the page and stsci logo [Y][N] + +**How easy or difficult was it for you to navigate and read the footer section?** + +**As a reader, what information do you expect to find in the notebook footer?** + +### Task 5 + +Now that you’ve explored the notebook, I’d like you to return to the top of the page + +**Please return to the top cell.** +1. Were they able to navigate back to the top? [Y][N] +2. Notes on how they did it: + +**Please summarize the general topics this notebook covers for me as if I hadn’t read it.** + +**Do you feel that you are able to access all the information in this notebook? What makes you feel that way?** + +## Follow Up Questions + +Now that you’ve explored the notebook, I’d like to ask you to reflect on how that experience went and any other feedback you might have. + +**Please complain – what was frustrating about navigating the Jupyter notebook with your assistive tech?** + +**Did the notebook provide enough (or the correct) information to help you know what to do and what decisions to make?** + +**Did the the notebook give you enough feedback to feel confident that you were completing the tasks successfully?** + +**Are you satisfied with the work-flow of moving through the notebook? (Navigation, number of steps, etc.) If so, what worked well? If not, what do you wish it did instead?** + +**Is there any information, options, or capability that is missing?** + +**Can you see yourself using Jupyter Notebooks in a non-editable form? (Y) (N) Why or why not?** + +**Do you have any other impressions or feedback that you would like to share?** + +**Do you have any questions for me?** + +Thank you so much for participating! + +--- + +## Interview Debrief + +After each ethnographic interview you complete, take a few minutes to perform an interview +debrief while the session is fresh in your mind. This ensures that key learnings and observations are not lost in the scramble of many interviews or long timelines. + +**What are our action items based on this feedback?** + +**Any more details on issues we already discovered?** + +**Are there any new questions I should explore in a further script?** + +**What are some key quotes that I heard?** \ No newline at end of file diff --git a/user-tests/README.md b/user-tests/README.md new file mode 100644 index 00000000..39941349 --- /dev/null +++ b/user-tests/README.md @@ -0,0 +1,19 @@ +# User testing information + +## Test support + +Resources not tied to any single test round. + +- [Outreach and recruitment email template](0-support/outreach-email-template.md) + +## Test 1: Navigation + +August 2022. How do users navigate and access different levels of structure within HTML outputs of Jupyter notebooks? + +- [Test script](1-navigation/test-script.md) +- [Test notebook () +- [Test results](1-navigation/results.md) + +## Test 2: Content types + +Work in progress. 2022. \ No newline at end of file From 7973ce54a7adc38e5a41bdccbc07909596556ee1 Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Fri, 21 Oct 2022 10:41:50 -0700 Subject: [PATCH 2/6] Add test 1 testing notebook --- .../1-navigation/stsci_example_notebook.ipynb | 337 ++++++++++++++++++ 1 file changed, 337 insertions(+) create mode 100644 user-tests/1-navigation/stsci_example_notebook.ipynb diff --git a/user-tests/1-navigation/stsci_example_notebook.ipynb b/user-tests/1-navigation/stsci_example_notebook.ipynb new file mode 100644 index 00000000..ea6af087 --- /dev/null +++ b/user-tests/1-navigation/stsci_example_notebook.ipynb @@ -0,0 +1,337 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "\n", + "# Tutorial Title" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ + "***" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Learning Goals\n", + "Write three to five learning goals. A learning goal should describe what a reader should know or be able to do by the end of the tutorial that they didn't know or couldn't do before.\n", + "\n", + "```\n", + "By the end of this tutorial, you will:\n", + "\n", + "- Understand how to use aperture photometry to turn a series of two-dimensional\n", + " images into a one-dimensional time series.\n", + "- Be able to determine the most useful aperture for photometry on a *Kepler/K2*\n", + " target.\n", + "- Create your own light curve for a single quarter/campaign of *Kepler/K2* data.\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## Introduction\n", + "Write a short introduction explaining the purpose of the tutorial. Define any terms or common acronyms that your audience may not know. If you're using some kind of domain-specific astronomical symbol or unusual mathematical concept, make sure you define it (for example, in its mathematical form) and link to any definitions (from literature, Wikipedia, etc.).\n", + "\n", + "If there are background materials or resources that may be useful to the reader to provide additional context, you may link to it here. If your tutorial is a continuation from another tutorial, or there are other tutorials that would be useful for the reader to read before or after your tutorial, mention that here as well." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## Imports\n", + "Describe the libraries we're using here. If there's something unusual, explain what the library is, and why we need it.\n", + "- *numpy* to handle array functions\n", + "- *astropy.io fits* for accessing FITS files\n", + "- *astropy.table Table* for creating tidy tables of the data\n", + "- *matplotlib.pyplot* for plotting data" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "import numpy as np\n", + "from astropy.io import fits\n", + "from astropy.table import Table\n", + "import matplotlib.pyplot as plt\n", + "from astroquery.mast import Mast\n", + "from astroquery.mast import Observations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "***" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Main Content\n", + "\n", + "The main content of your tutorial should be subdivided into numbered sections with useful, descriptive headings that make sense based on the content. Break sections up with standard Markdown syntax headings:\n", + "\n", + "```\n", + "## Section 1\n", + "\n", + "Intro to Section 1\n", + "\n", + "### Subsection 1a\n", + "\n", + "More detailed info about Section 1\n", + "\n", + "## Section 2\n", + "\n", + "A complete thought that's as important as Section 1 but doesn't need subsections.\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "### Loading Data\n", + "\n", + "Loading data and file information should appear within your main content, at the same time the data is going to be used, if possible. These elements of your tutorial can be their own sections within the main content, but avoid generic or vague headings like “Loading Data” and instead use descriptive headings pertinent to the content of the tutorial and the actual data being downloaded or files being used.\n", + "\n", + "If the user needs to download data to run the tutorial properly, where possible, use [Astroquery](https://astroquery.readthedocs.io/en/latest/) (or similar) to retrieve files. If this is not possible, see the [data guide](https://github.com/spacetelescope/style-guides/blob/master/guides/where-to-put-your-data.md) for other options." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "For example, if we wanted to query for data from MAST for Kepler we might do something like:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "keplerObs = Observations.query_criteria(target_name='kplr008957091', obs_collection='Kepler')\n", + "keplerProds = Observations.get_product_list(keplerObs[0])\n", + "yourProd = Observations.filter_products(keplerProds,extension='kplr008957091-2012277125453_lpd-targ.fits.gz',\n", + " mrp_only=False)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### File Information\n", + "\n", + "Explain pertinent details about the file you've just downloaded. For example, if working with Kepler light curves, explain what's in the different file extensions:\n", + "\n", + "```\n", + "- No. 0 (Primary): This HDU contains metadata related to the entire file.\n", + "- No. 1 (Light curve): This HDU contains a binary table that holds data like\n", + " flux measurements and times. We will extract information from here when we\n", + " define the parameters for the light curve plot.\n", + "- No. 2 (Aperture): This HDU contains the image extension with data collected\n", + " from the aperture. We will also use this to display a bitmask plot that\n", + " visually represents the optimal aperture used to create the SAP_FLUX column in\n", + " HDU1.\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Where possible (if the code supports it), use code examples that visually display the data in the tutorial. For example, if you are showing an object such as a Table, display a preview of the table:" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "Table masked=True length=5\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
obsIDobs_collectiondataproduct_typeobs_iddescriptiontypedataURIproductTypeproductGroupDescriptionproductSubGroupDescriptionproductDocumentationURLprojectprvversionproposal_idproductFilenamesizeparent_obsid
str10str6str10str36str59str1str110str7str28str1str1str6str1str7str44int64str10
9000159893Keplertimeserieskplr008957091_lc_Q000000000011111111Preview-FullCmast:Kepler/url/missions/kepler/previews/0089/008957091/kplr008957091-2013098041711_llc_bw_large.pngPREVIEW------Kepler--GO30032kplr008957091-2013098041711_llc_bw_large.png296539000159893
9000159893Keplertimeserieskplr008957091_lc_Q000000000011111111Lightcurve Long Cadence (CLC) - Q10Cmast:Kepler/url/missions/kepler/lightcurves/0089/008957091/kplr008957091-2011271113734_llc.fitsSCIENCE------Kepler--GO30032kplr008957091-2011271113734_llc.fits4867209000159893
9000159893Keplertimeserieskplr008957091_lc_Q000000000011111111Lightcurve Long Cadence (CLC) - Q11Cmast:Kepler/url/missions/kepler/lightcurves/0089/008957091/kplr008957091-2012004120508_llc.fitsSCIENCE------Kepler--GO30032kplr008957091-2012004120508_llc.fits5068809000159893
9000159893Keplertimeserieskplr008957091_lc_Q000000000011111111Lightcurve Long Cadence (CLC) - Q12Cmast:Kepler/url/missions/kepler/lightcurves/0089/008957091/kplr008957091-2012088054726_llc.fitsSCIENCE------Kepler--GO30032kplr008957091-2012088054726_llc.fits4348809000159893
9000159893Keplertimeserieskplr008957091_lc_Q000000000011111111Lightcurve Long Cadence (CLC) - Q13Cmast:Kepler/url/missions/kepler/lightcurves/0089/008957091/kplr008957091-2012179063303_llc.fitsSCIENCE------Kepler--GO30032kplr008957091-2012179063303_llc.fits4723209000159893
" + ], + "text/plain": [ + "\n", + " obsID obs_collection dataproduct_type ... size parent_obsid\n", + " str10 str6 str10 ... int64 str10 \n", + "---------- -------------- ---------------- ... ------ ------------\n", + "9000159893 Kepler timeseries ... 29653 9000159893\n", + "9000159893 Kepler timeseries ... 486720 9000159893\n", + "9000159893 Kepler timeseries ... 506880 9000159893\n", + "9000159893 Kepler timeseries ... 434880 9000159893\n", + "9000159893 Kepler timeseries ... 472320 9000159893" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "yourProd[0:5]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download the products\n", + "Observations.download_products(yourProd, mrp_only=False, cache=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Exercises\n", + "Exercises are optional, but encouraged. Exercises can be woven into the main content of your tutorial, or appear in their own section toward the end of the tutorial. Final exercises can be more challenging, similar to homework problems. They can be minimal or take as long as 30 minutes to an hour to complete. If you do have one or more exercises in your tutorial, be sure to leave a blank code cell underneath each to show the reader that they're meant to try out their new skill right there. You may also want to include a \"solutions\" notebook next to your main notebook for the reader to check their work after they have finished their attempt." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Additional Resources\n", + "\n", + "This section is optional. Try to weave resource links into the main content of your tutorial so that they are falling in line with the context of your writing. For resources that do not fit cleanly into your narrative, you may include an additional resources section at the end of your tutorial. Usually a list of links using Markdown bullet list plus link format is appropriate:\n", + "\n", + "- [MAST API](https://mast.stsci.edu/api/v0/index.html)\n", + "- [Kepler Archive Page (MAST)](https://archive.stsci.edu/kepler/)\n", + "- [Kepler Archive Manual](https://archive.stsci.edu/kepler/manuals/archive_manual.pdf)\n", + "- [Exo.MAST website](https://exo.mast.stsci.edu/)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, + "source": [ + "## About this Notebook\n", + "Let the world know who the author of this great tutorial is! If possible and appropriate, include a contact email address for users who might need support (for example, `archive@stsci.edu`). You can also optionally include keywords, your funding source, or a last update date in this section.\n", + "\n", + "**Author:** Jessie Blogs, Archive Scientist. \n", + "**Updated On:** YYYY-MM-DD" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Citations\n", + "Provide your reader with guidelines on how to cite open source software and other resources in their own published work.\n", + "\n", + "```\n", + "If you use `astropy` or `lightkurve` for published research, please cite the\n", + "authors. Follow these links for more information about citing `astropy` and\n", + "`lightkurve`:\n", + "\n", + "* [Citing `astropy`](https://www.astropy.org/acknowledging.html)\n", + "* [Citing `lightkurve`](http://docs.lightkurve.org/about/citing.html)\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "***" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[Top of Page](#top)\n", + "\"Space " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.8" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 86a23dc1f00bf2f1461c145617d28d46537dd075 Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Fri, 21 Oct 2022 11:55:54 -0700 Subject: [PATCH 3/6] Add more 0-support user testing files --- .../0-support/generic-recording-consent.md | 22 +++++++++++++ .../0-support/possible-test-notebooks.md | 29 +++++++++++++++++ .../0-support/user-testing-resources.md | 32 +++++++++++++++++++ 3 files changed, 83 insertions(+) create mode 100644 user-tests/0-support/generic-recording-consent.md create mode 100644 user-tests/0-support/possible-test-notebooks.md create mode 100644 user-tests/0-support/user-testing-resources.md diff --git a/user-tests/0-support/generic-recording-consent.md b/user-tests/0-support/generic-recording-consent.md new file mode 100644 index 00000000..aab686f0 --- /dev/null +++ b/user-tests/0-support/generic-recording-consent.md @@ -0,0 +1,22 @@ +# Recording consent form + +Thank you for participating in our usability research on Space Telescope Science Institute public notebooks. + +We will be recording your session to allow Space Telescope Science Institute and external grant team members who are unable to be here today to observe your session and benefit from your comments. Please read the statement below and sign where indicated. + +- I understand that my usability test session will be recorded. +- I understand that my usability test session recording will be stored on private Space Telescope Science Institute drives. +- I understand that anonymized summaries of information learned from my session may be publicly shared on the grant repository. +- I grant Space Telescope Science Institute permission to use this recording for internal use only, for the purpose of improving the designs being tested. + +Signature: + +Print name: + +Date: + +## Notes + +Adapted from [Steve Krug's Recording consent form](https://sensible.com/download-files/). + +This document was further adapted by the institutions handling all paperwork for participants to meet their own needs. This example is here as a reference for how we collaborated with other teams. \ No newline at end of file diff --git a/user-tests/0-support/possible-test-notebooks.md b/user-tests/0-support/possible-test-notebooks.md new file mode 100644 index 00000000..a6e3ab51 --- /dev/null +++ b/user-tests/0-support/possible-test-notebooks.md @@ -0,0 +1,29 @@ +# Possible test notebooks + +A list of existing notebooks that could be used during our rounds of user tests. They may also provide inspiration to custom test notebooks we may make. + +| Notebook or collection name | Test candidate? | Used in test? (date) | +| --- | --- | --- | +| [JupyterLab examples test.ipynb](https://github.com/jupyterlab/jupyterlab/blob/master/examples/notebook/test.ipynb) | | | +| [nbconvert examples](https://github.com/jupyter/nbconvert-examples) | | | +| [Lorenz notebook](https://github.com/jupyterlab/jupyterlab-demo/blob/master/notebooks/Lorenz.ipynb) | Yes | | +| [JupyterLab benchmarks test noteboks](https://github.com/jupyterlab/benchmarks/tree/master/examples/from-benchmarks) | | | +| [Cosmic Origins Spectograph notebooks](https://www.stsci.edu/hst/instrumentation/cos/documentation/notebooks) | | | +| [STScI notebook style guide template](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb) | | Yes (Test 1, August 2022) | +| [STScI notebook style guide](https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md) | Yes | | +| [STScI JDAT notebooks](https://github.com/spacetelescope/jdat_notebooks/tree/main/notebooks) [(Rendered here)](https://spacetelescope.github.io/jdat_notebooks/intro.html) | | | +| [STScI JWebbinar notebooks](https://github.com/spacetelescope/jwebbinar_prep) | Maybe (needs content review) | | + + +The following are axes we considered choosing notebooks based on. If they end up being a deciding factor we take note of, then they will be added to the above table. + +- length (in cells) +- has MD cells +- has code cells +- has text-only outputs +- has non-text outputs +- notes about outputs +- uses STScI data +- follows STScI notebook guidelines +- renders successfully (in viewer tool) +- Test hosts can explain it \ No newline at end of file diff --git a/user-tests/0-support/user-testing-resources.md b/user-tests/0-support/user-testing-resources.md new file mode 100644 index 00000000..36e615c3 --- /dev/null +++ b/user-tests/0-support/user-testing-resources.md @@ -0,0 +1,32 @@ +# User testing resources for notebooks + +Astronomy Notebooks for All - STScI + +## General user testing resources + +[Steve Krug usability testing guides](https://sensible.com/download-files/) +- Usability test script +- Recording consent form +- Usability Testing Checklists +- “Things a therapist would say” + +[Remote Usability Testing: Study Guide - Nielsen Norman Group](https://www.nngroup.com/articles/remote-usability-testing-study-guide/) + +[How you can perform cheap unmoderated usability testing using Zoom - UX Collective](https://uxdesign.cc/how-you-can-perform-cheap-unmoderated-usability-testing-using-zoom-9dbef023570e) + +## Test-specific resources + +- [jupyter/nbformat](https://github.com/jupyter/nbformat) +- [jupyter/nbconvert](https://github.com/jupyter/nbconvert) +- [jupyter/nbviewer](https://github.com/jupyter/nbviewer) + - What is the STScI equivalent for this? +- [jupyter/nbconvert-examples](https://github.com/jupyter/nbconvert-examples) +- [JupyterLab UI overview - Adobe Experience League](https://experienceleague.adobe.com/docs/experience-platform/data-science-workspace/jupyterlab/overview.html?lang=en#code-cells) +- [Some jupyterlab/jupyterlab user stories](https://github.com/jupyterlab/jupyterlab/tree/master/design) +- [The Lorenz Notebook](https://github.com/jupyterlab/jupyterlab-demo/blob/master/notebooks/Lorenz.ipynb) + +## Accessibility + +- [Browsing with assistive technology videos](https://tetralogical.com/blog/2021/12/24/browsing-with-assistive-technology-videos/) +- [Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs](https://dl.acm.org/doi/fullHtml/10.1145/3491102.3501843) +- [CAIR Lab - RIT](http://cair.rit.edu/projects.html) From 10edf9e7547dcde98b1bde91ce7e5a22a965a8f3 Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Fri, 21 Oct 2022 12:05:03 -0700 Subject: [PATCH 4/6] Update user testing readme --- user-tests/README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/user-tests/README.md b/user-tests/README.md index 39941349..ae73e4a2 100644 --- a/user-tests/README.md +++ b/user-tests/README.md @@ -1,18 +1,23 @@ # User testing information +User testing on this project was led by [Jenn Kotler](https://github.com/Jenneh) and [Isabela Presedo-Floyd](https://github.com/isabela-pf). Resources in this directory were created collaboratively. + ## Test support Resources not tied to any single test round. +- [Generic recording consent example](0-support/generic-recording-consent.md) - [Outreach and recruitment email template](0-support/outreach-email-template.md) +- [Possible test notebooks](0-support/possible-test-notebooks.md) +- [User testing resources](0-support/user-testing-resources.md) ## Test 1: Navigation August 2022. How do users navigate and access different levels of structure within HTML outputs of Jupyter notebooks? -- [Test script](1-navigation/test-script.md) -- [Test notebook () -- [Test results](1-navigation/results.md) +- [Test 1 script](1-navigation/test-script.md) +- [Test 1 notebook](1-navigation/stsci_example_notebook.ipynb). Originally sourced from the [STScI notebook style guide template](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb). +- [Test 1 results](1-navigation/results.md) ## Test 2: Content types From 4a88aa4077394951d9e9f54107473ebb8a3334a9 Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Fri, 28 Oct 2022 16:31:23 -0700 Subject: [PATCH 5/6] Add Test 1 results and readme fixes --- user-tests/1-navigation/results.md | 135 +++++++++++++++++++++++++++-- user-tests/README.md | 2 +- 2 files changed, 127 insertions(+), 10 deletions(-) diff --git a/user-tests/1-navigation/results.md b/user-tests/1-navigation/results.md index 198a89ca..ee8329ae 100644 --- a/user-tests/1-navigation/results.md +++ b/user-tests/1-navigation/results.md @@ -1,27 +1,144 @@ # Results: Structure & Navigation in Rendered Notebooks -These results are from user interviews conducted in August 2022 with [the navigation test script](test-script.md) on [the STScI tutorial sample notebook](). +These results are from user interviews conducted in August 2022 with [the navigation test script](test-script.md) on [the STScI tutorial sample notebook](stsci_example_notebook.ipynb). ## What we tested -Operating systems: -Browsers: -Assistive tech: -[Notebook (hosted at via GitHub pages)] () -Sample size: -Method: +**Operating systems:** Mac OS Monterey, Windows 10 +**Browsers:** Chrome, Firefox, Safari + +**Assistive tech:** JAWS (screen reader), NVDA (screen reader), VoiceOver (screen reader), Mantis (braille reader), Mac OS Zoom (built-in screen magnifier), built-in browser zoom controls, built-in large cursor and pointer settings + +**Interface:** Browser and [notebook](stsci_example_notebook.ipynb) in HTML form via nbconvert(hosted via GitHub pages) + +**Sample size:** 6 participants + +**Method:** Combination of qualitative usability testing and user interviews ## How users navigated +The following sections describe the ways participants chose to navigate through a single notebook (one browser page). These methods are likely not notebook-specific, but they have only been noted in that context. They are not listed in any particular order. + ### By headings +This navigation method appeared exclusively with participants using screen readers. + +1. Have a screen reader active. +2. Using that [screen reader's Headings shortcut](https://dequeuniversity.com/screenreaders/), open the Headings list. This moves focus outside the notebook and browser. +3. Review and select heading to jump to. +4. Keyboard focus and page scroll jumps to that heading. + +Because nbconvert for notebook to HTML properly captures Markdown cell headings as HTML headings, heading organization is technically available regardless of assistive tech or other settings. At the time of writing, though, there is no way for someone not using a screen reader to interact with the headings in the same way. + +Non-screen reader using participants frequently requested a table of contents to jump to major content areas in the same way that we saw screen reader using participants do. Only one screen reader using participants made the request for a table of contents, and even then it was mentioned as a personal preference over headings and not as a blocking issue. No screen reader using participants expressed trouble jumping between content areas, and they all used heading navigation at some point. + ### By preset keys +This navigation method is not reliant on any assitive tech or setting. Because keys are configurable, keyboard language/region can be configured, and physical keyboards can be different, there is a lot of variation on how users might have preset keys to navigate. During our sessions, we saw + +1. Have a keyboard with `end of page` and/or `top of page` keys configured. +2. When participants knew or expected the goal would be near one end of a page, or simply closer to shorten navigation, they would use the relevant key to jump to a different area. +3. Keyboard focus and page scroll jumps to that area. +3. If needed, participants navigate the rest of the way to goal using an additional navigation method. + +Navigating by preset page navigation keys rarely took users exactly where they wanted to go on its own; it was used in combination with another method in any case where the goal was not singularly "go to the bottom of the document" (as in Task 4) or "go to the top of the document" (as in Task 5). + +This was used by participants who used browser zoom, or in one case a screen reader. Participants expect this to work at a browser level, so it is not tied specifically to the notebook. + ### By zooming and skimming +This navigation method appeared exclusively with participants using browser zoom/magnification. They are similar, but have been broken into different steps to preserve the nuances of each. + +For a magnifier: + +1. Magnify the left side of the notebook. Amount needed may vary per participant, but the magnification stays at a consistent amount at first. Content does not reflow. +2. Scroll along the left side of the notebook. What is visible depends on participants' magnifier settings; it could be that only a section of the browser window is visible on screen, that a single rectangular area is magnified while the rest of the window remains visible at 100% in the background, or things in between. +3. Find an area that appears to be realted to the goal. Stop scrolling and adjust magnification as desired. +4. Read the content while moving magnifier to the right to complete a line. +5. At the end of each line, participants would move back to the left to start the next line unless they either found their goal or decided they would not complete their goal in the area. +6. When they need to navigate again, participants would start at step one again. + +For browser zoom: + +1. Zoom browser to about 150–250%. Content should reflow. +2. Scroll along the left side of the notebook. Only the start and far right of each cell and output are visible in the window. +3. Find an area that appears to be realted to the goal. Stop scrolling with it roughly centered on the page. +4. Increase browser zoom to 300–500%. Rescroll to desired point if necessary; the notebook did not hold scroll poisitons in the center so it was necessary for our tests, but participants noted this was not always their expectation. +5. Read the content scrolling to the right to complete a line. +6. At the end of each line, participants would scroll back to the left to start the next line unless they either found their goal or decided they would not complete their goal in the area. +7. When they need to navigate again, participants would lower browser zoom to start at step one again. + +When navigating with this method, participants emphasized how important the far left content was to them, whether zooming in a way that caused the content to reflow or not. Working with notebooks in English means that we did not have the opportunity to test where participants would go to skim content in a right-to-left language or in non-document formatting, but it is safe to conclude that skimming and reading content in full are two different modes for people who navigate this way. + +Participants using this method also frequently brought up the utility of table of contents to hasten their navigation and lower the physical demands of reading as a result. + ### With `find` controls -## Major takeaways +This navigation method appeared exclusively with participants using screen readers. It only came up once throughout sessions and test hosts were only able to get limited information on it. it may not be entirely accurate. Reference this section with that in mind. + +1. From anywhere on the page or in the browser, open the screen reader's built in `find` type of controls. +2. Input filtering criteria, review and select an option. +3. Keyboard focus and page scroll jump directly to the selected option. + +This can be done with features like [NVDA's Search for a word or a phrase](https://dequeuniversity.com/screenreaders/nvda-keyboard-shortcuts) or [VoiceOver's rotor](https://dequeuniversity.com/screenreaders/voiceover-keyboard-shortcuts#vo-mac-the-rotor). It is similar to a browser's `find` features in that it filters the application content and allows users to navigate based on that. + +This method was used by participants most frequently when other navigation and skimming methods failed to help them complete a task. For example, it was common for participants to use several navigation methods when completing Task 2 since many expressed the author's name was not where they expected it to be (it was at the bottom of the document rather than the top). + +There was one instance where a screen reader participant used this navigation method right away, and that was because they had found the content needed to complete the task when skimming in a prior task and did not remember what heading it was under. They noted this was feasible because they remembered the content and knew what to filter for. + +### By tabbing through interactive areas + +This navigation method is not reliant on any assitive tech or setting. While this technique is possible to anyone regardless of OS, browser, or assistive tech, the only participants we saw using this were screen reader users. + +1. Use the `tab` key. The next interactive element in the focus order (ie. a link, a button, so on) will have keyboard focus. +2. Use the `tab` key repeatedly until reaching the desired area. Focus order does loop, so one may jump from the bottom of the notebook to the top of the browser, for example. + +This navigation method was used infrequently, and it was used in combination with other navigation methods. Most frequently, tabbing was used as a fine-grain navigation once participants were in the general region they wanted to be. For example, a participant used a screen reader to jump to a content heading and then skip through cells via tabbing to skim for an area they were searching for. + +## Common feedback + +This is a list of the feedback that was most frequently or emphatically given. It is in no paritcular order. + +- Requests for a table of contents. This was particularly important to participants not using screen readers. +- Notebooks need to be edited too, not just read. Participants that gave this feedback were aware of the scope of these tests and they wanted to emphasize that accessibility fixes also needed to happen for editable states. +- Notebook cells were of varying importance. How people want to understand and navigate the notebooks seemed to depend most on their expectations. Some participants talked in terms of cells or noted that they couldn't find non-visual cell sections (this was more common of participants who author notebooks, but not exclusive to them). Some participants talked in terms of content headings from the notebook cells. Some didn't mention either. This test did not allow us time to dive into why different participants had different mental models, but it was worth noting. + +## Issues + +Bugs, issues, and other specific feedback or discussions from this round of tests can be found throughout the repository in issues. Listed below, they are + +- [Explore landmark options in rendered notebook](https://github.com/Iota-School/notebooks-for-all/issues/5) +- [Automatically add link to rendered notebook source](https://github.com/Iota-School/notebooks-for-all/issues/8) +- [Add a table of contents to rendered notebooks](https://github.com/Iota-School/notebooks-for-all/issues/9) +- [Page title and notebook title do not match](https://github.com/Iota-School/notebooks-for-all/issues/10) +- ["Top of Page" link in template footer bug](https://github.com/Iota-School/notebooks-for-all/issues/11) +- [Explore options for minimizing content on left side of rendered notebook](https://github.com/Iota-School/notebooks-for-all/issues/12) +- [Review/explore keyboard shortcuts in rendered notebooks](https://github.com/Iota-School/notebooks-for-all/issues/13) +- [Explore ARIA options in rendered notebook](https://github.com/Iota-School/notebooks-for-all/issues/14) +- [Code cells cut off content at high zoom in rendered notebook](https://github.com/Iota-School/notebooks-for-all/issues/15) +- [Vertical scroll jumping when adjusting browser zoom in rendered notebooks](https://github.com/Iota-School/notebooks-for-all/issues/17) +- [Table Reading with screenreaders](https://github.com/Iota-School/notebooks-for-all/issues/18) +- [Navigate to cells using keyboard commands](https://github.com/Iota-School/notebooks-for-all/issues/19) +- [Share Cell Content with screenreaders](https://github.com/Iota-School/notebooks-for-all/issues/20) +- [Move Metadata to the top](https://github.com/Iota-School/notebooks-for-all/issues/21) +- [Notebook Tutorial Link](https://github.com/Iota-School/notebooks-for-all/issues/22) +- [Markdown should be used only as intended](https://github.com/Iota-School/notebooks-for-all/issues/23) +- [Best Practice for Documenting Table Headers](https://github.com/Iota-School/notebooks-for-all/issues/24) + +## Questions for future tests + +At the end of each session, we noted questions we wanted to further explore. This is the cumulative list. + +- For a screen reader reading a code block in a Markdown cell, it read the content line by line instead of as a whole block. This was different than inline code styling or a code cell. Some participants expressed confuison. This would be good to text for mixed content types in a single cell, or even in the same line, perhaps. +- How easy is it to navigate in/out, and between certain content types (ie. tables were mentioned as “if you’re in a table its a pain to jump to another part of the page and then back to the same part of the table”)? +- Should we explore UX for a table of contents? It is helpful for keeping context, but should be collapsible because of the space it can take up. +- Consider zooming in on content types as a task for future content-type tests. +- Should we let participants read the whole notebook first? Let them give first impressions and see how they decide to read the whole notebook. + +## What we would do differently next time + +Reflecting on these sessions as hosts, for future tests we would like to -## What we would do differently next time \ No newline at end of file +- Have a non-template notebook to work with. Multiple participants spent some of the tasks getting caught up figuring out why the notebook content switched topics often, expressed confusion that the notebook did not follow the narrative it expected when searching for information in multiple tasks, or found it difficult to summarize then notebook when asked in Task 5. +- Considering comparing some solutions side by side. As the first test, it is important to have a sense of the current state of nbconverted HTML notebooks but it doesn't always give us clarity on desired UX. \ No newline at end of file diff --git a/user-tests/README.md b/user-tests/README.md index ae73e4a2..15ae130c 100644 --- a/user-tests/README.md +++ b/user-tests/README.md @@ -16,7 +16,7 @@ Resources not tied to any single test round. August 2022. How do users navigate and access different levels of structure within HTML outputs of Jupyter notebooks? - [Test 1 script](1-navigation/test-script.md) -- [Test 1 notebook](1-navigation/stsci_example_notebook.ipynb). Originally sourced from the [STScI notebook style guide template](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb). +- [Test 1 notebook](1-navigation/stsci_example_notebook.ipynb). Originally sourced from the [STScI notebook style guide template](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb). This is a template notebook for creating STScI tutorials with a mix of instruction and example. - [Test 1 results](1-navigation/results.md) ## Test 2: Content types From 95064a57e7c417fc1ff175bee579f74e0722d840 Mon Sep 17 00:00:00 2001 From: isabela-pf Date: Tue, 8 Nov 2022 12:30:43 -0800 Subject: [PATCH 6/6] Add directory license and readme note --- user-tests/LICENSE | 395 +++++++++++++++++++++++++++++++++++++++++++ user-tests/README.md | 2 + 2 files changed, 397 insertions(+) create mode 100644 user-tests/LICENSE diff --git a/user-tests/LICENSE b/user-tests/LICENSE new file mode 100644 index 00000000..be764a72 --- /dev/null +++ b/user-tests/LICENSE @@ -0,0 +1,395 @@ +Creative Commons Attribution 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution 4.0 International Public License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution 4.0 International Public License ("Public License"). To the +extent this Public License may be interpreted as a contract, You are +granted the Licensed Rights in consideration of Your acceptance of +these terms and conditions, and the Licensor grants You such rights in +consideration of benefits the Licensor receives from making the +Licensed Material available under these terms and conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + d. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + e. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + f. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + g. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + h. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + i. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + j. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + k. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part; and + + b. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + 4. If You Share Adapted Material You produce, the Adapter's + License You apply must not prevent recipients of the Adapted + Material from complying with this Public License. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material; and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + + +======================================================================= + +Creative Commons is not a party to its public licenses. +Notwithstanding, Creative Commons may elect to apply one of its public +licenses to material it publishes and in those instances will be +considered the “Licensor.” The text of the Creative Commons public +licenses is dedicated to the public domain under the CC0 Public Domain +Dedication. Except for the limited purpose of indicating that material +is shared under a Creative Commons public license or as otherwise +permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the public +licenses. + +Creative Commons may be contacted at creativecommons.org. \ No newline at end of file diff --git a/user-tests/README.md b/user-tests/README.md index 15ae130c..b67d532a 100644 --- a/user-tests/README.md +++ b/user-tests/README.md @@ -2,6 +2,8 @@ User testing on this project was led by [Jenn Kotler](https://github.com/Jenneh) and [Isabela Presedo-Floyd](https://github.com/isabela-pf). Resources in this directory were created collaboratively. +Because this directory holds mostly non-code content, it is licensed under a [CC-BY 4.0 license](LICENSE). + ## Test support Resources not tied to any single test round.