T264831_var_CD_data_gathering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variant C/D experiment data gathering\n",
    "\n",
    "The Growth Team ran an experiment with two variants of the Newcomer Homepage starting on 2020-10-19. We're now going to analyse the results of that experiment. The phab task for this analysis is [T264831](https://phabricator.wikimedia.org/T264831). Per the task, we're interested in answering the following questions:\n",
    "\n",
    "* Visits (mobile only): what percent of newcomers visit the full suggested edits module on mobile? This only applies on mobile because mobile users must tap the module preview or go through onboarding before getting to the full module.\n",
    "* Interaction: what percent of newcomers interact with the suggested edits module? Note that we want to only count interactions with the fully-initiated suggested edits module. Interacting with topic selection and difficulty selection during the onboarding for Variant D does not count.\n",
    "* Navigation: what percent of newcomers navigate to multiple tasks in the module?\n",
    "* Task selection: what percent of newcomers click on a task to do?\n",
    "* Edit success: what percent of newcomers save an edit?\n",
    "\n",
    "Also, per that task, we'll start our data gathering at the point when the Newcomer Task edit tag bug was fixed, which happened on 2020-10-28 (specifically 2020-10-28T18:40:02 per [T266600#6585899](https://phabricator.wikimedia.org/T266600#6585899). In this case, that means we'll only investigate users who registered *after* that backport happened. We'll use four weeks of registrations, which is 2020-11-25.\n",
    "\n",
    "Lastly, per the task, we're looking at activity within 24 hours of registration. It's likely that the last activity we're looking at is saving a tagged edit, and as my analysis found (see the phab task for details and link to the notebook), the 90th percentile of \"time to first tagged edit\" is just over 24 hours.\n",
    "\n",
    "## Analysis notes\n",
    "\n",
    "We'll use HomepageModule as the basis for all analysis. This should make non-JS and ad blocking affect both variants equally.\n",
    "\n",
    "Note that all questions ask \"what percent of *newcomers*\" (my emphasis). That means that we'll need to account for non-JS and ad blockers in some fashion. How about we measure what proportion of users visiting the Homepage doesn't have data in HomepageModule, and then scale *down* registrations accordingly? Meaning that we're assuming that non-JS/ad blockers affect registrations equally, and that users visiting the Homepage are not a special group. Or should we instead measure percent of visitors (as measured using HomepageModule)? The latter is a different number, but is a lot easier. Partly because the proportion of users visiting the Homepage should be equal in all variants due to the randomisation. (I checked the Variant A/B report, and there we used HomepageModule as the reference for measuring proportions, meaning it's of users visiting the Homepage as recorded through that schema)\n",
    "\n",
    "For mobile visits: measure the proportion of users who have an impression of the full suggested edits module on mobile out of users who visit the Homepage. They can only have an impression of the full module by clicking through, or going through onboarding, so we'll only need to count those.\n",
    "\n",
    "Interaction: In Variant C on desktop, the users can interact with the module immediately, it's initialized but has no topics set (and the default difficulties). There's an onboarding popup, which triggers the topic and difficulty selection. We need to discard that funnel in order to identify actual \"interaction\" events.\n",
    "\n",
    "In Variant D on desktop, the module loads to show the topic overlay. How about we identify when users activated the module, and then use any events occurring after that as \"interacting\"?\n",
    "\n",
    "In Variant C on mobile, the drawer opens onboarding. Should this onboarding be part of \"interaction\"? Otherwise, if the user dismisses the drawer, they can interact with the module because it's already initialized.\n",
    "\n",
    "In Variant D on mobile, the users choose to go through module initialization on their own accord. Therefore, any interaction with the module is valid interaction.\n",
    "\n",
    "Navigation and task selection: these are specific events that require an initialized module. In this case, we'll just determine if the user logged a navigation or task selection in HomepageModule.\n",
    "\n",
    "Edit success: we'll combine registrations with user preferences to identify their variant, then aggregate edits from `event.mediawiki_revision_tags_change` to get the most accurate count here. That means we'll use platform of registration as their platform, and not `is_mobile` in HomepageModule."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}