Set Up Fretboard Experiment for TV Guide

[athomasmoz]

Why/User Benefit/User Problem

As a Product Manager, I want an A/B test set up for the TV Guide feature, so that I can measure whether it's providing value to users

Acceptance Criteria (how do I know when I’m done?)

• Set up an experiment placeholder where our changes can be added
• Turn experiment to 0%

• Come up with a plan with QA to test "channels on" and communicate it over email - Timebox this to max 1 day
• On June 3: add channels to experiment Set Up Fretboard Experiment for TV Guide #2320 (we want the categories to be visible for testing our release build before users see it so we can find regressions.) and pref to 0% for release builds

• Group A is the control (do not show new Channels or Channels onboarding)
• Group B will show:
  • Each of the 5 new categories
  • Onboarding screen
• TV Guide experiment is set to X% (TBD from @athomasmoz) - Fretboard bug is Set Up Fretboard Experiment for TV Guide #2320

Not Included

• This does not include turning on or running the experiment (that is Add TV Guide channels to Fretboard Experiment #2321 ) That bug is closed as a dupe of this one.

[liuche]

@athomasmoz has already started the conversation w/ Felix from DS to get percentages.

[liuche]

Looking at whether we can pass in an experiment on the commandline (looking at Fretboard docs for “Setting override values”)

[liuche]

Fennec uses setOverride for allowing live-toggling of experiments through about:experiments, in the Switchboard addon.

This uses SharedPreferences to store override values that get read by Fretboard/Switchboard before loading experiment config.

We don't necessarily need the live overrides - and in fact don't want them to be persisted (so that we don't mix up experiments).

[severinrudie]

Looking at whether we can pass in an experiment on the commandline (looking at Fretboard docs for “Setting override values”)

We will also have to look into mocking locale.

[liuche]

With this patch, you can now specify a set of experiments to start with the following command:
adb shell am start -n org.mozilla.tv.firefox/org.mozilla.tv.firefox.MainActivity -a android.intent.action.MAIN --esa activeExperiments TvGuideChannels-2195:A,HintBar-2011:B

Experiment input is of the format (and the list must be comma-separated, no spaces):
experimentName:branch

where:

• experimentName: experiment names can be found in this config file
• branch: this determines if the experiment is "on" or "off" (this is inconsistent: for Channels, B is on; for HintBar B is off)

@npark-mozilla does that seem reasonable to you? One caveat is that after running this command, subsequent restarts of that app profile will continue to have those defaults (and you'd need to clear the app data to reset them).

[nojunpark]

@liuche Thanks! so in above example, it is setting the app to be the A of TvGuideChannels, and B of HintBar I assume, and I can specify multiple fretboard experiements with one command.

Also, when we say experiement is "on", then it means it's NOT a control group (i.e. the feature is enabled) right? Sorry, just wanted to be super clear on that.

[liuche]

@liuche Thanks! so in above example, it is setting the app to be the A of TvGuideChannels, and B of HintBar I assume, and I can specify multiple fretboard experiements with one command.

Exactly - currently we only have 2 visible experiments, but this would allow you to specify any number of "branches" for the client to be on.

Also, when we say experiement is "on", then it means it's NOT a control group (i.e. the feature is enabled) right? Sorry, just wanted to be super clear on that.

Yes, you're right that this isn't totally clear, but that's the gist of it. You can think of this as: HintBarA is the branch where the hint bar is shown, and HintBarB is the branch where the hint bar is not shown. In theory we could have multiple "visible" branches (like showing 3 different kinds of onboarding), so the idea if you want to select which one is the state you want to see. This means you can't specify multiple branches from the same experiment.

Caveat from before stands, if you set an override this way, it'll persist that experiment on subsequent runs.
Also, this must be done from cold start (e.g. app can't be open, might not work).

[mcomella]

Reopening: liuche has to follow up with fixes for nits.

[liuche]

Follow-up items:

• Address nits
• Email out to QA
• Tests for this

[abodea]

Hello, please note that the test cases are done in test rail.