From 1524953d15a39fb2098d030a0b5dfd13c134acde Mon Sep 17 00:00:00 2001 From: Ian Vanagas <34755028+ivanagas@users.noreply.github.com> Date: Tue, 14 Jan 2025 11:29:02 -0800 Subject: [PATCH 1/5] add experiment metrics doc --- .../experiments/creating-an-experiment.mdx | 4 - contents/docs/experiments/metrics.mdx | 75 +++++++++++++++++++ src/navs/index.js | 6 ++ 3 files changed, 81 insertions(+), 4 deletions(-) create mode 100644 contents/docs/experiments/metrics.mdx diff --git a/contents/docs/experiments/creating-an-experiment.mdx b/contents/docs/experiments/creating-an-experiment.mdx index f65c5a3dace4..7cda7ce5ce15 100644 --- a/contents/docs/experiments/creating-an-experiment.mdx +++ b/contents/docs/experiments/creating-an-experiment.mdx @@ -56,10 +56,6 @@ Participants are automatically split equally between variants. If you want assig By default, PostHog runs experiments at a user-level. This means that participants are assigned to a variant based on their person properties. However, if you've created [groups](/docs/product-analytics/group-analytics), you can also run [group-targeted experiments](/blog/running-group-targeted-ab-tests). This will test how a change affects your product at a group-level by providing the same variant to every member of a group. -## Experiment goal - -Setting your goal metric enables PostHog to calculate the impact of your experiment and if your results are [statistically significant](/docs/experiments/significance). You can select between a `trend` or `conversion funnel` goal. - ## Distribution and release conditions By default, your experiment will target 100% of participants. If you want to target a more specific set of participants or change the rollout percentage, you need to do this by changing the [release conditions](/docs/feature-flags/creating-feature-flags#release-conditions) for the feature flag used by your experiment. diff --git a/contents/docs/experiments/metrics.mdx b/contents/docs/experiments/metrics.mdx new file mode 100644 index 000000000000..663b0fc29378 --- /dev/null +++ b/contents/docs/experiments/metrics.mdx @@ -0,0 +1,75 @@ +--- +title: Experiment metrics +sidebar: Docs +showTitle: true +--- + +After creating an experiment, you need a goal to track your experiment's impact and if your results are [statistically significant](/docs/experiments/significance). This is done with **metrics**. + +## Metric types + +There are two types of experiment metrics: + +1. **Funnels:** Evaluates using funnel conversions. Read more about how we calculate these in our [funnel statistic docs](/docs/experiments/funnels-statistics) or by reading about [funnel insights](/docs/product-analytics/funnels). + +2. **Trends:** Evaluates using aggregate metrics like total count, average count per user, unique sessions, property values, and more. Trends also let you set the exposure event if you want it to be something other than `$feature_flag_called`. Read more about how we calculate these in our [trend count](/docs/experiments/trends-count-statistics) or [trend continuous](/docs/experiments/trends-continuous-statistics) statistic docs or by reading about [trend insights](/docs/product-analytics/trends/overview). + +## Primary and secondary metrics + +Each metric can be set as either a **primary** or **secondary** metric. You can set up to 10 primary and secondary metrics per experiment. + +**Primary metrics** represent the main goal of your experiment. They directly measure whether your hypothesis was successful and are the key factor in deciding if the test achieved its primary objective. + + + +**Secondary metrics** are metrics provide more context for your experiment. They can help you understand the impact of your primary metric and ensure your experiment didn't have negative side effects. + +Once added and data starts being collected, they will show the results as a credible interval bar chart along with details about significance, win probability, and delta between variants. + +### How to read credible interval bar charts + +Each bar shows how a variant is performing compared to the control (the gray bar) for the metric, using a 95% credible interval. That means there's a 95% chance the true difference for that variant falls within this range. The vertical "0%" line is your baseline: + +- To the right (green): The metric is higher (an improvement). +- To the left (red): The metric is lower (a decrease). + + + +The width of the bar represents uncertainty. A narrower bar means we're more confident in that result, while a wider bar means it could shift either way. + +The control (baseline) is always shown in gray. Other bars will be green or red—or even a mix—depending on whether the change is positive or negative. + +## Shared metrics + +Shared metrics are ones that can be easily reused across experiments. This is ideal for key company metrics like conversion, revenue, churn, and more. + +To create one, go to the [shared metrics tab](https://us.posthog.com/experiments/shared-metrics) and click **New shared metric**. From here, they are created the same way as one-off metrics. + +dark=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_03_11_2x_0e3853ba73.png +light=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_02_56_2x_1a7cf4bd0e.png + + + +Once created, choose **Shared** as a metric source when adding a primary or secondary metric to an experiment and then select the shared metric(s) you want to use. + + \ No newline at end of file diff --git a/src/navs/index.js b/src/navs/index.js index c3b1a8a1ef23..bb609b8d7789 100644 --- a/src/navs/index.js +++ b/src/navs/index.js @@ -2622,6 +2622,12 @@ export const docsMenu = { className: 'uppercase !bg-blue/10 !text-blue !dark:text-white !dark:bg-blue/50', }, }, + { + name: 'Experiment metrics', + url: '/docs/experiments/metrics', + icon: 'IconGraph', + color: 'red', + }, { name: 'Adding your code', url: '/docs/experiments/adding-experiment-code', From d7137d6f5d7c4778147c6551729b090f94767af8 Mon Sep 17 00:00:00 2001 From: Ian Vanagas <34755028+ivanagas@users.noreply.github.com> Date: Tue, 14 Jan 2025 11:33:00 -0800 Subject: [PATCH 2/5] edits --- contents/docs/experiments/metrics.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contents/docs/experiments/metrics.mdx b/contents/docs/experiments/metrics.mdx index 663b0fc29378..6b798a4af57e 100644 --- a/contents/docs/experiments/metrics.mdx +++ b/contents/docs/experiments/metrics.mdx @@ -10,15 +10,15 @@ After creating an experiment, you need a goal to track your experiment's impact There are two types of experiment metrics: -1. **Funnels:** Evaluates using funnel conversions. Read more about how we calculate these in our [funnel statistic docs](/docs/experiments/funnels-statistics) or by reading about [funnel insights](/docs/product-analytics/funnels). +1. **Funnels:** Evaluates using funnel conversions. Read more about how we calculate these in our [funnel statistic](/docs/experiments/funnels-statistics) or [funnel insights](/docs/product-analytics/funnels) docs. -2. **Trends:** Evaluates using aggregate metrics like total count, average count per user, unique sessions, property values, and more. Trends also let you set the exposure event if you want it to be something other than `$feature_flag_called`. Read more about how we calculate these in our [trend count](/docs/experiments/trends-count-statistics) or [trend continuous](/docs/experiments/trends-continuous-statistics) statistic docs or by reading about [trend insights](/docs/product-analytics/trends/overview). +2. **Trends:** Evaluates using aggregate metrics such as total count, average count per user, unique sessions, property values, and more. Trends also let you set the exposure event if you want it to be something other than `$feature_flag_called`. Read more about how we calculate these in our [trend count](/docs/experiments/trends-count-statistics) or [trend continuous](/docs/experiments/trends-continuous-statistics) statistic or [trend insights](/docs/product-analytics/trends/overview) docs. ## Primary and secondary metrics Each metric can be set as either a **primary** or **secondary** metric. You can set up to 10 primary and secondary metrics per experiment. -**Primary metrics** represent the main goal of your experiment. They directly measure whether your hypothesis was successful and are the key factor in deciding if the test achieved its primary objective. +**Primary metrics** represent the main goal of your experiment. They directly measure if your hypothesis was successful and are the key factor in deciding if the test achieved its primary objective. -**Secondary metrics** are metrics provide more context for your experiment. They can help you understand the impact of your primary metric and ensure your experiment didn't have negative side effects. +**Secondary metrics** provide more context for your experiment. They can help you understand the impact of your primary metric and ensure your experiment didn't have negative side effects. -Once added and data starts being collected, they will show the results as a credible interval bar chart along with details about significance, win probability, and delta between variants. +Once added and data starts being collected, we show the results as a credible interval bar chart along with details about significance, win probability, and delta between variants. ### How to read credible interval bar charts From 9a59f00bb99520d1eb611be537a1a547da7ef9a1 Mon Sep 17 00:00:00 2001 From: Ian Vanagas <34755028+ivanagas@users.noreply.github.com> Date: Tue, 14 Jan 2025 11:35:08 -0800 Subject: [PATCH 3/5] link --- contents/docs/experiments/data-warehouse.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/experiments/data-warehouse.mdx b/contents/docs/experiments/data-warehouse.mdx index 9934d83ed29d..be0f70a976d7 100644 --- a/contents/docs/experiments/data-warehouse.mdx +++ b/contents/docs/experiments/data-warehouse.mdx @@ -4,7 +4,7 @@ title: Using data warehouse tables in experiments > 🚧 **Note:** Data warehouse integration is currently considered in `beta`. We are keen to gather as much feedback as possible so if you try this out please let us know. You can email [daniel.b@posthog.com](mailto:daniel.b@posthog.com), send feedback via the [in-app support panel](https://us.posthog.com#panel=support%3Afeedback%3Aexperiments%3Alow), or use one of our other [support options](/docs/support-options). -Evaluating experiment metrics always depends on events. They rely on something happening at a certain point in time. If one of your [data warehouse](/docs/data-warehouse) tables includes event-like data, you can use it as a primary or secondary metric for your trends experiment. +Evaluating [experiment metrics](/docs/experiments/metrics) always depends on events. They rely on something happening at a certain point in time. If one of your [data warehouse](/docs/data-warehouse) tables includes event-like data, you can use it as a primary or secondary metric for your trends experiment. To use a data warehouse table with an experiment, you'll first need to join the 'events' table to your data warehouse table: From db37e6f31f974050d1d3ec1ac4c7056ac0fcde4b Mon Sep 17 00:00:00 2001 From: Ian Vanagas <34755028+ivanagas@users.noreply.github.com> Date: Tue, 14 Jan 2025 13:20:49 -0800 Subject: [PATCH 4/5] Daniel's suggestions Co-authored-by: Daniel Bachhuber --- contents/docs/experiments/metrics.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contents/docs/experiments/metrics.mdx b/contents/docs/experiments/metrics.mdx index 6b798a4af57e..b41d20d59993 100644 --- a/contents/docs/experiments/metrics.mdx +++ b/contents/docs/experiments/metrics.mdx @@ -4,7 +4,7 @@ sidebar: Docs showTitle: true --- -After creating an experiment, you need a goal to track your experiment's impact and if your results are [statistically significant](/docs/experiments/significance). This is done with **metrics**. +Once you've created your experiment, you can assign metrics to let you evaluate your experiment's impact and determine if the observed results are [statistically significant](/docs/experiments/significance). ## Metric types @@ -51,9 +51,9 @@ The control (baseline) is always shown in gray. Other bars will be green or red ## Shared metrics -Shared metrics are ones that can be easily reused across experiments. This is ideal for key company metrics like conversion, revenue, churn, and more. +Create a shared metric to easily reuse a funnel or a trend metric across experiments. This is ideal for key company metrics like conversion, revenue, churn, and more. -To create one, go to the [shared metrics tab](https://us.posthog.com/experiments/shared-metrics) and click **New shared metric**. From here, they are created the same way as one-off metrics. +To create one, go to the [shared metrics tab](https://us.posthog.com/experiments/shared-metrics) and click **New shared metric**. From here, they are created the same way as single-use metrics. dark=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_03_11_2x_0e3853ba73.png light=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_02_56_2x_1a7cf4bd0e.png From ba42dc5fd2b53db6448135da2d6ae41652d4cbe3 Mon Sep 17 00:00:00 2001 From: Ian Vanagas <34755028+ivanagas@users.noreply.github.com> Date: Tue, 14 Jan 2025 13:25:05 -0800 Subject: [PATCH 5/5] fix unused images --- contents/docs/experiments/metrics.mdx | 3 --- 1 file changed, 3 deletions(-) diff --git a/contents/docs/experiments/metrics.mdx b/contents/docs/experiments/metrics.mdx index b41d20d59993..b79449ad0b9d 100644 --- a/contents/docs/experiments/metrics.mdx +++ b/contents/docs/experiments/metrics.mdx @@ -55,9 +55,6 @@ Create a shared metric to easily reuse a funnel or a trend metric across experim To create one, go to the [shared metrics tab](https://us.posthog.com/experiments/shared-metrics) and click **New shared metric**. From here, they are created the same way as single-use metrics. -dark=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_03_11_2x_0e3853ba73.png -light=https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_02_56_2x_1a7cf4bd0e.png -