Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Experiment metrics #10349

Merged
merged 5 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions contents/docs/experiments/creating-an-experiment.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,6 @@ Participants are automatically split equally between variants. If you want assig

By default, PostHog runs experiments at a user-level. This means that participants are assigned to a variant based on their person properties. However, if you've created [groups](/docs/product-analytics/group-analytics), you can also run [group-targeted experiments](/blog/running-group-targeted-ab-tests). This will test how a change affects your product at a group-level by providing the same variant to every member of a group.

## Experiment goal

Setting your goal metric enables PostHog to calculate the impact of your experiment and if your results are [statistically significant](/docs/experiments/significance). You can select between a `trend` or `conversion funnel` goal.

## Distribution and release conditions

By default, your experiment will target 100% of participants. If you want to target a more specific set of participants or change the rollout percentage, you need to do this by changing the [release conditions](/docs/feature-flags/creating-feature-flags#release-conditions) for the feature flag used by your experiment.
Expand Down
2 changes: 1 addition & 1 deletion contents/docs/experiments/data-warehouse.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Using data warehouse tables in experiments

> 🚧 **Note:** Data warehouse integration is currently considered in `beta`. We are keen to gather as much feedback as possible so if you try this out please let us know. You can email [daniel.b@posthog.com](mailto:daniel.b@posthog.com), send feedback via the [in-app support panel](https://us.posthog.com#panel=support%3Afeedback%3Aexperiments%3Alow), or use one of our other [support options](/docs/support-options).

Evaluating experiment metrics always depends on events. They rely on something happening at a certain point in time. If one of your [data warehouse](/docs/data-warehouse) tables includes event-like data, you can use it as a primary or secondary metric for your trends experiment.
Evaluating [experiment metrics](/docs/experiments/metrics) always depends on events. They rely on something happening at a certain point in time. If one of your [data warehouse](/docs/data-warehouse) tables includes event-like data, you can use it as a primary or secondary metric for your trends experiment.

To use a data warehouse table with an experiment, you'll first need to join the 'events' table to your data warehouse table:

Expand Down
72 changes: 72 additions & 0 deletions contents/docs/experiments/metrics.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: Experiment metrics
sidebar: Docs
showTitle: true
---

Once you've created your experiment, you can assign metrics to let you evaluate your experiment's impact and determine if the observed results are [statistically significant](/docs/experiments/significance).

## Metric types

There are two types of experiment metrics:

1. **Funnels:** Evaluates using funnel conversions. Read more about how we calculate these in our [funnel statistic](/docs/experiments/funnels-statistics) or [funnel insights](/docs/product-analytics/funnels) docs.

2. **Trends:** Evaluates using aggregate metrics such as total count, average count per user, unique sessions, property values, and more. Trends also let you set the exposure event if you want it to be something other than `$feature_flag_called`. Read more about how we calculate these in our [trend count](/docs/experiments/trends-count-statistics) or [trend continuous](/docs/experiments/trends-continuous-statistics) statistic or [trend insights](/docs/product-analytics/trends/overview) docs.

## Primary and secondary metrics

Each metric can be set as either a **primary** or **secondary** metric. You can set up to 10 primary and secondary metrics per experiment.

**Primary metrics** represent the main goal of your experiment. They directly measure if your hypothesis was successful and are the key factor in deciding if the test achieved its primary objective.

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_10_17_53_2x_928092c2ee.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_10_17_22_2x_e0b179b143.png"
alt="Primary metrics in PostHog experiments"
classes="rounded"
/>

**Secondary metrics** provide more context for your experiment. They can help you understand the impact of your primary metric and ensure your experiment didn't have negative side effects.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exciting!


Once added and data starts being collected, we show the results as a credible interval bar chart along with details about significance, win probability, and delta between variants.

### How to read credible interval bar charts

Each bar shows how a variant is performing compared to the control (the gray bar) for the metric, using a 95% credible interval. That means there's a 95% chance the true difference for that variant falls within this range. The vertical "0%" line is your baseline:

- To the right (green): The metric is higher (an improvement).
- To the left (red): The metric is lower (a decrease).

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_19_24_2x_3cadbb532c.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_19_43_2x_730755c2c5.png"
alt="Credible interval bar chart"
classes="rounded"
/>

The width of the bar represents uncertainty. A narrower bar means we're more confident in that result, while a wider bar means it could shift either way.

The control (baseline) is always shown in gray. Other bars will be green or red—or even a mix—depending on whether the change is positive or negative.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good explanation!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied the one from in-app lol


## Shared metrics
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about putting "Shared metrics" on a separate page so it's visible in the navigation? Not a strong opinion, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is too little here at the moment I think. If/when there are some more shared metrics features, we can split it out.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good


Create a shared metric to easily reuse a funnel or a trend metric across experiments. This is ideal for key company metrics like conversion, revenue, churn, and more.

To create one, go to the [shared metrics tab](https://us.posthog.com/experiments/shared-metrics) and click **New shared metric**. From here, they are created the same way as single-use metrics.

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_02_56_2x_1a7cf4bd0e.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_03_11_2x_0e3853ba73.png"
alt="Shared metrics"
classes="rounded"
/>

Once created, choose **Shared** as a metric source when adding a primary or secondary metric to an experiment and then select the shared metric(s) you want to use.

<ProductScreenshot
imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_07_27_2x_9d55300e49.png"
imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/Clean_Shot_2025_01_14_at_11_07_12_2x_9043573390.png"
alt="Shared metrics"
classes="rounded"
/>
6 changes: 6 additions & 0 deletions src/navs/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -2622,6 +2622,12 @@ export const docsMenu = {
className: 'uppercase !bg-blue/10 !text-blue !dark:text-white !dark:bg-blue/50',
},
},
{
name: 'Experiment metrics',
url: '/docs/experiments/metrics',
icon: 'IconGraph',
color: 'red',
},
{
name: 'Adding your code',
url: '/docs/experiments/adding-experiment-code',
Expand Down
Loading