Funnel conversion rate appears to change, even after the window has closed.

[kpthatsme]

Bug description

This morning (Apr 5 21) I ran this chart .

I saw a value of 11% this morning for the week of Mar 28.

I reran this query and the value has dropped to 10%. I'm confused to how the conversion rate for week of Mar 28 is able to change given today is April 5th. I believe I've seen this happen with a funnel before. It could be I'm missing something with how we resolve weeks.

Expected behavior

• When looking at a weekly trends funnel, I'd expect dates in a completed week to have stable conversion rates.

How to reproduce

1. Run a funnel trends chart, observe value of completed week
2. Rerun the query, observe an updated value for the data point, even though the week was completed

It's a bit tricky since the data stays at the new value once calculated. I also don't know how to repro consistently since it's unclear to me which sets of data is correct (since we can't see the users at each step).

Environment

• PostHog Cloud
• self-hosted PostHog, version/commit: please provide

Additional context

It could just be misinterpreting the UX and somehow the week of Mar 28 isn't completed.

Thank you for your bug report – we love squashing them!

[paolodamico]

Seems strongly related to https://github.com/PostHog/product-internal/issues/10

[EDsCODE]

I'm guessing it is same underlying issue with queries dependent on cohort. Maybe it's time to add some sort of notice on queries that might change due to cohort recalculation?

[paolodamico]

Not 100% sure this is the problem, but if this is it one thing we could do is when showing when a graph was last refreshed, we also take into account when all the cohorts were last refreshed, and show the oldest data from all. We could also automatically trigger a cohort refresh when you click on refresh.

[kpthatsme]

Hey @paolodamico I was thinking about this some more, and I think one thing that could help as well is defining a 'conversion window' for funnels.

This would be a range of time (hours, days, months, weeks) you define when creating your chart, and what it does is say for someone to get through this funnel, they need to go from event A onwards within the conversion window.

So for example, if we have a weekly funnel with a 7 day conversion window, the last data point isn't 'complete' until 7 days after it entered into the funnel.

I think that could partially be what's going on here, I feel like we have a bit of a lagging trend to discovering learnings and I wonder if some data points are incomplete even though the overall time window is marked solid. For instance, on May 9, the week of May 2 looks 'complete' but if we had a 5 day conversion window, the week of May 2 wouldn't actually be complete until May 8 + 5 days.

Let me know if you'd prefer I make a separate ticket re: conversion windows instead!