Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(trends): add moving average smoothing_intervals param #6435

Merged
merged 58 commits into from
Mar 23, 2022
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
d871824
feat(trends): add moving average `smoothing_intervals` param
Oct 14, 2021
f0a9623
WIP UI for selecting smoothing
MarconLP Oct 14, 2021
b58f38b
Merge branch 'feat/trend-smoothing' of https://github.com/PostHog/pos…
MarconLP Oct 14, 2021
bff2105
WIP: Fix typing and appearce logic
MarconLP Oct 14, 2021
8ef04d6
Made smoothing extensible beyond day
MarconLP Oct 15, 2021
d2e0d76
Test now considers a non-trivial case
marcushyett-ph Oct 15, 2021
240135c
chore(smoothing): use url params for state storage of smoothing value
Oct 15, 2021
9085cba
Added further tests for smoothing intervals
marcushyett-ph Oct 15, 2021
1e08f21
fix test_to_dict
Oct 15, 2021
e5c9db7
fix trend tests
Oct 15, 2021
12789d1
Added test to validate preceeding limit
marcushyett-ph Oct 15, 2021
0a5b888
Fixed test for 2 interval
marcushyett-ph Oct 15, 2021
aa52f33
clear django redis cache before each test
Oct 18, 2021
06daa1a
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Oct 18, 2021
ce0ac92
add explicit ordering to dashboard items in refresh test
Oct 18, 2021
c2be07e
Revert "clear django redis cache before each test"
Oct 18, 2021
8950bd2
format
Oct 18, 2021
559b0c6
clickhouse trends: SET allow_experimental_window_functions = 1
Oct 18, 2021
1fc81a4
fix setting of allow_experimental_window_functions
Oct 18, 2021
ae73e13
add cache clears for requests
neilkakkar Oct 22, 2021
8cf47d4
Delete SmoothingFilterLogic.ts
marcushyett-ph Oct 28, 2021
b194837
Use `SMOOTHING_INTERVALS` instead of string literal
Oct 28, 2021
397be27
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Oct 28, 2021
493581f
move test_trends to ee
Oct 28, 2021
81d2997
actually delete test_trends.py
Oct 28, 2021
7f61950
Transitioned smoothing logic to use Logic
MarconLP Nov 4, 2021
7be6a20
WIP: Moving away from url based logic
MarconLP Nov 5, 2021
15dc062
WIP: Refactored logic on listeners
MarconLP Nov 5, 2021
a6c6de0
fix storybook
Nov 5, 2021
e71b36e
refactor
Nov 5, 2021
7857074
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Nov 23, 2021
7ba88ff
set redis url
Nov 23, 2021
71d4e4c
remove keastory
Nov 23, 2021
37e1be4
Move interval filter to setFilters insight logic
marcushyett-ph Nov 23, 2021
2f59db3
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
marcushyett-ph Dec 2, 2021
ec9a486
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Dec 14, 2021
8b23858
update trend snapshots
Dec 14, 2021
df2bd21
run prettier
Dec 14, 2021
363a427
Add stickiness test deps that were in test_trends
Dec 14, 2021
3a253d7
Fix storybook
Dec 14, 2021
01142b2
fix stickiness tests
Dec 14, 2021
e379d1e
update snapshot
Dec 14, 2021
e4d32f0
fix cohort test
Dec 14, 2021
67116ca
only display smoothing on linear
Dec 14, 2021
ef05820
fix tests
Dec 17, 2021
95ad774
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Dec 17, 2021
a7992b3
Add some tests
Dec 17, 2021
9359322
Merge remote-tracking branch 'origin/master' into feat/trend-smoothing
Dec 17, 2021
5a2509d
added box to make dropdown consistent
marcushyett-ph Jan 25, 2022
34f83f9
Co-authored-by: Harry Waye <harry@microwayes.net>
marcushyett-ph Jan 31, 2022
0df2239
Merge branch 'master' into feat/trend-smoothing
timgl Mar 22, 2022
e65559c
Merge branch 'master' of github.com:PostHog/posthog into feat/trend-s…
timgl Mar 22, 2022
f224521
Floor
timgl Mar 22, 2022
2963f45
fix tests
timgl Mar 22, 2022
e28a3cf
fix tests
timgl Mar 22, 2022
46766ae
fix test
timgl Mar 23, 2022
b1602f2
Add feature flag
timgl Mar 23, 2022
bc8bbd2
Merge branch 'master' into feat/trend-smoothing
timgl Mar 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions ee/clickhouse/queries/trends/total_volume.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,31 @@ def _total_volume_query(self, entity: Entity, filter: Filter, team_id: int) -> T
**content_sql_params,
parsed_date_to=trend_event_query.parsed_date_to,
parsed_date_from=trend_event_query.parsed_date_from,
**trend_event_query.active_user_params
**trend_event_query.active_user_params,
)
else:
content_sql = VOLUME_SQL.format(event_query=event_query, **content_sql_params)

null_sql = NULL_SQL.format(trunc_func=trunc_func, interval_func=interval_func)
params["interval"] = filter.interval
final_query = AGGREGATE_SQL.format(null_sql=null_sql, content_sql=content_sql)

# If we have a smoothing interval > 1 then add in the sql to
# handling rolling average. Else just do a sum. This is possibly an
# nessacary optimization.
if filter.smoothing_intervals > 1:
smoothing_operation = f"""
AVG(SUM(total))
hazzadous marked this conversation as resolved.
Show resolved Hide resolved
OVER (
ORDER BY day_start
ROWS BETWEEN {filter.smoothing_intervals - 1} PRECEDING
AND CURRENT ROW
)"""
else:
smoothing_operation = "SUM(total)"

final_query = AGGREGATE_SQL.format(
null_sql=null_sql, content_sql=content_sql, smoothing_operation=smoothing_operation
)
return final_query, params, self._parse_total_volume_result(filter)

def _parse_total_volume_result(self, filter: Filter) -> Callable:
Expand Down
10 changes: 9 additions & 1 deletion ee/clickhouse/sql/trends/aggregate.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
AGGREGATE_SQL = """
SELECT groupArray(day_start) as date, groupArray(count) as data FROM (
SELECT SUM(total) AS count, day_start from ({null_sql} UNION ALL {content_sql}) group by day_start order by day_start
SELECT {smoothing_operation} AS count, day_start
from (
{null_sql}
UNION ALL
{content_sql}
)
group by day_start
order by day_start
SETTINGS allow_experimental_window_functions = 1
)
"""
Loading