Add dbt metrics submodule

[damian3031]

resolves #26
Add dbt metrics in dbt-trino-utils:

• add dbt-metrics submodule
• bump Trino and SEP versions
• add integration tests. Lots of models from dbt-metrics needs to be overridden with models pointing to trino__fact_orders_source.csv, which have date format that is understood by Trino
• add trino shim for dbt-metrics gen_calendar_join macro

[hovaesco]

LGTM % some comments

[hovaesco]

Perhaps we could use other function to calculate median WDYT?

[damian3031]

We could use approx_percentile function. But in order to calculate median for odd and even number of values in the column, we need to add workaround, as using just approx_percentile(col_name, 0.50) for even number of values in the column is not working (workaround source):

Create sample table with even nr of rows:

CREATE TABLE memory.default.test_table AS (
select 1 AS col_name
union all
select 2 AS col_name
);

Calculate median:

SELECT
IF (
    COUNT(*) % 2 = 0,
    (approx_percentile(col_name, 0.499999) + approx_percentile(col_name, 0.50))/2.0,
    approx_percentile(col_name, 0.50)
) AS median_value
FROM memory.default.test_table;

So, it is a bit of code to calculate median in Trino. For sure the best way would be to implement MEDIAN in Trino. There is issue for that, but it is very stale and gained no attention trinodb/trino#6309
WDYT?

[hovaesco]

Just create a follow-up issue and include this code.

[SeungHuLee]

Anything wrong with getting median by value_at_quantile(tdigest_agg(col_name), 0.5)

[hovaesco]

cc @damian3031

[damian3031]

@SeungHuLee your solution has the same downside as using approx_percentile alone - it doesn't work if number of rows is even, and two middle values are not the same numbers. In this case, function should return arithmetic mean of these middle values. In your solution, it returns the bigger value.

So, running this query on table which I defined in my previous comment:

SELECT
value_at_quantile(tdigest_agg(col_name), 0.5) as median_value
FROM memory.default.test_table;

returns 2.0, but correct result is 1.5.

That's the reason why above mentioned workaround is needed.

[SeungHuLee]

@damian3031 Thanks for detailed answer! I'm not really a statistics expert, so I was just using tdigest methods according to following:
trinodb/trino#5158
trinodb/trino#4975
https://arxiv.org/pdf/1902.04023.pdf
By the way, when I use the workaround suggested above, for me the median result changes slightly everytime I execute the same query.
Is there more stable or accurate solution suggested?

[hovaesco]

Tests are red.