*: add builtin aggregate function `var_pop` #11155

githubFZX · 2019-07-09T14:16:22Z

What problem does this PR solve?

Tests

mysql> create table numerical_t(
    -> a tinyint,
    -> b smallint,
    -> c mediumint,
    -> d int,
    -> e bigint,
    -> f float,
    -> g double,
    -> h decimal
    -> );
Query OK, 0 rows affected (0.10 sec)

mysql> insert into numerical_t 
    -> values
    -> (1, 2, 3, 4, 5, 6.1, 7.2, 8.3), 
    -> (2, 3, 4, 5, 6, 7.2, 8.3, 9.4),
    -> (1, 3, 4, 5, 6, 7.1, 8.2, 9.3);
Query OK, 3 rows affected, 3 warnings (0.01 sec)

mysql> select a, var_pop(b),var_pop(c),var_pop(d),var_pop(e),var_pop(f),var_pop(g),var_pop(h) from numerical_t group by a;
+------+------------+------------+------------+------------+------------+--------------------+------------+
| a    | var_pop(b) | var_pop(c) | var_pop(d) | var_pop(e) | var_pop(f) | var_pop(g)         | var_pop(h) |
+------+------------+------------+------------+------------+------------+--------------------+------------+
|    1 |       0.25 |       0.25 |       0.25 |       0.25 |       0.25 | 0.2500000000000071 |       0.25 |
|    2 |          0 |          0 |          0 |          0 |          0 |                  0 |          0 |
+------+------------+------------+------------+------------+------------+--------------------+------------+
2 rows in set (0.00 sec)

CLAassistant · 2019-07-09T14:16:34Z

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
19 out of 21 committers have signed the CLA.

✅ lonng
✅ crazycs520
✅ qw4990
✅ foreyes
✅ tiancaiamao
✅ lysu
✅ jackysp
✅ zz-jason
✅ sticnarf
✅ SunRunAway
✅ eurekaka
✅ disksing
✅ EmilStenstrom
✅ wshwsh12
✅ Reminiscent
✅ WangXiangUSTC
✅ lamxTyler
✅ ZYunH
✅ jiyfhust
❌ root
❌ githubFZX

root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

… of boolean type (pingcap#11239)

…ngcap#10962)

…cap#11089)

…#11137)

…cap#11114) Signed-off-by: H-ZeX <hzx20112012@gmail.com>

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…10861)

…11032) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

zz-jason · 2019-07-17T09:34:02Z

@githubFZX Thanks for your contribution, please follow the Contribution Guide to add a proper PR title.

Signed-off-by: Ryan Leung <rleungx@gmail.com>

githubFZX · 2019-07-17T13:25:31Z

Okay， thank you for your guidance and wish you have a happy life.

…

------------------ 原始邮件 ------------------ 发件人: "Zhang Jian"<notifications@github.com>; 发送时间: 2019年7月17日(星期三) 下午5:35 收件人: "pingcap/tidb"<tidb@noreply.github.com>; 抄送: "1411679031"<1411679031@qq.com>;"Mention"<mention@noreply.github.com>; 主题: Re: [pingcap/tidb] Add var_pop (#11155) @githubFZX Thanks for your contribution, please follow the Contribution Guide to add a proper PR title. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

…cap#11310)

…inconsistent 0 results. (pingcap#11251)

…orted (pingcap#11184)

zz-jason · 2019-09-01T03:25:43Z

/run-all-tests

zz-jason · 2019-09-01T03:26:36Z

@githubFZX please sign the CLA. Ref to #11155 (comment)

zz-jason · 2019-09-01T03:21:31Z

executor/aggregate_test.go

@@ -344,11 +352,11 @@ func (s *testSuite1) TestAggregation(c *C) {
 	_, err = tk.Exec("select std_samp(a) from t")
 	// TODO: Fix this error message.
 	c.Assert(errors.Cause(err).Error(), Equals, "[expression:1305]FUNCTION std_samp does not exist")
-	_, err = tk.Exec("select variance(a) from t")
+	//_, err = tk.Exec("select variance(a) from t")
+	//c.Assert(errors.Cause(err).Error(), Equals, "unsupported agg function: variance")


please do not comment them out, you can simply rewrite the expected behavior of this test.

zz-jason · 2019-09-01T03:21:38Z

executor/aggregate_test.go

-	_, err = tk.Exec("select var_pop(a) from t")
-	c.Assert(errors.Cause(err).Error(), Equals, "unsupported agg function: var_pop")
+	//_, err = tk.Exec("select var_pop(a) from t")
+	//c.Assert(errors.Cause(err).Error(), Equals, "unsupported agg function: var_pop")


zz-jason · 2019-09-01T03:23:29Z

expression/aggregation/base_func.go

@@ -233,6 +235,12 @@ func (a *baseFuncDesc) typeInfer4LeadLag(ctx sessionctx.Context) {
 	}
 }

+func (a *baseFuncDesc) typeInfer4VarPop(ctx sessionctx.Context) {
+	//var_pop's return value type is double


Suggested change

//var_pop's return value type is double

// var_pop's return value type is double

It's recommended to add a space between // and the first word.

zz-jason · 2019-09-01T05:14:15Z

executor/aggfuncs/func_varpop.go

+	baseAggFunc
+}
+
+type partialResult4Float64 struct {


It's better to use the algorithm described by Chan, Golub, and LeVeque in "Algorithms for computing the sample variance: analysis and recommendations" to calculate the variance. You can find how apache hive implement this at here: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java#L194

Okay. I will refer to it.

zz-jason · 2019-11-05T01:07:12Z

@githubFZX friendly ping, any update?

zhaoyanxing and others added 17 commits July 15, 2019 11:33

Function INSERT should be NULL if any argument is NULL (pingcap#11237)

07e971f

*: improve column size calculation of statistics (pingcap#11091)

478d9af

executor,sessionctx: refine output format for tidb-specific variables…

5ce64bd

… of boolean type (pingcap#11239)

*: reduce tikvrpc request size (pingcap#11055)

fb64a2a

planner: disable projection elimination for select plan of update (pi…

0e65e35

…ngcap#10962)

executor: compute ADMIN CHECKSUM for partitioned tables correc… (ping…

6c44887

…cap#11089)

planner: fix NAME_CONST function compatibility (pingcap#11241)

32e4f82

*: reduce tikvrpc resp struct size (pingcap#11056)

a48894b

expression: add max_allowed_packet check in concat/concat_ws (pingcap…

cbf44e9

…#11137)

types: fix overflow check of types/convert.go::floatStrToIntStr (ping…

711f674

…cap#11114) Signed-off-by: H-ZeX <hzx20112012@gmail.com>

config: refine comment (pingcap#11272)

bc7885f

store/tikv: Make GC's tests more strict (pingcap#11084)

dfece20

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

types: fix delete error when convert string to float or int (pingcap#…

a19c896

…10861)

Update README.md (pingcap#11267)

17d1259

*: refine text protocol multiple query response (pingcap#11263)

8a37aa2

fix microseconds behaviour in DATE_ADD() (pingcap#11280)

81fbd14

store/tikv: Avoid sending to channel everywhere in runGCJob (pingcap#…

a3b8d79

…11032) Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

zz-jason added contribution This PR is from a community contributor. sig/execution SIG execution labels Jul 17, 2019

fix scatter range (pingcap#11281)

069db90

Signed-off-by: Ryan Leung <rleungx@gmail.com>

jackysp and others added 7 commits July 18, 2019 16:20

metrics: add missing tikv_txn_cmd_counter (pingcap#11292)

166a054

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

executor: speedup unit test (pingcap#10930)

eacfd0c

plugin: add "exec start-time" to GeneralEvent (pingcap#11293)

3f19784

expression: let testEvaluatorSuite run serially to avoid aff… (ping…

7190987

…cap#11310)

expression: fix getIntervalFromDecimal in DATE_ADD() (pingcap#11297)

16da65a

expression, types: fix Mod(%), Multiple(*), Minus(-) operators meets …

b80e883

…inconsistent 0 results. (pingcap#11251)

executor: update test since DELETE FROM ... AS ... syntax is now supp…

b8395a8

…orted (pingcap#11184)

githubFZX added 9 commits August 30, 2019 20:59

modify format

10d5747

modify format

bf4cd39

Merge branch 'varpop' of github.com:githubFZX/tidb into varpop

375360e

modify file aggregate_test.go

6b1db90

modify aggregate_test.go

884af95

modify aggregate_test.go

52e8d0b

modify test case for varpop

a26e876

modify test case for varpop

fa74939

modify test case for varpop

6fb9d2e

zz-jason reviewed Sep 1, 2019

View reviewed changes

zz-jason changed the title ~~Add var_pop~~ *: add builtin aggregate function var_pop Sep 1, 2019

zz-jason reviewed Sep 1, 2019

View reviewed changes

githubFZX force-pushed the varpop branch from d85844e to 6fb9d2e Compare September 29, 2019 07:47

using new algorithm to calculate variance

cb94e59

githubFZX requested review from a team as code owners December 3, 2019 12:04

ghost requested review from wshwsh12, XuHuaiyu, alivxxx and lzmhhh123 and removed request for a team December 3, 2019 12:04

lzmhhh123 closed this Dec 3, 2019

zz-jason reopened this Dec 11, 2019

alivxxx removed their request for review December 12, 2019 04:34

lzmhhh123 removed their request for review December 12, 2019 07:07

githubFZX closed this Dec 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add builtin aggregate function `var_pop` #11155

*: add builtin aggregate function `var_pop` #11155

githubFZX commented Jul 9, 2019 •

edited

Loading

CLAassistant commented Jul 9, 2019 •

edited

Loading

zz-jason commented Jul 17, 2019

githubFZX commented Jul 17, 2019 via email

zz-jason commented Sep 1, 2019

zz-jason commented Sep 1, 2019

zz-jason Sep 1, 2019

zz-jason Sep 1, 2019

zz-jason Sep 1, 2019

zz-jason Sep 1, 2019

githubFZX Sep 1, 2019

zz-jason commented Nov 5, 2019

	//var_pop's return value type is double
	// var_pop's return value type is double

*: add builtin aggregate function var_pop #11155

*: add builtin aggregate function var_pop #11155

Conversation

githubFZX commented Jul 9, 2019 • edited Loading

What problem does this PR solve?

Tests

CLAassistant commented Jul 9, 2019 • edited Loading

zz-jason commented Jul 17, 2019

githubFZX commented Jul 17, 2019 via email

zz-jason commented Sep 1, 2019

zz-jason commented Sep 1, 2019

zz-jason Sep 1, 2019

Choose a reason for hiding this comment

zz-jason Sep 1, 2019

Choose a reason for hiding this comment

zz-jason Sep 1, 2019

Choose a reason for hiding this comment

zz-jason Sep 1, 2019

Choose a reason for hiding this comment

githubFZX Sep 1, 2019

Choose a reason for hiding this comment

zz-jason commented Nov 5, 2019

*: add builtin aggregate function `var_pop` #11155

*: add builtin aggregate function `var_pop` #11155

githubFZX commented Jul 9, 2019 •

edited

Loading

CLAassistant commented Jul 9, 2019 •

edited

Loading