Skip to content

Commit

Permalink
[SPARK-23091][ML] Incorrect unit test for approxQuantile
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Narrow bound on approx quantile test to epsilon from 2*epsilon to match paper

## How was this patch tested?

Existing tests.

Author: Sean Owen <sowen@cloudera.com>

Closes #20324 from srowen/SPARK-23091.

(cherry picked from commit 396cdfb)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
  • Loading branch information
srowen authored and gatorsmile committed Jan 20, 2018
1 parent c647f91 commit 0cde521
Showing 1 changed file with 12 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -154,24 +154,24 @@ class DataFrameStatSuite extends QueryTest with SharedSQLContext {
val Array(d1, d2) = df.stat.approxQuantile("doubles", Array(q1, q2), epsilon)
val Array(s1, s2) = df.stat.approxQuantile("singles", Array(q1, q2), epsilon)

val error_single = 2 * 1000 * epsilon
val error_double = 2 * 2000 * epsilon
val errorSingle = 1000 * epsilon
val errorDouble = 2.0 * errorSingle

assert(math.abs(single1 - q1 * n) < error_single)
assert(math.abs(double2 - 2 * q2 * n) < error_double)
assert(math.abs(s1 - q1 * n) < error_single)
assert(math.abs(s2 - q2 * n) < error_single)
assert(math.abs(d1 - 2 * q1 * n) < error_double)
assert(math.abs(d2 - 2 * q2 * n) < error_double)
assert(math.abs(single1 - q1 * n) <= errorSingle)
assert(math.abs(double2 - 2 * q2 * n) <= errorDouble)
assert(math.abs(s1 - q1 * n) <= errorSingle)
assert(math.abs(s2 - q2 * n) <= errorSingle)
assert(math.abs(d1 - 2 * q1 * n) <= errorDouble)
assert(math.abs(d2 - 2 * q2 * n) <= errorDouble)

// Multiple columns
val Array(Array(ms1, ms2), Array(md1, md2)) =
df.stat.approxQuantile(Array("singles", "doubles"), Array(q1, q2), epsilon)

assert(math.abs(ms1 - q1 * n) < error_single)
assert(math.abs(ms2 - q2 * n) < error_single)
assert(math.abs(md1 - 2 * q1 * n) < error_double)
assert(math.abs(md2 - 2 * q2 * n) < error_double)
assert(math.abs(ms1 - q1 * n) <= errorSingle)
assert(math.abs(ms2 - q2 * n) <= errorSingle)
assert(math.abs(md1 - 2 * q1 * n) <= errorDouble)
assert(math.abs(md2 - 2 * q2 * n) <= errorDouble)
}

// quantile should be in the range [0.0, 1.0]
Expand Down

0 comments on commit 0cde521

Please sign in to comment.