Skip to content

Commit

Permalink
[SPARK-20423][ML] fix MLOR coeffs centering when reg == 0
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

When reg == 0, MLOR has multiple solutions and we need to centralize the coeffs to get identical result.
BUT current implementation centralize the `coefficientMatrix` by the global coeffs means.

In fact the `coefficientMatrix` should be centralized on each feature index itself.
Because, according to the MLOR probability distribution function, it can be proven easily that:
suppose `{ w0, w1, .. w(K-1) }` make up the `coefficientMatrix`,
then `{ w0 + c, w1 + c, ... w(K - 1) + c}` will also be the equivalent solution.
`c` is an arbitrary vector of `numFeatures` dimension.
reference
https://core.ac.uk/download/pdf/6287975.pdf

So that we need to centralize the `coefficientMatrix` on each feature dimension separately.

**We can also confirm this through R library `glmnet`, that MLOR in `glmnet` always generate coefficients result that the sum of each dimension is all `zero`, when reg == 0.**

## How was this patch tested?

Tests added.

Author: WeichenXu <WeichenXu123@outlook.com>

Closes #17706 from WeichenXu123/mlor_center.
  • Loading branch information
WeichenXu123 authored and dbtsai committed Apr 21, 2017
1 parent a750a59 commit eb00378
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -609,9 +609,14 @@ class LogisticRegression @Since("1.2.0") (
Friedman, et al. "Regularization Paths for Generalized Linear Models via
Coordinate Descent," https://core.ac.uk/download/files/153/6287975.pdf
*/
val denseValues = denseCoefficientMatrix.values
val coefficientMean = denseValues.sum / denseValues.length
denseCoefficientMatrix.update(_ - coefficientMean)
val centers = Array.fill(numFeatures)(0.0)
denseCoefficientMatrix.foreachActive { case (i, j, v) =>
centers(j) += v
}
centers.transform(_ / numCoefficientSets)
denseCoefficientMatrix.foreachActive { case (i, j, v) =>
denseCoefficientMatrix.update(i, j, v - centers(j))
}
}

// center the intercepts when using multinomial algorithm
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1139,6 +1139,9 @@ class LogisticRegressionSuite
0.10095851, -0.85897154, 0.08392798, 0.07904499), isTransposed = true)
val interceptsR = Vectors.dense(-2.10320093, 0.3394473, 1.76375361)

model1.coefficientMatrix.colIter.foreach(v => assert(v.toArray.sum ~== 0.0 absTol eps))
model2.coefficientMatrix.colIter.foreach(v => assert(v.toArray.sum ~== 0.0 absTol eps))

assert(model1.coefficientMatrix ~== coefficientsR relTol 0.05)
assert(model1.coefficientMatrix.toArray.sum ~== 0.0 absTol eps)
assert(model1.interceptVector ~== interceptsR relTol 0.05)
Expand Down Expand Up @@ -1204,6 +1207,9 @@ class LogisticRegressionSuite
-0.3180040, 0.9679074, -0.2252219, -0.4319914,
0.2452411, -0.6046524, 0.1050710, 0.1180180), isTransposed = true)

model1.coefficientMatrix.colIter.foreach(v => assert(v.toArray.sum ~== 0.0 absTol eps))
model2.coefficientMatrix.colIter.foreach(v => assert(v.toArray.sum ~== 0.0 absTol eps))

assert(model1.coefficientMatrix ~== coefficientsR relTol 0.05)
assert(model1.coefficientMatrix.toArray.sum ~== 0.0 absTol eps)
assert(model1.interceptVector.toArray === Array.fill(3)(0.0))
Expand Down

0 comments on commit eb00378

Please sign in to comment.