Hyukjin Kwon created SPARK-57957:
------------------------------------
Summary: Deflake GaussianMixtureSuite 'GMM support instance
weighting' on macOS by using well-posed data
Key: SPARK-57957
URL: https://issues.apache.org/jira/browse/SPARK-57957
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 4.3.0
Reporter: Hyukjin Kwon
The `GMM support instance weighting` test in `GaussianMixtureSuite` fits k=5
Gaussians on `KMeansSuite.generateKMeansData(50, 3, 5)`, which produces 5
clusters of identical points (zero within-cluster variance). With k=5 the
covariances are singular and the EM fit is ill-posed, so the weighted (uniform
weight) and unweighted fits converge to different component-collapse patterns.
On the macOS-26 CI runner (build_maven_java21_macos26) this is deterministic
and the test fails on the mixture weight comparison (e.g. 0.0197 vs 0.1047).
Reducing the instance weight (SPARK-37317 style) and increasing maxIter both
fail to help (verified on macOS-26); increasing maxIter makes a component
collapse further (7.6e-11 vs 0.116). Fix: run the invariant on well-posed data
- fit rDataset with k=2 (real-variance data already used stably by another test
in the suite). Test-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
