[
https://issues.apache.org/jira/browse/SPARK-57957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-57957:
------------------------------------
Assignee: Hyukjin Kwon
> Deflake GaussianMixtureSuite 'GMM support instance weighting' on macOS by
> using well-posed data
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-57957
> URL: https://issues.apache.org/jira/browse/SPARK-57957
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The `GMM support instance weighting` test in `GaussianMixtureSuite` fits k=5
> Gaussians on `KMeansSuite.generateKMeansData(50, 3, 5)`, which produces 5
> clusters of identical points (zero within-cluster variance). With k=5 the
> covariances are singular and the EM fit is ill-posed, so the weighted
> (uniform weight) and unweighted fits converge to different component-collapse
> patterns. On the macOS-26 CI runner (build_maven_java21_macos26) this is
> deterministic and the test fails on the mixture weight comparison (e.g.
> 0.0197 vs 0.1047). Reducing the instance weight (SPARK-37317 style) and
> increasing maxIter both fail to help (verified on macOS-26); increasing
> maxIter makes a component collapse further (7.6e-11 vs 0.116). Fix: run the
> invariant on well-posed data - fit rDataset with k=2 (real-variance data
> already used stably by another test in the suite). Test-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
