Dear Apache Spark Community/Development Team, I hope this message finds you well.
I am writing to inquire about the roadmap and future plans for extending Spark ML support through Spark Connect to the Scala API in a manner analogous to SPARK-50812. Specifically, my team is very interested in leveraging Spark Connect to invoke ML pipelines from a Java-based service; however, the current implementation primarily targets Python. Background & Motivation: - SPARK-50812 introduced API support for Python ML via Spark Connect. - We would like to understand whether there are any ongoing efforts or planned JIRA tickets for a similar extension enabling Scala (and by extension Java) ML usage through Spark Connect. - Our use case is to embed ML pipelines directly within a Java service, taking advantage of Spark Connect’s client-server architecture without resorting to Python. My Questions: 1. Roadmap Status: Are there any existing proposals or tickets related to adding Scala/Java ML bindings in Spark Connect? 2. Timeline: If so, could you share an anticipated timeline or milestones for these enhancements? 3. Contribution Opportunities: Would the community welcome contributions toward this feature, and if so, are there any guidelines or specifications we should follow? References*:* - SPARK-50812: https://issues.apache.org/jira/browse/SPARK-50812 Understanding your timeline and priorities will greatly assist us in planning our integration strategy and, if feasible, contributing to the effort. Thank you for your time and for your ongoing work on Apache Spark. I look forward to your guidance. -- Daniel Filev Software Engineer Ontotext doing business as Graphwise | Making sense of data one triple at a time. https://graphwise.ai <http://graphwise.ai/> LinkedIn <https://www.linkedin.com/company/graphwise/> | X <https://x.com/graphwise>
