Hi all, I would like to discuss Java specific APIs and which design we will choose. This has been discussed in multiple places so far, for example, at https://github.com/apache/spark/pull/28085#discussion_r407334754
*The problem:* In short, I would like us to have clear guidance on how we support Java specific APIs when it requires to return a Java instance. The problem is simple: def requests: Map[String, ExecutorResourceRequest] = ... def requestsJMap: java.util.Map[String, ExecutorResourceRequest] = ... vs def requests: java.util.Map[String, ExecutorResourceRequest] = ... *Current codebase:* My understanding so far was that the latter is preferred and more consistent and prevailing in the existing codebase, for example, see StateOperatorProgress and StreamingQueryProgress in Structured Streaming. However, I realised that we also have other approaches in the current codebase. There look four approaches to deal with Java specifics in general: 1. Java specific classes such as JavaRDD and JavaSparkContext. 2. Java specific methods with the same name that overload its parameters, see functions.scala. 3. Java specific methods with a different name that needs to return a different type such as TaskContext.resourcesJMap vs TaskContext.resources. 4. One method that returns a Java instance for both Scala and Java sides. see StateOperatorProgress and StreamingQueryProgress. *Analysis on the current codebase:* I agree with 2. approach because the corresponding cases give you a consistent API usage across other language APIs in general. Approach 1. is from the old world when we didn't have unified APIs. This might be the worst approach. 3. and 4. are controversial. For 3., if you have to use Java APIs, then, you should search if there is a variant of that API every time specifically for Java APIs. But yes, it gives you Java/Scala friendly instances. For 4., having one API that returns a Java instance makes you able to use it in both Scala and Java APIs sides although it makes you call asScala in Scala side specifically. But you don’t have to search if there’s a variant of this API and it gives you a consistent API usage across languages. Also, note that calling Java in Scala is legitimate but the opposite case is not, up to my best knowledge. In addition, you should have a method that returns a Java instance for PySpark or SparkR to support. *Proposal:* I would like to have a general guidance on this that the Spark dev agrees upon: Do 4. approach. If not possible, do 3. Avoid 1 almost at all cost. Note that this isn't a hard requirement but *a general guidance*; therefore, the decision might be up to the specific context. For example, when there are some strong arguments to have a separate Java specific API, that’s fine. Of course, we won’t change the existing methods given Micheal’s rubric added before. I am talking about new methods in unreleased branches. Any concern or opinion on this?
