You should find that 'surprisingly public' classes are there because of language technicalities. For example DummySerializerInstance is public because it's a Java class, and can't be used outside its package otherwise.
LIkewise I think MiMa just looks at bytecode, and private[spark] classes are public in the bytecode for similar reasons (although Scala enforces the access within Scala as expected). Hence it will flag changes to "nonpublic" private[spark] classes. I think things that are meant to be marked private are, well, marked private, or else as private as possible and flagged with annotations like @Private. (It does sound like DummySerializerInstance should be so annotated?) Yes, the catalyst package in its entirety is one big exception - private by fiat, not by painstaking flagging of every class. The issue to me is really docs. If we have java/scaladoc of private classes, and there's a way to avoid that like with annotations, that should be fixed. On Tue, Nov 13, 2018 at 6:26 PM Wenchen Fan <cloud0...@gmail.com> wrote: > > Hi all, > > Recently I updated the MiMa exclusion rules, and found MiMa tracks some > private classes/methods unexpectedly. > > Note that, "private" here means that, we have no guarantee about > compatibility. We don't provide documents and users need to take the risk > when using them. > > In the API document, it has some obvious private classes, e.g. > https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.serializer.DummySerializerInstance > , which is not expected either. > > I looked around and can't find a clear definition of "private" in Spark. > > AFAIK, we have several rules: > 1. everything which is really private that end users can't access, e.g. > package private classes, private methods, etc. > 2. classes under certain packages. I don't know if we have a list, the > catalyst package is considered as a private package. > 3. everything which has a @Private annotation. > > I'm sending this email to collect more feedback, and hope we can come up with > a clear definition about what is "private". > > Thanks, > Wenchen --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
