Forgot to link the ticket that removed the global ScalaReflectionLock: https://issues.apache.org/jira/browse/SPARK-19810
Best Regards, Ryan On Fri, Mar 15, 2019 at 10:40 AM Shixiong(Ryan) Zhu <shixi...@databricks.com> wrote: > Hey Sean, > > Sounds good to me. At least, it's not worse than any versions prior to > 2.3.0 which has a global ScalaReflectionLock. In addition, if someone hits > a performance regression caused by this, they probably are creating too > many Encoders. Reusing Encoders is a better solution for this case. > > Best regards, > Shixiong > > On Thu, Mar 14, 2019 at 2:25 PM Sean Owen <sro...@gmail.com> wrote: > >> This is worth a look: https://github.com/apache/spark/pull/24085 >> >> Scala has a surprising thread-safety bug in the "<:<" operator that's >> used to check subtypes, which can lead to incorrect results in >> non-trivial situations. >> >> The fix on the table is to introduce a global lock to protect a lot of >> the Scala-related reflection code to resolve it. This may be the best >> we can do, but 'global lock' sounds ominous. >> >> Any thoughts on other ways to resolve it? I'm not sure there are. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
