Hi Patrick, You need to make sure, the pre-packaged Scala runtime is removed from the Flink docker image:
> rm flink-dist/lib/flink-scala* This is also mentioned in the Flink blog-post Martijn mentioned earlier. The smallest reproducible example would be required, if you want us to find the root cause of the issue. Best regards, Alexey On Wed, Jan 10, 2024 at 3:06 PM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Patrick, > > Ideally it would be great if you could provide a small reproducer. It > could be that something isn't working properly in hiding the > internally used version of Scala from users, or something else. > Without a reproducer, it's quite hard to debug. > > Best regards, > > Martijn > > On Wed, Jan 10, 2024 at 2:49 PM Eifler, Patrick <patrick.eif...@sony.com> > wrote: > > > > Hi Martijn, > > > > > > > > Many thanks for your reply. Yes I have seen the examples. I removed the > flink-scala dependency and only use the java libs for everything. So there > should no flink-scala API references in the stack. > > > > > > > > These are the flink dependencies we are using: > > > > > > > > "org.apache.flink" % "flink-core" % flinkVersion, > > "org.apache.flink" % "flink-streaming-java" % flinkVersion, > > "org.apache.flink" % "flink-table-api-java" % flinkVersion, > > "org.apache.flink" % "flink-table-api-java-bridge" % flinkVersion, > > "org.apache.flink" % "flink-table-runtime" % flinkVersion, > > "org.apache.flink" % "flink-clients" % flinkVersion, > > "org.apache.flink" % "flink-connector-base" % flinkVersion, > > "org.apache.flink" % "flink-table-planner-loader" % flinkVersion, > > "org.apache.flink" % "flink-connector-kafka" % flinkVersion, > > "org.apache.flink" % "flink-statebackend-rocksdb" % flinkVersion % > Provided, > > "org.apache.flink" % "flink-avro" % flinkVersion, > > "org.apache.flink" % "flink-avro-confluent-registry" % flinkVersion, > > > > > > > > And the code path where it fails is not directly related to Flink: (It > does not fail on the unit tests). > > > > > > > > Map( > > PsNowTag -> new OutputTag[T](PsNowTag, typeInformation), > > SCRNTag -> new OutputTag[T](SCRNTag, typeInformation), > > GMSTag -> new OutputTag[T](GMSTag, typeInformation), > > MuleTag -> new OutputTag[T](MuleTag, typeInformation), > > FutureTag -> new OutputTag[T](FutureTag, typeInformation), > > ) > > > > > > > > The failure looks like an incompatibility issue Scala runtime. Scala > 2.12 vs. Scala 2.13 collections are not compatible: > > > > > > > > java.lang.NoSuchMethodError: 'scala.collection.immutable.ArraySeq > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])' > > > > > > > > So this issue seems weird and does look like Flink is using the Scala > 2.12 runtime even if the flink-scala packages are not installed. The > question is why? > > > > Then I would wonder if it is possible to completely exclude Scala from > the Flink docker image. > > > > > > > > Best, > > > > > > > > Patrick > > > > -- > > > > Patrick Eifler > > > > Staff Software Engineer > > > > FTG Data Engineering > > > > Sony Interactive Entertainment > > > > Kemperplatz 1 > > > > 10785 Berlin > > > > > > > > From: Martijn Visser <martijnvis...@apache.org> > > Date: Tuesday, 9. January 2024 at 16:41 > > To: Eifler, Patrick <patrick.eif...@sony.com> > > Cc: user@flink.apache.org <user@flink.apache.org> > > Subject: Re: Flink 1.17 with Scala 2.13 or Scala 3 > > > > Hi Patrick, > > > > You're on the right track, because you can't use any of the Flink > > Scala APIs in order to use any arbitrary Scala version. Have you seen > > the examples with Scala 3? [2] Do you have an example of your > > code/setup? > > > > Best regards, > > > > Martijn > > > > [1] https://flink.apache.org/2022/02/22/scala-free-in-one-fifteen/ > > [2] https://github.com/sjwiesman/flink-scala-3 > > > > On Tue, Jan 9, 2024 at 4:16 PM Eifler, Patrick <patrick.eif...@sony.com> > wrote: > > > > > > Hi, > > > > > > > > > > > > The Flink image still has Scala 2.12 dependencies. I tried to run a > flink job written in Scala 2.13 avoiding the usage of all flink-scala api’s > but getting an incompatibility issue (regarding scala.collections) that I > would normally expect in a project were Scala 2.13 and Scala 2.12 are > running alongside. > > > > > > So I wonder what am I missing, as in the docs and jira tickets > everyone says it works now with any Scala Version. > > > Any pointers are appreciated, Thanks. > > > > > > > > > > > > Best, > > > > > > -- > > > > > > Patrick Eifler >
