https://issues.apache.org/jira/browse/FLINK-2940
There is now a pending pull request: https://github.com/apache/flink/pull/1529 As I was working on the changes, I discovered we have some more modules which have a Scala dependency which could be avoided. Namely these are "flink-java8", and "flink-streaming-java". The latter probably affects a lot of users. So it would be nice to make it Scala-free. Do you think that would be feasible? Cheers, Max On Mon, Nov 2, 2015 at 9:01 PM, Fabian Hueske <fhue...@gmail.com> wrote: > +1 for that. > > 2015-11-02 20:52 GMT+01:00 Stephan Ewen <se...@apache.org>: > >> +1 for Max' suggestion to fix that for 1.0 (next release). >> >> Hot fixing of this thing so short before a release is a bit risky in my >> opinion. It is easy to make errors (overlooking something, error not >> visible because of cached older dependencies, ...), it happened more than >> once during version upgrades, maven project re-organizations, etc. >> >> Doing it after 0.10 and having a few week to let it sink in and errors >> surface would probably be much safer... >> >> On Mon, Nov 2, 2015 at 5:19 AM, Fabian Hueske <fhue...@gmail.com> wrote: >> >> > Ah OK. Sorry, I misunderstood your intention. >> > >> > 2015-11-02 14:07 GMT+01:00 Maximilian Michels <m...@apache.org>: >> > >> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11" >> > artifacts >> > > > (and others that depend on flink-java and have no other Scala >> > dependency) >> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release. >> > > >> > > >> > > My suggestion was to keep the Scala unsuffixed Scala 2.10 and add a >> > > suffix for Scala 2.11. That's the way we are currently doing it (also >> > > deployed on Maven like this). For the next release after 0.10, we can >> > > do it properly. >> > > >> > > On Mon, Nov 2, 2015 at 1:30 PM, Chiwan Park <chiwanp...@apache.org> >> > wrote: >> > > > If we choose selective Scala version suffix for artifacts, we have to >> > > tell which artifacts have the version suffix to newcomers. Some >> artifacts >> > > such as "flink-java”, "flink-streaming-java" are easily recognized. But >> > > IMO, knowing whether artifacts such as "flink-ml", "flink-clients", >> > > "flink-table" have the version suffix or not is difficult for >> newcomers. >> > > > >> > > > This is why we are adding the version suffix to all Scala 2.11 >> > artifacts >> > > currently. For Scala 2.10 artifacts, we aren’t adding the version >> suffix >> > > for Flink with Java users. >> > > > >> > > > I’m for adding the version suffix to Scala 2.10 artifacts also. But >> I’m >> > > not sure that removing the version suffix from Java-only artifacts >> would >> > be >> > > good. As I said above, It seems difficult for newcomers. >> > > > >> > > > Regards, >> > > > Chiwan Park >> > > > >> > > > On November 2, 2015 at 8:19:15 PM, Fabian Hueske (fhue...@gmail.com) >> > > wrote: >> > > > >> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11" >> > artifacts >> > > > (and others that depend on flink-java and have no other Scala >> > dependency) >> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release. >> > > > >> > > > Do we want that? >> > > > >> > > > 2015-11-02 11:37 GMT+01:00 Maximilian Michels <m...@apache.org>: >> > > > >> > > >> I'm for leaving it as-is and renaming all artifacts which depend on >> > > >> Scala for the release following 0.10. >> > > >> >> > > >> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhue...@gmail.com> >> > > wrote: >> > > >> > OK, let me try to summarize the discussion (and please correct me >> > if I >> > > >> got >> > > >> > something wrong). >> > > >> > >> > > >> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have >> > to >> > > >> > release 2.11 artifacts for the 0.10.0 release version as well. >> > > >> > >> > > >> > 2) Everybody agrees to appropriately tag all artifacts that have a >> > > >> > (transitive) Scala dependency. ATM, that would also include >> > flink-java >> > > >> > which is a bit awkward. The Scala dependency in flink-java >> > originates >> > > >> from >> > > >> > the Chill library which is used to obtain a Kryo serializer which >> is >> > > >> > initialized with serializers for Scala classes. We could resolve >> > this >> > > >> issue >> > > >> > by providing Java and Scala specific implementations of the Kryo >> > > >> > serializers and have KryoTypeInfos for Java and Scala. >> > > >> > >> > > >> > The question to answer right now is, do we want to have >> "correctly" >> > > >> labeled >> > > >> > artifacts for the next 0.10.0 release or do we defer that for 1.0? >> > > >> > If we want to solve it for 0.10.0 we need to cancel the current RC >> > and >> > > >> > provide a fix to remove the Scala dependency in flink-java, IMO. >> > > >> > >> > > >> > Opinions? >> > > >> > >> > > >> > Cheers, Fabian >> > > >> > >> > > >> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <se...@apache.org>: >> > > >> > >> > > >> >> +1 for the approach discusses here, and for removing Scala >> > > dependencies >> > > >> >> from modules that can be Scala independent. >> > > >> >> >> > > >> >> It would be nice if pure Java users would not see any Scala >> > > versioning >> > > >> (on >> > > >> >> flink-core, flink-java, later also flink-sreaming-java). I guess >> > for >> > > any >> > > >> >> runtime-related parts (including flink-client and currently all >> > > >> streaming >> > > >> >> projects), we need the Scala versions... >> > > >> >> >> > > >> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels < >> m...@apache.org >> > > >> > > >> wrote: >> > > >> >> >> > > >> >> > Good point. Didn't know that. We can still add them for the >> > > release. >> > > >> >> > >> > > >> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov >> > > >> >> > <alexander.s.alexand...@gmail.com> wrote: >> > > >> >> > > My two cents - there are already Maven artifacts deployed for >> > > 2.11 >> > > >> in >> > > >> >> the >> > > >> >> > > SNAPSHOT repository. I think it might be confusing if they >> > > suddenly >> > > >> >> > > disappear for the stable release. >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels < >> m...@apache.org >> > >: >> > > >> >> > > >> > > >> >> > >> Seems like we agree that we need artifacts for different >> > > versions >> > > >> of >> > > >> >> > Scala >> > > >> >> > >> on Maven. There also seems to be a preference for including >> > the >> > > >> >> version >> > > >> >> > in >> > > >> >> > >> the artifact name. >> > > >> >> > >> >> > > >> >> > >> I've created an issue and marked it to be resolved for 1.0. >> > For >> > > the >> > > >> >> 0.10 >> > > >> >> > >> release, we will have binaries but no Maven artifacts. The >> > > biggest >> > > >> >> > >> challenge I see is to remove Scala from as many modules as >> > > >> possible. >> > > >> >> For >> > > >> >> > >> example, flink-java depends on Scala at the moment.. >> > > >> >> > >> >> > > >> >> > >> https://issues.apache.org/jira/browse/FLINK-2940 >> > > >> >> > >> >> > > >> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz IV < >> > > >> >> > fka...@redhat.com> >> > > >> >> > >> wrote: >> > > >> >> > >> >> > > >> >> > >> > No idea if I get a vote ;) Nevertheless, +1 to have >> binaries >> > > for >> > > >> >> both >> > > >> >> > >> > versions in Maven and explicitly "scala versioned". >> > > >> >> > >> > >> > > >> >> > >> > Some background on this for those not as familiar with >> scala >> > > >> >> > versioning: >> > > >> >> > >> > >> > > >> >> > >> > It's considered best practice to label what version of >> > scala a >> > > >> >> library >> > > >> >> > >> > uses in the artifact id. >> > > >> >> > >> > >> > > >> >> > >> > The reason is compiled scala code is only compatible with >> > the >> > > >> major >> > > >> >> > >> > version of scala it was compiled for. For example, a >> library >> > > >> >> > compatible >> > > >> >> > >> > with 2.10 is not compatible with 2.11. The same will be >> true >> > > with >> > > >> >> 2.12 >> > > >> >> > >> once >> > > >> >> > >> > it is released. Mixing versions will result in undefined >> > > behavior >> > > >> >> > which >> > > >> >> > >> > will likely manifest itself as runtime exceptions. >> > > >> >> > >> > >> > > >> >> > >> > The convention to fix this problem is for all published >> > > >> libraries to >> > > >> >> > >> > specify the version of scala they are compatible with. >> > Leaving >> > > >> out >> > > >> >> the >> > > >> >> > >> > scala version in a library is akin to saying "We don't >> > depend >> > > on >> > > >> >> scala >> > > >> >> > >> for >> > > >> >> > >> > this library, so feel free to use whatever you want." Sbt >> > > users >> > > >> will >> > > >> >> > >> > typically specify the version of scala they use and >> tooling >> > is >> > > >> built >> > > >> >> > >> around >> > > >> >> > >> > ensuring consistency with the "%%" operator. >> > > >> >> > >> > >> > > >> >> > >> > E.g. >> > > >> >> > >> > >> > > >> >> > >> > scalaVersion := "2.11.4" >> > > >> >> > >> > >> > > >> >> > >> > // this resolves to to artifactID: "scalacheck_2.11" >> > > >> >> > >> > libraryDependencies += "org.scalacheck" %% "scalacheck" % >> > > >> "1.12.0" % >> > > >> >> > >> "test" >> > > >> >> > >> > >> > > >> >> > >> > The most important part of this is that the scala version >> is >> > > >> >> explicit >> > > >> >> > >> > which eliminates the problem for downstream users. >> > > >> >> > >> > >> > > >> >> > >> > Cheers, >> > > >> >> > >> > Frederick >> > > >> >> > >> > >> > > >> >> > >> > >> > > >> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote: >> > > >> >> > >> > >> > > >> >> > >> >> +1 to have binaries for both versions in Maven and as >> build >> > > to >> > > >> >> > download. >> > > >> >> > >> >> >> > > >> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis < >> > > >> >> > >> >> theodoros.vasilou...@gmail.com>: >> > > >> >> > >> >> >> > > >> >> > >> >> +1 for having binaries, I'm working on a Spark >> application >> > > >> >> currently >> > > >> >> > >> with >> > > >> >> > >> >>> Scala 2.11 and having to rebuild everything when >> deploying >> > > >> e.g. to >> > > >> >> > EC2 >> > > >> >> > >> >>> is a >> > > >> >> > >> >>> pain. >> > > >> >> > >> >>> >> > > >> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk Celebi < >> > > u...@apache.org> >> > > >> >> > wrote: >> > > >> >> > >> >>> >> > > >> >> > >> >>> I agree with Till, but is this something you want to >> > > address in >> > > >> >> this >> > > >> >> > >> >>>> release already? >> > > >> >> > >> >>>> >> > > >> >> > >> >>>> I would postpone it to 1.0.0. >> > > >> >> > >> >>>> >> > > >> >> > >> >>>> – Ufuk >> > > >> >> > >> >>>> >> > > >> >> > >> >>>> On 26 Oct 2015, at 16:17, Till Rohrmann < >> > > trohrm...@apache.org >> > > >> > >> > > >> >> > wrote: >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>>> I would be in favor of deploying also Scala 2.11 >> > > artifacts to >> > > >> >> > Maven >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> since >> > > >> >> > >> >>> >> > > >> >> > >> >>>> more and more people will try out Flink with Scala >> 2.11. >> > > >> Having >> > > >> >> the >> > > >> >> > >> >>>>> dependencies in the Maven repository makes it >> > considerably >> > > >> >> easier >> > > >> >> > for >> > > >> >> > >> >>>>> people to get their Flink jobs running. >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>>> Furthermore, I observed that people are not aware that >> > our >> > > >> >> > deployed >> > > >> >> > >> >>>>> artifacts, e.g. flink-runtime, are built with Scala >> > 2.10. >> > > As >> > > >> a >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> consequence, >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> they mix flink dependencies with other dependencies >> > > pulling >> > > >> in >> > > >> >> > Scala >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> 2.11 >> > > >> >> > >> >>> >> > > >> >> > >> >>>> and then they wonder that the program crashes. It would >> > be, >> > > >> imho, >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> clearer >> > > >> >> > >> >>> >> > > >> >> > >> >>>> if all our dependencies which depend on a specific >> Scala >> > > >> version >> > > >> >> > would >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> have >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> the corresponding Scala suffix appended. >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>>> Adding the 2.10 suffix would also spare us the hassle >> of >> > > >> >> upgrading >> > > >> >> > >> to a >> > > >> >> > >> >>>>> newer Scala version in the future, because then the >> > > artifacts >> > > >> >> > >> wouldn't >> > > >> >> > >> >>>>> share the same artifact name. >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>>> Cheers, >> > > >> >> > >> >>>>> Till >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>>> On Mon, Oct 26, 2015 at 4:04 PM, Maximilian Michels < >> > > >> >> > m...@apache.org> >> > > >> >> > >> >>>>> >> > > >> >> > >> >>>> wrote: >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> Hi Flinksters, >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> We have recently committed an easy way to change >> > Flink's >> > > >> Scala >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> version. >> > > >> >> > >> >>> >> > > >> >> > >> >>>> The >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> question arises now whether we should ship Scala 2.11 >> as >> > > >> >> binaries >> > > >> >> > and >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> via >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> Maven. For the rc0, I created all binaries twice, for >> > > Scala >> > > >> 2.10 >> > > >> >> > and >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> 2.11. >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> However, I didn't create Maven artifacts. This follows >> > our >> > > >> >> current >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> shipping >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> strategy where we only ship Hadoop1 and Hadoop 2.3.0 >> > Maven >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> dependencies >> > > >> >> > >> >>> >> > > >> >> > >> >>>> but >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> additionally Hadoop 2.4, 2.6, 2.7 as binaries. >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> Should we also upload Maven dependencies for Scala >> > 2.11? >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> If so, the next question arises: What version pattern >> > > >> should we >> > > >> >> > have >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> for >> > > >> >> > >> >>> >> > > >> >> > >> >>>> the Flink Scala 2.11 dependencies? For Hadoop, we >> append >> > > >> -hadoop1 >> > > >> >> > to >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> the >> > > >> >> > >> >>> >> > > >> >> > >> >>>> VERSION, e.g. artifactID=flink-core, >> > version=0.9.1-hadoop1. >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> However, it is common practice to append the suffix >> to >> > > the >> > > >> >> > >> artifactID >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> of >> > > >> >> > >> >>> >> > > >> >> > >> >>>> the Maven dependency, e.g. artifactID=flink-core_2.11, >> > > >> >> > version=0.9.1. >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> This >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> has mostly historic reasons but is widely used. >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> Whatever naming pattern we choose, it should be >> > > consistent. >> > > >> I >> > > >> >> > would >> > > >> >> > >> be >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>> in >> > > >> >> > >> >>>> >> > > >> >> > >> >>>>> favor of changing our artifact names to contain the >> > Hadoop >> > > >> and >> > > >> >> > Scala >> > > >> >> > >> >>>>>> version. This would also imply that all Scala >> dependent >> > > >> Maven >> > > >> >> > >> modules >> > > >> >> > >> >>>>>> receive a Scala suffix (also the default Scala 2.10 >> > > >> modules). >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> Cheers, >> > > >> >> > >> >>>>>> Max >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>>>> >> > > >> >> > >> >>>> >> > > >> >> > >> > >> > > >> >> > >> >> > > >> >> > >> > > >> >> >> > > >> >> > > >> > >>
