OK, the situation is the following: The test class (org.apache.flink.yarn.UtilsTest) implements a Hadoop interface (Container) that was extended in Hadoop 2.9.0 by a getter and setter. By adding the methods, we can compile Flink for Hadoop 2.9.0. However, the getter/setter add a dependency on a class that was also added in Hadoop 2.9.0. Therefore, the implementation is not backwards compatible with Hadoop versions < 2.9.0.
Not sure how we can fix the problem. We would need two version of the class that are chosen based on the Hadoop version. Do we have something like that somewhere else? Since this is only a problem in a test class, Flink 1.4.0 might still work very well with Hadoop 2.9.0. However, this has not been tested AFAIK. Cheers, Fabian 2017-11-29 12:47 GMT+01:00 Fabian Hueske <fhue...@gmail.com>: > I just tried to build the release-1.4 branch for Hadoop 2.9.0 (released a > few days ago) and got a compilation failure in a test class. > > Right now, I'm assessing how much we need to fix to support Hadoop 2.9.0. > I'll report later. > > Best, Fabian > > 2017-11-29 11:16 GMT+01:00 Aljoscha Krettek <aljos...@apache.org>: > >> Agreed, this is a regression compared to the previous functionality. I >> updated the issue to "Blocker". >> >> > On 29. Nov 2017, at 10:01, Gyula Fóra <gyula.f...@gmail.com> wrote: >> > >> > Hi all, >> > >> > I have found the following issue: >> > https://issues.apache.org/jira/browse/FLINK-8165 >> > >> > I would say this is a blocker (I personally pass the ParameterTool all >> over >> > the place in my production apps), but a pretty trivial issue to fix, we >> can >> > wait a little to find other potential problems. >> > >> > I can submit a fix in a little bit. >> > >> > Cheers, >> > Gyula >> > >> > Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: 2017. nov. >> > 29., Sze, 9:23): >> > >> >> +1 >> >> >> >> Verified: >> >> - No missing release Maven artifacts >> >> - Staged Apache source & binary convenience releases looks good >> >> - NOTICE / LICENSE is correct, README is sane >> >> - Built from source (macOS, Scala 2.11, Hadoop-free & Hadoop 2.8) >> >> - Cluster testing on AWS EMR (see release-testing-doc for configuration >> >> details) >> >> - Tested Kinesis / Elasticsearch connector (no dependency clashes on >> >> cluster execution, works locally in IDE) >> >> >> >> Thanks a lot for managing the release Aljoscha! >> >> >> >> Cheers, >> >> Gordon >> >> >> >> On 28 November 2017 at 8:32:42 PM, Stefan Richter ( >> >> s.rich...@data-artisans.com) wrote: >> >> >> >> +1 (non-binding) >> >> >> >> I tested Flink in a cluster setup on Google Cloud, YARN-per-job, >> checked >> >> that for all backends that HA, recovery, at-least-once, end-to-end >> exactly >> >> once (with Kafka11 Producer), savepoints, externalized checkpoints, and >> >> rescaling work correctly. >> >> >> >>> Am 28.11.2017 um 11:47 schrieb Aljoscha Krettek <aljos...@apache.org >> >: >> >>> >> >>> +1 >> >>> >> >>> Verified: >> >>> - NOTICE and LICENSE are correct >> >>> - source doesn't contain binaries >> >>> - verified signatures >> >>> - verified hashes >> >>> - cluster testing on AWS and Cloudera VM (with Kerberos) (see >> >> release-testing doc) >> >>> >> >>> >> >>>> On 28. Nov 2017, at 11:20, Aljoscha Krettek <aljos...@apache.org> >> >> wrote: >> >>>> >> >>>> Phew, thanks for the update! >> >>>> >> >>>>> On 28. Nov 2017, at 11:19, Gyula Fóra <gyf...@apache.org> wrote: >> >>>>> >> >>>>> Ok seems like I had to remove the snappy jar as it was corrupted >> (makes >> >>>>> total sense) :P >> >>>>> >> >>>>> Gyula Fóra <gyf...@apache.org> ezt írta (időpont: 2017. nov. 28., >> K, >> >> 11:13): >> >>>>> >> >>>>>> Hi Aljoscha, >> >>>>>> >> >>>>>> Thanks for the release candidate. I am having a hard time building >> >> the rc, >> >>>>>> I seem to get this error no matter what I do: >> >>>>>> >> >>>>>> [ERROR] Failed to execute goal >> >>>>>> org.apache.maven.plugins:maven-shade-plugin:2.4.1:shade >> >> (shade-hadoop) on >> >>>>>> project flink-shaded-hadoop2-uber: Error creating shaded jar: >> invalid >> >> LOC >> >>>>>> header (bad signature) -> [Help 1] >> >>>>>> >> >>>>>> (Apache Maven 3.3.9) >> >>>>>> >> >>>>>> Any idea what I am missing? >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Gyula >> >>>>>> >> >>>>>> >> >>>>>> Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: 2017. >> nov. >> >> 27., >> >>>>>> H, 19:35): >> >>>>>> >> >>>>>>> Hi everyone, >> >>>>>>> >> >>>>>>> Please review and vote on release candidate #2 for the version >> >> 1.4.0, as >> >>>>>>> follows: >> >>>>>>> [ ] +1, Approve the release >> >>>>>>> [ ] -1, Do not approve the release (please provide specific >> comments) >> >>>>>>> >> >>>>>>> >> >>>>>>> The complete staging area is available for your review, which >> >> includes: >> >>>>>>> * JIRA release notes [1], >> >>>>>>> * the official Apache source release and binary convenience >> releases >> >> to >> >>>>>>> be deployed to dist.apache.org[2], which are signed with the key >> >> with >> >>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3], >> >>>>>>> * all artifacts to be deployed to the Maven Central Repository >> [4], >> >>>>>>> * source code tag "release-1.4.0-rc1" [5], >> >>>>>>> * website pull request listing the new release [6]. >> >>>>>>> >> >>>>>>> Please have a careful look at the website PR because I changed >> some >> >>>>>>> wording and we're now also releasing a binary without Hadoop >> >> dependencies. >> >>>>>>> >> >>>>>>> Please use this document for coordinating testing efforts: [7] >> >>>>>>> >> >>>>>>> The only change between RC1 and this RC2 is that the source >> release >> >>>>>>> package does not include the erroneously included binary Ruby >> >> dependencies >> >>>>>>> of the documentation anymore. Because of this I would like to >> >> propose a >> >>>>>>> shorter voting time and close the vote around the time that RC1 >> >> would have >> >>>>>>> closed. This would mean closing by end of Wednesday. Please let me >> >> know if >> >>>>>>> you disagree with this. The vote is adopted by majority approval, >> >> with at >> >>>>>>> least 3 PMC affirmative votes. >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Your friendly Release Manager >> >>>>>>> >> >>>>>>> [1] >> >>>>>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje >> ctId=12315522&version=12340533 >> >>>>>>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc2/ >> >>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >> >>>>>>> [4] >> >>>>>>> >> >> https://repository.apache.org/content/repositories/orgapacheflink-1140 >> >>>>>>> [5] >> >>>>>>> >> >> https://git-wip-us.apache.org/repos/asf?p=flink.git;a=tag;h= >> ea751b7b23b23446ed3fcdeed564bbe8bf4adf9c >> >>>>>>> [6] https://github.com/apache/flink-web/pull/95 >> >>>>>>> [7] >> >>>>>>> >> >> https://docs.google.com/document/d/1HqYyrNoMSXwo8zBpZj7s39Uz >> UdlFcFO8TRpHNZ_cl44/edit?usp=sharing >> >>>>>>> >> >>>>>>> Pro-tip: you can create a settings.xml file with these contents: >> >>>>>>> >> >>>>>>> <settings> >> >>>>>>> <activeProfiles> >> >>>>>>> <activeProfile>flink-1.4.0</activeProfile> >> >>>>>>> </activeProfiles> >> >>>>>>> <profiles> >> >>>>>>> <profile> >> >>>>>>> <id>flink-1.4.0</id> >> >>>>>>> <repositories> >> >>>>>>> <repository> >> >>>>>>> <id>flink-1.4.0</id> >> >>>>>>> <url> >> >>>>>>> >> >>>>>>> >> >> https://repository.apache.org/content/repositories/orgapache >> flink-1140/ >> >>>>>>> </url> >> >>>>>>> </repository> >> >>>>>>> <repository> >> >>>>>>> <id>archetype</id> >> >>>>>>> <url> >> >>>>>>> >> >>>>>>> >> >> https://repository.apache.org/content/repositories/orgapache >> flink-1140/ >> >>>>>>> </url> >> >>>>>>> </repository> >> >>>>>>> </repositories> >> >>>>>>> </profile> >> >>>>>>> </profiles> >> >>>>>>> </settings> >> >>>>>>> >> >>>>>>> And reference that in you maven commands via --settings >> >>>>>>> path/to/settings.xml. This is useful for creating a quickstart >> based >> >> on the >> >>>>>>> staged release and for building against the staged jars. >> >>>>>> >> >>>>>> >> >>>> >> >>> >> >> >> >> >> >> >
