Thanks for investigating this, Jared. I would summarize it as Flink-on-Mesos cannot be used in Hadoop-free mode in Flink 1.4.0. I filed an improvement bug to support this scenario: FLINK-8247
On Tue, Dec 12, 2017 at 11:46 AM, Jared Stehler < jared.steh...@intellifylearning.com> wrote: > I had been excluding all transitive dependencies from the lib dir; it > seems to be working when I added the following deps: > > <dependency> > <groupId>commons-configuration</groupId> > <artifactId>commons-configuration</artifactId> > <version>1.7</version> > </dependency> > > <dependency> > <groupId>commons-lang</groupId> > <artifactId>commons-lang</artifactId> > <version>2.6</version> > </dependency> > > > -- > Jared Stehler > Chief Architect - Intellify Learning > o: 617.701.6330 x703 <(617)%20701-6330> > > > > On Dec 12, 2017, at 2:10 PM, Chesnay Schepler <ches...@apache.org> wrote: > > Could you look into the flink-shaded-hadoop jar to check whether the > missing class is actually contained? > > Where did the flink-shaded-hadoop jar come from? I'm asking because when > building flink-dist from source the jar is called > flink-shaded-hadoop2-uber-1.4.0.jar, which does indeed contain the jar. > (the uber jar is created by building flink-shaded-hadoop2*-uber*) > > On 12.12.2017 19:28, Jared Stehler wrote: > > After upgrading to flink 1.4.0 using the hadoop-free build option, I’m > seeing the following error on startup in the app master: > > 2017-12-12 18:23:15.473 [main] ERROR > o.a.f.m.r.clusterframework.MesosApplicationMasterRunner > - Mesos JobManager initialization failed > > <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34208> > java.lang.NoClassDefFoundError: Could not initialize class > org.apache.hadoop.security.UserGroupInformation > > <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34315> > at org.apache.flink.runtime.clusterframework.overlays. > HadoopUserOverlay$Builder.fromEnvironment(HadoopUserOverlay.java:74) > > <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34439> > at org.apache.flink.mesos.entrypoint.MesosEntrypointUtils.applyOverlays( > MesosEntrypointUtils.java:145) > Looking at the code, it appears that the HadoopUserOverlay always tries to > init the UserGroupInformation class, and is failing. Same error with or > without the flink-shaded-hadoop2 library included. > > This is my lib dir: > > flink-appmaster-1.0-SNAPSHOT.jar flink-s3-fs-presto-1.4.0.jar > jul-to-slf4j-1.7.25.jar sentry-1.5.3.jar > flink-dist_2.11-1.4.0.jar flink-shaded-hadoop2-1.4.0.jar > log4j-over-slf4j-1.7.25.jar sentry-logback-1.5.3.jar > flink-metrics-prometheus-1.4.0.jar jackson-core-2.8.10.jar > logback-classic-1.1.11.jar > flink-python_2.11-1.4.0.jar jcl-over-slf4j-1.7.25.jar > logback-core-1.1.11.jar > > > -- > Jared Stehler > Chief Architect - Intellify Learning > o: 617.701.6330 x703 <(617)%20701-6330> > > > > > >
