Thanks for investigating this, Jared.  I would summarize it as
Flink-on-Mesos cannot be used in Hadoop-free mode in Flink 1.4.0.  I filed
an improvement bug to support this scenario: FLINK-8247



On Tue, Dec 12, 2017 at 11:46 AM, Jared Stehler <
jared.steh...@intellifylearning.com> wrote:

> I had been excluding all transitive dependencies from the lib dir; it
> seems to be working when I added the following deps:
>
>     <dependency>
>       <groupId>commons-configuration</groupId>
>       <artifactId>commons-configuration</artifactId>
>       <version>1.7</version>
>     </dependency>
>
>     <dependency>
>       <groupId>commons-lang</groupId>
>       <artifactId>commons-lang</artifactId>
>       <version>2.6</version>
>     </dependency>
>
>
> --
> Jared Stehler
> Chief Architect - Intellify Learning
> o: 617.701.6330 x703 <(617)%20701-6330>
>
>
>
> On Dec 12, 2017, at 2:10 PM, Chesnay Schepler <ches...@apache.org> wrote:
>
> Could you look into the flink-shaded-hadoop jar to check whether the
> missing class is actually contained?
>
> Where did the flink-shaded-hadoop jar come from? I'm asking because when
> building flink-dist from source the jar is called
> flink-shaded-hadoop2-uber-1.4.0.jar, which does indeed contain the jar.
> (the uber jar is created by building flink-shaded-hadoop2*-uber*)
>
> On 12.12.2017 19:28, Jared Stehler wrote:
>
> After upgrading to flink 1.4.0 using the hadoop-free build option, I’m
> seeing the following error on startup in the app master:
>
> 2017-12-12 18:23:15.473 [main] ERROR 
> o.a.f.m.r.clusterframework.MesosApplicationMasterRunner
> - Mesos JobManager initialization failed
>
> <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34208>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.security.UserGroupInformation
>
> <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34315>
> at org.apache.flink.runtime.clusterframework.overlays.
> HadoopUserOverlay$Builder.fromEnvironment(HadoopUserOverlay.java:74)
>
> <https://internal.dev3.master.us-west-2.prod.aws.intellify.io/singularity/task/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail/flink-appmaster-1513102980-1513102985422-1-10.80.53.70-us_west_2b/tail_of_finished_service.log?offset=34439>
> at org.apache.flink.mesos.entrypoint.MesosEntrypointUtils.applyOverlays(
> MesosEntrypointUtils.java:145)
> Looking at the code, it appears that the HadoopUserOverlay always tries to
> init the UserGroupInformation class, and is failing. Same error with or
> without the flink-shaded-hadoop2 library included.
>
> This is my lib dir:
>
> flink-appmaster-1.0-SNAPSHOT.jar   flink-s3-fs-presto-1.4.0.jar
> jul-to-slf4j-1.7.25.jar            sentry-1.5.3.jar
> flink-dist_2.11-1.4.0.jar          flink-shaded-hadoop2-1.4.0.jar
> log4j-over-slf4j-1.7.25.jar        sentry-logback-1.5.3.jar
> flink-metrics-prometheus-1.4.0.jar jackson-core-2.8.10.jar
> logback-classic-1.1.11.jar
> flink-python_2.11-1.4.0.jar        jcl-over-slf4j-1.7.25.jar
> logback-core-1.1.11.jar
>
>
> --
> Jared Stehler
> Chief Architect - Intellify Learning
> o: 617.701.6330 x703 <(617)%20701-6330>
>
>
>
>
>
>

Reply via email to