On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:
1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1 2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos 3. cd flink-dist 4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos 5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar | grep MoreExecutors And I still see guava dependencies: org/apache/flink/hadoop/shaded/com/google/common/util/ concurrent/MoreExecutors$1.class org/apache/flink/hadoop/shaded/com/google/common/util/ concurrent/MoreExecutors$SameThreadExecutorService.class org/apache/flink/hadoop/shaded/com/google/common/util/ concurrent/MoreExecutors$ListeningDecorator.class org/apache/flink/hadoop/shaded/com/google/common/util/ concurrent/MoreExecutors$ScheduledListeningDecorator.class org/apache/flink/hadoop/shaded/com/google/common/util/ concurrent/MoreExecutors.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$1.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$2.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$3.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$4.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$Application$1.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$Application.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors$DirectExecutor.class org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ DirectExecutorService.class org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ ListeningDecorator.class org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ ScheduledListeningDecorator$ListenableScheduledTask.class org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ ScheduledListeningDecorator.class org/apache/flink/shaded/com/google/common/util/concurrent/ MoreExecutors.class com/google/common/util/concurrent/MoreExecutors$1.class com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService. class com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator. class com/google/common/util/concurrent/MoreExecutors.class It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes. Also using *maven 3.0.5* lead to the same output :( Best, Flavio On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Yes, those should not be in the flink-dist jar, so the root reason should > be that the shading isn’t working properly for your custom build. > > If possible, could you try building Flink again with a lower Maven version > as specified in the doc, and see if that works? > If so, it could be that Maven 3.3.x simply isn’t shading properly even > with the double compilation trick. > > > On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pomperma...@okkam.it) > wrote: > > What I did was to take the sources of the new ES connector and I took them > into my code. > Flink was compiled with maven 3.3+ but I did the double compilation as > specified in the Flink build section. > In flink dist I see guava classes, e.g.: > > com/google/common/util/concurrent/MoreExecutors$1.class > com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService. > class > com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class > com/google/common/util/concurrent/MoreExecutors$ > ScheduledListeningDecorator.class > com/google/common/util/concurrent/MoreExecutors.class > > Is it a problem of the shading with Maven 3.3+? > > Best, > Flavio > > On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES >> connector). >> >> Another thing to check, if you built Flink yourself, make sure you’re not >> using Maven 3.3+. There are shading problems when Flink is built with Maven >> versions higher then that. >> The flink-dist jar should not contain any non-shaded Guava dependencies, >> could you also quickly check that? >> >> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pomperma...@okkam.it) >> wrote: >> >> I shaded the Elasticsearch dependency [1] and now the job works. >> So I cannot run a job that needs guava 18 on Flink 1.2.1... >> >> [1] https://www.elastic.co/blog/to-shade-or-not-to-shade >> >> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> Hi Flavio, >>> >>> Could there be another dependency in your job that requires a >>> conflicting version (w.r.t. ES 2.4.1) of Guava? >>> I’ve just double checked the flink-dist jar, there doesn’t seem to be >>> any non-shaded Guava dependencies there, so the conflict should not have >>> been caused by Flink. >>> >>> Cheers, >>> Gordon >>> >>> >>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pomperma...@okkam.it) >>> wrote: >>> >>> Hi to all, >>> I'm trying to use the new ES connector to index data from Flink (with ES >>> 2.4.1). >>> When I try to run it from Eclipse everything is ok, when I run it from >>> the cluster I get the following exception: >>> >>> java.lang.NoSuchMethodError: com.google.common.util.concurr >>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; >>> at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool. >>> java:192) >>> at org.elasticsearch.client.transport.TransportClient$Builder.b >>> uild(TransportClient.java:131) >>> >>> In my fat jar there are the classes of guava 18 (ES requires that >>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I >>> think that there's guava 11 classes while in flink-hadoop-compatibility >>> there are shade guava 18 dependencies. >>> >>> How can I make the job successfully run on the cluster? >>> >>> Best, >>> Flavio >>> >>> >>
