Hi Gordon, any news on this? On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote:
> This seems like a shading problem then. > I’ve tested this again with Maven 3.0.5, even without building against CDH > Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies. > > Let me investigate a bit and get back to this! > > Cheers, > Gordon > > > On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier (pomperma...@okkam.it) > wrote: > > On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local > repo I did: > > 1. git clone https://github.com/apache/flink.git && cd flink && git > checkout tags/release-1.2.1 > 2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install > -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 > -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos > 3. cd flink-dist > 4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install > -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 > -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos > 5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar > | grep MoreExecutors > > And I still see guava dependencies: > > org/apache/flink/hadoop/shaded/com/google/common/util/concur > rent/MoreExecutors$1.class > org/apache/flink/hadoop/shaded/com/google/common/util/concur > rent/MoreExecutors$SameThreadExecutorService.class > org/apache/flink/hadoop/shaded/com/google/common/util/concur > rent/MoreExecutors$ListeningDecorator.class > org/apache/flink/hadoop/shaded/com/google/common/util/concur > rent/MoreExecutors$ScheduledListeningDecorator.class > org/apache/flink/hadoop/shaded/com/google/common/util/concur > rent/MoreExecutors.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$1.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$2.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$3.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$4.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$Application$1.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$Application.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$DirectExecutor.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$DirectExecutorService.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$ListeningDecorator.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$ScheduledListeningDecorator$NeverSuccessfulListe > nableFutureTask.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors$ScheduledListeningDecorator.class > org/apache/flink/shaded/com/google/common/util/concurrent/Mo > reExecutors.class > com/google/common/util/concurrent/MoreExecutors$1.class > com/google/common/util/concurrent/MoreExecutors$SameThreadEx > ecutorService.class > com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class > com/google/common/util/concurrent/MoreExecutors$ScheduledLis > teningDecorator.class > com/google/common/util/concurrent/MoreExecutors.class > > It seems that it mix togheter guava 11 (probably coming from CDH > dependencies) and guava 18 classes. > > Also using *maven 3.0.5* lead to the same output :( > > Best, > Flavio > > On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Yes, those should not be in the flink-dist jar, so the root reason should >> be that the shading isn’t working properly for your custom build. >> >> If possible, could you try building Flink again with a lower Maven >> version as specified in the doc, and see if that works? >> If so, it could be that Maven 3.3.x simply isn’t shading properly even >> with the double compilation trick. >> >> >> On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pomperma...@okkam.it) >> wrote: >> >> What I did was to take the sources of the new ES connector and I took >> them into my code. >> Flink was compiled with maven 3.3+ but I did the double compilation as >> specified in the Flink build section. >> In flink dist I see guava classes, e.g.: >> >> com/google/common/util/concurrent/MoreExecutors$1.class >> com/google/common/util/concurrent/MoreExecutors$SameThreadEx >> ecutorService.class >> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class >> com/google/common/util/concurrent/MoreExecutors$ScheduledLis >> teningDecorator.class >> com/google/common/util/concurrent/MoreExecutors.class >> >> Is it a problem of the shading with Maven 3.3+? >> >> Best, >> Flavio >> >> On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES >>> connector). >>> >>> Another thing to check, if you built Flink yourself, make sure you’re >>> not using Maven 3.3+. There are shading problems when Flink is built with >>> Maven versions higher then that. >>> The flink-dist jar should not contain any non-shaded Guava dependencies, >>> could you also quickly check that? >>> >>> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pomperma...@okkam.it) >>> wrote: >>> >>> I shaded the Elasticsearch dependency [1] and now the job works. >>> So I cannot run a job that needs guava 18 on Flink 1.2.1... >>> >>> [1] https://www.elastic.co/blog/to-shade-or-not-to-shade >>> >>> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >>> > wrote: >>> >>>> Hi Flavio, >>>> >>>> Could there be another dependency in your job that requires a >>>> conflicting version (w.r.t. ES 2.4.1) of Guava? >>>> I’ve just double checked the flink-dist jar, there doesn’t seem to be >>>> any non-shaded Guava dependencies there, so the conflict should not have >>>> been caused by Flink. >>>> >>>> Cheers, >>>> Gordon >>>> >>>> >>>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pomperma...@okkam.it) >>>> wrote: >>>> >>>> Hi to all, >>>> I'm trying to use the new ES connector to index data from Flink (with >>>> ES 2.4.1). >>>> When I try to run it from Eclipse everything is ok, when I run it from >>>> the cluster I get the following exception: >>>> >>>> java.lang.NoSuchMethodError: com.google.common.util.concurr >>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; >>>> at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool. >>>> java:192) >>>> at org.elasticsearch.client.transport.TransportClient$Builder.b >>>> uild(TransportClient.java:131) >>>> >>>> In my fat jar there are the classes of guava 18 (ES requires that >>>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I >>>> think that there's guava 11 classes while in flink-hadoop-compatibility >>>> there are shade guava 18 dependencies. >>>> >>>> How can I make the job successfully run on the cluster? >>>> >>>> Best, >>>> Flavio >>>> >>>> >>> >
