I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?
On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Flavio, > > I was just doing some end-to-end rebuild Flink + cluster execution with ES > sink tests, and it seems like the Guava shading problem isn’t there anymore > in the flink-dist jar. > > On the `release-1.3` branch, built with Maven 3.0.5, the Guava > dependencies in flink-dist are all properly shaded. > Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and > ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all > worked fine without the Guava conflict issue. > > So I’m pretty sure that if the problem still exists for you, the conflict > would have came from other dependencies in your code. > > Cheers, > Gordon > > > On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier (pomperma...@okkam.it) > wrote: > > Hi Gordon, > any news on this? > > On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> This seems like a shading problem then. >> I’ve tested this again with Maven 3.0.5, even without building against >> CDH Hadoop binaries the flink-dist jar contains non-shaded Guava >> dependencies. >> >> Let me investigate a bit and get back to this! >> >> Cheers, >> Gordon >> >> >> On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier (pomperma...@okkam.it) >> wrote: >> >> On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local >> repo I did: >> >> 1. git clone https://github.com/apache/flink.git && cd flink && git >> checkout tags/release-1.2.1 >> 2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install >> -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 >> -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos >> 3. cd flink-dist >> 4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install >> -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 >> -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos >> 5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar >> | grep MoreExecutors >> >> And I still see guava dependencies: >> >> org/apache/flink/hadoop/shaded/com/google/common/util/concur >> rent/MoreExecutors$1.class >> org/apache/flink/hadoop/shaded/com/google/common/util/concur >> rent/MoreExecutors$SameThreadExecutorService.class >> org/apache/flink/hadoop/shaded/com/google/common/util/concur >> rent/MoreExecutors$ListeningDecorator.class >> org/apache/flink/hadoop/shaded/com/google/common/util/concur >> rent/MoreExecutors$ScheduledListeningDecorator.class >> org/apache/flink/hadoop/shaded/com/google/common/util/concur >> rent/MoreExecutors.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$1.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$2.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$3.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$4.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$Application$1.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$Application.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$DirectExecutor.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$DirectExecutorService.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$ListeningDecorator.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$ScheduledListeningDecorator$NeverSuccessfulListe >> nableFutureTask.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors$ScheduledListeningDecorator.class >> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >> reExecutors.class >> com/google/common/util/concurrent/MoreExecutors$1.class >> com/google/common/util/concurrent/MoreExecutors$SameThreadEx >> ecutorService.class >> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class >> com/google/common/util/concurrent/MoreExecutors$ScheduledLis >> teningDecorator.class >> com/google/common/util/concurrent/MoreExecutors.class >> >> It seems that it mix togheter guava 11 (probably coming from CDH >> dependencies) and guava 18 classes. >> >> Also using *maven 3.0.5* lead to the same output :( >> >> Best, >> Flavio >> >> On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> Yes, those should not be in the flink-dist jar, so the root reason >>> should be that the shading isn’t working properly for your custom build. >>> >>> If possible, could you try building Flink again with a lower Maven >>> version as specified in the doc, and see if that works? >>> If so, it could be that Maven 3.3.x simply isn’t shading properly even >>> with the double compilation trick. >>> >>> >>> On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pomperma...@okkam.it) >>> wrote: >>> >>> What I did was to take the sources of the new ES connector and I took >>> them into my code. >>> Flink was compiled with maven 3.3+ but I did the double compilation as >>> specified in the Flink build section. >>> In flink dist I see guava classes, e.g.: >>> >>> com/google/common/util/concurrent/MoreExecutors$1.class >>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx >>> ecutorService.class >>> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class >>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis >>> teningDecorator.class >>> com/google/common/util/concurrent/MoreExecutors.class >>> >>> Is it a problem of the shading with Maven 3.3+? >>> >>> Best, >>> Flavio >>> >>> On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >>> > wrote: >>> >>>> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES >>>> connector). >>>> >>>> Another thing to check, if you built Flink yourself, make sure you’re >>>> not using Maven 3.3+. There are shading problems when Flink is built with >>>> Maven versions higher then that. >>>> The flink-dist jar should not contain any non-shaded Guava >>>> dependencies, could you also quickly check that? >>>> >>>> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pomperma...@okkam.it) >>>> wrote: >>>> >>>> I shaded the Elasticsearch dependency [1] and now the job works. >>>> So I cannot run a job that needs guava 18 on Flink 1.2.1... >>>> >>>> [1] https://www.elastic.co/blog/to-shade-or-not-to-shade >>>> >>>> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai < >>>> tzuli...@apache.org> wrote: >>>> >>>>> Hi Flavio, >>>>> >>>>> Could there be another dependency in your job that requires a >>>>> conflicting version (w.r.t. ES 2.4.1) of Guava? >>>>> I’ve just double checked the flink-dist jar, there doesn’t seem to be >>>>> any non-shaded Guava dependencies there, so the conflict should not have >>>>> been caused by Flink. >>>>> >>>>> Cheers, >>>>> Gordon >>>>> >>>>> >>>>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pomperma...@okkam.it) >>>>> wrote: >>>>> >>>>> Hi to all, >>>>> I'm trying to use the new ES connector to index data from Flink (with >>>>> ES 2.4.1). >>>>> When I try to run it from Eclipse everything is ok, when I run it from >>>>> the cluster I get the following exception: >>>>> >>>>> java.lang.NoSuchMethodError: com.google.common.util.concurr >>>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; >>>>> at org.elasticsearch.threadpool.ThreadPool.<clinit>( >>>>> ThreadPool.java:192) >>>>> at org.elasticsearch.client.transport.TransportClient$ >>>>> Builder.build(TransportClient.java:131) >>>>> >>>>> In my fat jar there are the classes of guava 18 (ES requires that >>>>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I >>>>> think that there's guava 11 classes while in flink-hadoop-compatibility >>>>> there are shade guava 18 dependencies. >>>>> >>>>> How can I make the job successfully run on the cluster? >>>>> >>>>> Best, >>>>> Flavio >>>>> >>>>> >>>> >> >
