That’s good to hear :) Unfortunately at this moment dependencies can pollute class path in both ways (Flink’s can mess with user’s application and also the other way around).
Cheers, Piotrek > On 20 Oct 2017, at 15:11, r. r. <rob...@abv.bg> wrote: > > By Ali Baba's beard and his forty bandits, Piotrek, this worked! > My understanding was that I have to prevent Flink from loading the older > compress.jar and force the newer one. > One I shade-relocated org.apache.commons.compress for my project the problem > went away > > Many thanks! > > > > > > >> -------- Оригинално писмо -------- > >> От: Piotr Nowojski pi...@data-artisans.com > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >> До: "r. r." <rob...@abv.bg> > >> Изпратено на: 20.10.2017 14:46 > > > > >> But you said > >> > >>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 > >> > >> To avoid this error that you describe I think that you have to ensure, that >> no 1.14 commons-compress comes from your application, because it can >> conflict with 1.4.1 used by flink cluster. > >> > >> By shading I meant that you could shade/relocate 1.14 usages in your >> application, so that they don’t collide with Flink’s 1.4.1. > >> > >> Piotrek > >> > >>> On 19 Oct 2017, at 19:58, r. r. <rob...@abv.bg> wrote: > >>> > >>> Thanks, Piotr > >>> but my app code is self-contained in a fat-jar with maven-shade, so why >>> would the class path affect this? > >>> > >>> by shade commons-compress do you mean : > >>> <!-- <exclude>org.apache.commons:commons-compress</exclude> --> > >>> it doesn't have effect either > >>> > >>> as a last resort i may try to rebuild Flink to use 1.14, but don't want to >>> go there yet =/ > >>> > >>> > >>> Best regards > >>> > >>> > >>> > >>> > >>> > >>> > >>>> -------- Оригинално писмо -------- > >>> > >>>> От: Piotr Nowojski pi...@data-artisans.com > >>> > >>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >>> > >>>> До: "r. r." <rob...@abv.bg> > >>> > >>>> Изпратено на: 19.10.2017 20:04 > >>> > >>> > >>> > >>> > >>>> I’m not 100% sure, so treat my answer with a grain of salt. > >>> > >>>> > >>> > >>>> I think when you start the cluster this way, dependencies (some? all?) are >>>> being loaded to the class path before loading user’s application. At that >>>> point, it doesn’t matter whether you have excluded commons-compress 1.4.1 >>>> in yours application pom.xml. I’m not sure if this is solvable in some >>>> way, or not. > >>> > >>>> > >>> > >>>> Maybe as a walk around, you could shade commons-compress usages in your >>>> pom.xml? > >>> > >>>> > >>> > >>>> Piotr Nowojski > >>> > >>>> > >>> > >>>>> On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > >>> > >>>>> > >>> > >>>>> flink is started with bin/start-local.sh > >>> > >>>>> > >>> > >>>>> there is no classpath variable in the environment; >>>>> flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it >>>>> should be overridden by the dependencyManagement directive > >>> > >>>>> > >>> > >>>>> here is the stacktrace: > >>> > >>>>> > >>> > >>>>> The program finished with the following exception: > >>> > >>>>> > >>> > >>>>> org.apache.flink.client.program.ProgramInvocationException: The program >>>>> execution failed: Job execution failed. > >>> > >>>>> at >>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) > >>> > >>>>> at >>>>> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) > >>> > >>>>> at >>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) > >>> > >>>>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) > >>> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> > >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> > >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> > >>>>> at java.lang.reflect.Method.invoke(Method.java:497) > >>> > >>>>> at >>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > >>> > >>>>> at >>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) > >>> > >>>>> at >>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) > >>> > >>>>> at >>>>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) > >>> > >>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) > >>> > >>>>> at >>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) > >>> > >>>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) > >>> > >>>>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) > >>> > >>>>> at >>>>> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) > >>> > >>>>> at java.security.AccessController.doPrivileged(Native Method) > >>> > >>>>> at javax.security.auth.Subject.doAs(Subject.java:422) > >>> > >>>>> at >>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > >>> > >>>>> at >>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) > >>> > >>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) > >>> > >>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job >>>>> execution failed. > >>> > >>>>> at >>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) > >>> > >>>>> at >>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > >>> > >>>>> at >>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > >>> > >>>>> at >>>>> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > >>> > >>>>> at >>>>> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > >>> > >>>>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > >>> > >>>>> at >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > >>> > >>>>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > >>> > >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >>> > >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > >>> > >>>>> at >>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > >>> > >>>>> Caused by: java.lang.NoSuchMethodError: >>>>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; > >>> > >>>>> at >>>>> org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) > >>> > >>>>> at >>>>> org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) > >>> > >>>>> at >>>>> org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) > >>> > >>>>> at >>>>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) > >>> > >>>>> at >>>>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) > >>> > >>>>> at >>>>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) > >>> > >>>>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) > >>> > >>>>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) > >>> > >>>>> at >>>>> org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) > >>> > >>>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) > >>> > >>>>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > >>> > >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) > >>> > >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) > >>> > >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) > >>> > >>>>> at >>>>> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) > >>> > >>>>> at >>>>> org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) > >>> > >>>>> at >>>>> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) > >>> > >>>>> at >>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) > >>> > >>>>> at >>>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) > >>> > >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) > >>> > >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > >>> > >>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > >>> > >>>>> at java.lang.Thread.run(Thread.java:745) > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>>> -------- Оригинално писмо -------- > >>> > >>>>> > >>> > >>>>>> От: Piotr Nowojski pi...@data-artisans.com > >>> > >>>>> > >>> > >>>>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >>> > >>>>> > >>> > >>>>>> До: "r. r." <rob...@abv.bg> > >>> > >>>>> > >>> > >>>>>> Изпратено на: 19.10.2017 18:00 > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>>> Hi, > >>> > >>>>> > >>> > >>>>>> > >>> > >>>>> > >>> > >>>>>> What is the full stack trace of the error? > >>> > >>>>> > >>> > >>>>>> Are you sure that there is no commons-compresss somewhere in the >>>>>> classpath (like in the lib directory)? How are you running your Flink >>>>>> cluster? > >>> > >>>>> > >>> > >>>>>> > >>> > >>>>> > >>> > >>>>>> Piotrek > >>> > >>>>> > >>> > >>>>>> > >>> > >>>>> > >>> > >>>>>>> On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> Hello > >>> > >>>>> > >>> > >>>>>>> I have a job that runs an Apache Tika pipeline and it fails with >>>>>>> "Caused by: java.lang.NoSuchMethodError: >>>>>>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. > >>> > >>>>> > >>> > >>>>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, >>>>>>> so I force 1.14 with > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> <dependencyManagement> > >>> > >>>>> > >>> > >>>>>>> <dependencies> > >>> > >>>>> > >>> > >>>>>>> <dependency> > >>> > >>>>> > >>> > >>>>>>> <groupId>org.apache.commons</groupId> > >>> > >>>>> > >>> > >>>>>>> <artifactId>commons-compress</artifactId> > >>> > >>>>> > >>> > >>>>>>> <version>1.14</version> > >>> > >>>>> > >>> > >>>>>>> </dependency> > >>> > >>>>> > >>> > >>>>>>> </dependencies> > >>> > >>>>> > >>> > >>>>>>> </dependencyManagement> > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 >>>>>>> and after purge, the local maven repo also only contains 1.14 > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> yet, after i deploy the job and it reads an Avro package from kafka and >>>>>>> passes it to Tika, it fails with the error above, which leads me to >>>>>>> think it somehow uses commons-compress at a version prior to 1.14, >>>>>>> because method 'detect' is not present in older versions > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> I excluded/included it from the fat-jar > >>> > >>>>> > >>> > >>>>>>> <exclude>org.apache.commons:commons-compress</exclude> > >>> > >>>>> > >>> > >>>>>>> still the same problem > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>> thanks for any hints! > >>> > >>>>> > >>> > >>>>>>> > >>> > >>>>> > >>> > >>>>>>>
