By Ali Baba's beard and his forty bandits, Piotrek, this worked! My understanding was that I have to prevent Flink from loading the older compress.jar and force the newer one. One I shade-relocated org.apache.commons.compress for my project the problem went away
Many thanks! >-------- Оригинално писмо -------- >От: Piotr Nowojski pi...@data-artisans.com >Относно: Re: java.lang.NoSuchMethodError and dependencies problem >До: "r. r." <rob...@abv.bg> >Изпратено на: 20.10.2017 14:46 > But you said > > > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 > > To avoid this error that you describe I think that you have to ensure, that > no 1.14 commons-compress comes from your application, because it can conflict > with 1.4.1 used by flink cluster. > > By shading I meant that you could shade/relocate 1.14 usages in your > application, so that they don’t collide with Flink’s 1.4.1. > > Piotrek > > > On 19 Oct 2017, at 19:58, r. r. <rob...@abv.bg> wrote: > > > > Thanks, Piotr > > but my app code is self-contained in a fat-jar with maven-shade, so why > > would the class path affect this? > > > > by shade commons-compress do you mean : > > <!-- <exclude>org.apache.commons:commons-compress</exclude> --> > > it doesn't have effect either > > > > as a last resort i may try to rebuild Flink to use 1.14, but don't want to > > go there yet =/ > > > > > > Best regards > > > > > > > > > > > > > >> -------- Оригинално писмо -------- > > > >> От: Piotr Nowojski pi...@data-artisans.com > > > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > > > >> До: "r. r." <rob...@abv.bg> > > > >> Изпратено на: 19.10.2017 20:04 > > > > > > > > > >> I’m not 100% sure, so treat my answer with a grain of salt. > > > >> > > > >> I think when you start the cluster this way, dependencies (some? all?) are > >> being loaded to the class path before loading user’s application. At that > >> point, it doesn’t matter whether you have excluded commons-compress 1.4.1 > >> in yours application pom.xml. I’m not sure if this is solvable in some > >> way, or not. > > > >> > > > >> Maybe as a walk around, you could shade commons-compress usages in your > >> pom.xml? > > > >> > > > >> Piotr Nowojski > > > >> > > > >>> On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > > > >>> > > > >>> flink is started with bin/start-local.sh > > > >>> > > > >>> there is no classpath variable in the environment; > >>> flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it > >>> should be overridden by the dependencyManagement directive > > > >>> > > > >>> here is the stacktrace: > > > >>> > > > >>> The program finished with the following exception: > > > >>> > > > >>> org.apache.flink.client.program.ProgramInvocationException: The program > >>> execution failed: Job execution failed. > > > >>> at > >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) > > > >>> at > >>> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) > > > >>> at > >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) > > > >>> at > >>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) > > > >>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) > > > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >>> at > >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > >>> at > >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > >>> at java.lang.reflect.Method.invoke(Method.java:497) > > > >>> at > >>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > > > >>> at > >>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) > > > >>> at > >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) > > > >>> at > >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) > > > >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) > > > >>> at > >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) > > > >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) > > > >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) > > > >>> at > >>> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) > > > >>> at java.security.AccessController.doPrivileged(Native Method) > > > >>> at javax.security.auth.Subject.doAs(Subject.java:422) > > > >>> at > >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > > > >>> at > >>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) > > > >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) > > > >>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > >>> execution failed. > > > >>> at > >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) > > > >>> at > >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > > > >>> at > >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > > > >>> at > >>> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > > > >>> at > >>> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > > > >>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > > > >>> at > >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > > > >>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > > >>> at > >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > > >>> at > >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > > >>> at > >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > >>> Caused by: java.lang.NoSuchMethodError: > >>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; > > > >>> at > >>> org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) > > > >>> at > >>> org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) > > > >>> at > >>> org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) > > > >>> at > >>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) > > > >>> at > >>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) > > > >>> at > >>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) > > > >>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) > > > >>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) > > > >>> at > >>> org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) > > > >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) > > > >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) > > > >>> at > >>> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > > >>> at > >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) > > > >>> at > >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) > > > >>> at > >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) > > > >>> at > >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) > > > >>> at > >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) > > > >>> at > >>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) > > > >>> at > >>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) > > > >>> at > >>> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) > > > >>> at > >>> org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) > > > >>> at > >>> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) > > > >>> at > >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) > > > >>> at > >>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) > > > >>> at > >>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) > > > >>> at > >>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) > > > >>> at > >>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > > > >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > > > >>> at java.lang.Thread.run(Thread.java:745) > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>> -------- Оригинално писмо -------- > > > >>> > > > >>>> От: Piotr Nowojski pi...@data-artisans.com > > > >>> > > > >>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > > > >>> > > > >>>> До: "r. r." <rob...@abv.bg> > > > >>> > > > >>>> Изпратено на: 19.10.2017 18:00 > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>> Hi, > > > >>> > > > >>>> > > > >>> > > > >>>> What is the full stack trace of the error? > > > >>> > > > >>>> Are you sure that there is no commons-compresss somewhere in the > >>>> classpath (like in the lib directory)? How are you running your Flink > >>>> cluster? > > > >>> > > > >>>> > > > >>> > > > >>>> Piotrek > > > >>> > > > >>>> > > > >>> > > > >>>>> On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > > > >>> > > > >>>>> > > > >>> > > > >>>>> Hello > > > >>> > > > >>>>> I have a job that runs an Apache Tika pipeline and it fails with > >>>>> "Caused by: java.lang.NoSuchMethodError: > >>>>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > > > >>> > > > >>>>> > > > >>> > > > >>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. > > > >>> > > > >>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, > >>>>> so I force 1.14 with > > > >>> > > > >>>>> > > > >>> > > > >>>>> <dependencyManagement> > > > >>> > > > >>>>> <dependencies> > > > >>> > > > >>>>> <dependency> > > > >>> > > > >>>>> <groupId>org.apache.commons</groupId> > > > >>> > > > >>>>> <artifactId>commons-compress</artifactId> > > > >>> > > > >>>>> <version>1.14</version> > > > >>> > > > >>>>> </dependency> > > > >>> > > > >>>>> </dependencies> > > > >>> > > > >>>>> </dependencyManagement> > > > >>> > > > >>>>> > > > >>> > > > >>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 > >>>>> and after purge, the local maven repo also only contains 1.14 > > > >>> > > > >>>>> > > > >>> > > > >>>>> yet, after i deploy the job and it reads an Avro package from kafka and > >>>>> passes it to Tika, it fails with the error above, which leads me to > >>>>> think it somehow uses commons-compress at a version prior to 1.14, > >>>>> because method 'detect' is not present in older versions > > > >>> > > > >>>>> > > > >>> > > > >>>>> I excluded/included it from the fat-jar > > > >>> > > > >>>>> <exclude>org.apache.commons:commons-compress</exclude> > > > >>> > > > >>>>> still the same problem > > > >>> > > > >>>>> > > > >>> > > > >>>>> thanks for any hints! > > > >>> > > > >>>>> > > > >>> > > > >>>>>
