In your jar, are you submitting multiple jobs in parallel?
On 15.03.2019 10:05, Papadopoulos, Konstantinos wrote:
We had some progress since the job seems to be submitted and its
execution has been started, but, now, I am getting a
ProgramAbortException as follows:
05:01:01.788 [ERROR] SpringApplication – Application run failed
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException:
null
at
org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:54)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
com.iri.aa.etl.lmx.entity.service.AbstractLmxService.getCampaigns(AbstractLmxService.java:105)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
com.iri.aa.etl.lmx.entity.service.MediaSpentServiceImpl.transform(MediaSpentServiceImpl.java:25)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:58)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
com.iri.aa.etl.lmx.AbstractLmxJob.abstractExecute(AbstractLmxJob.java:27)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.execute(MediaSpentJob.java:38)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.lmx.MediaSpentJob.executeDry(MediaSpentJob.java:44)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
com.iri.aa.etl.job.JobExecutor.lambda$executeDryRunners$6(JobExecutor.java:44)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_201]
at
com.iri.aa.etl.job.JobExecutor.executeDryRunners(JobExecutor.java:44)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:35)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
~[d7378efa-919f-4bd3-a320-d74a5a76f80e_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_201]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_201]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
~[flink-dist_2.12-1.7.1.jar:1.7.1]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[?:1.8.0_201]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
*From:*Chesnay Schepler <ches...@apache.org>
*Sent:* Παρασκευή, 15 Μαρτίου 2019 10:56 πμ
*To:* Papadopoulos, Konstantinos
<konstantinos.papadopou...@iriworldwide.com>; user@flink.apache.org
*Subject:* Re: ProgramInvocationException when trying to submit a job
by running a jar using Monitoring REST API
Please separate your program arguments by a space instead of a comma
and try again.
On 15.03.2019 09:34, Papadopoulos, Konstantinos wrote:
Hi Chesnay,
Sorry for the misunderstanding. I get the following exception:
2019-03-15 04:31:26,826 ERROR
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler -
Exception occurred in REST handler.
org.apache.flink.runtime.rest.handler.RestHandlerException:
org.apache.flink.client.program.ProgramInvocationException: The
main method caused an error.
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$4(JarRunHandler.java:114)
at
java.util.concurrent.CompletableFuture.uniExceptionally(Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown
Source)
at
java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException:
org.apache.flink.client.program.ProgramInvocationException: The
main method caused an error.
at
org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:119)
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$7(JarRunHandler.java:151)
... 4 more
Caused by:
org.apache.flink.client.program.ProgramInvocationException: The
main method caused an error.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at
org.apache.flink.runtime.webmonitor.handlers.utils.JarHandlerUtils$JarHandlerContext.toJobGraph(JarHandlerUtils.java:117)
... 5 more
Caused by: java.lang.IllegalStateException: Failed to execute
ApplicationRunner
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807)
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:794)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:324)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
at
com.iri.aa.etl.EtlApplication.main(EtlApplication.java:21)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 10 more
Caused by: java.lang.IllegalArgumentException: Argument
'initialScopeId' is missing.
at
com.iri.aa.etl.job.JobArgumentsUtility.getArgument(JobArgumentsUtility.java:58)
at
com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getLmxArguments(JobArgumentsServiceImpl.java:86)
at
com.iri.aa.etl.lmx.job.JobArgumentsServiceImpl.getJobArguments(JobArgumentsServiceImpl.java:39)
at
com.iri.aa.etl.job.JobExecutor.run(JobExecutor.java:30)
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804)
... 20 more
*From:*Chesnay Schepler <ches...@apache.org>
<mailto:ches...@apache.org>
*Sent:* Παρασκευή, 15 Μαρτίου 2019 10:20 πμ
*To:* Papadopoulos, Konstantinos
<konstantinos.papadopou...@iriworldwide.com>
<mailto:konstantinos.papadopou...@iriworldwide.com>;
user@flink.apache.org <mailto:user@flink.apache.org>
*Subject:* Re: ProgramInvocationException when trying to submit a
job by running a jar using Monitoring REST API
Please provide the logged exception, I cannot help you otherwise.
On 14.03.2019 14:20, Papadopoulos, Konstantinos wrote:
It seems that Flink cluster does not retrieve program
arguments correctly. For reference, I sent the following request:
Method Type: POST
URL:
http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run
<https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdbtpa05p.ch3.dev.i.com%3A9171%2Fv1%2Fjars%2F321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar%2Frun&data=02%7C01%7C%7Cdf15b45eb7814617865208d6a9240306%7C43728c2044474b27ac2e4bdabb3c0121%7C0%7C0%7C636882369474729054&sdata=2i%2BnBpElTlCgf8SAyc13xtCY8%2FMk%2B2TNA3sLMzSJ5O4%3D&reserved=0>
Body: {"programArgs" :
"--job=mediaSpent,--initialScopeId=b494c35d-4c37-4338-8d23-0fc947bef690,--integratedScopeId=91769bd8-df4d-436c-b8d0-2e23ce862859,--projectId=333,--log.path=../log"}
Content-Type: application/json
*From:*Chesnay Schepler <ches...@apache.org>
<mailto:ches...@apache.org>
*Sent:* Πέμπτη, 14 Μαρτίου 2019 2:24 μμ
*To:* Papadopoulos, Konstantinos
<konstantinos.papadopou...@iriworldwide.com>
<mailto:konstantinos.papadopou...@iriworldwide.com>;
user@flink.apache.org <mailto:user@flink.apache.org>
*Subject:* Re: ProgramInvocationException when trying to
submit a job by running a jar using Monitoring REST API
Please enable debug logging, re-submit the job, check the
JobManager logs and report back the logged exception.
On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote:
Hi all,
As part of our projects, I experiment with Flink
Monitoring REST API and, especially, its capabilities of
uploading and running jar files.
When I am trying to submit one of our jobs by running a
jar previously uploaded via '/jars/upload', I am getting
an 500 Internal Server Error response with the following body:
{ "errors": [
"org.apache.flink.client.program.ProgramInvocationException:
The main method caused an error." ] }
On the other hand, when I am trying the same with the
‘Word Count’ batch example, execution succeeds.
It must be pointed out that I tried to execute the
respective POST request both via ‘curl’ and Postman, but
both failed with the same error.
Do anyone have idea why this may happen?
Thanks in advance,
Konstantinos
