Hi Weihua, Thanks for your reply!
Even when I use jarURI: "local:///path/to/job.jar" I can see that the classes loaded by the taskmanager are from something like: file:/tmp/tm_flink-app-wdqs-taskmanager-1-10/blobStorage/job_94ec61137dc8814f4558116848dd29cc/blob_p-0037843784c2e3ee289c9cc3ec644bc0df2c1084-39e459efea8e0348e5a3c635e1c947c2 when logging: aRandomJobObject.getClass().getProtectionDomain().getCodeSource().getLocation() This file does seem present on the object storage under the flink_ha_storage/flink-app-wdqs/blob/job_94ec61137dc8814f4558116848dd29cc/ si it's uploaded at some point. I can confirm that I see things like in taskmanager logs: Loading configuration property: pipeline.jars, local:///opt/flink/streaming-updater-producer.jar [...] Loading dynamic configuration property: pipeline.jars, file:/opt/flink/streaming-updater-producer.jar So I'm not sure what makes flink willing to use the H/A blob storage rather than the path provided in the image. Regarding jarURI being optional I found https://github.com/apache/flink-kubernetes-operator/pull/370 but I'm unsure if this is going to help my case. Thanks! On Thu, Apr 6, 2023 at 7:38 AM Weihua Hu <huweihua....@gmail.com> wrote: > Hi, David > > The jarURI is required[1], otherwise Flink doesn't know which jar should > be used. > > If you are using application mode, you can set jarURI to > "local:///opt/flink/usrlib/your-job.jar", and the jar will not upload to > H/A storage. > > Best, > Weihua > > > On Wed, Apr 5, 2023 at 5:51 PM David Causse <dcau...@wikimedia.org> wrote: > >> Hi, >> >> I'm trying to deploy a job (flink 1.16) with the flink-operator, the job >> jar is part of the image and placed under /opt/flink/usrlib. >> I thought that by placing the job jar there I could avoid setting the >> jarURI in the JobSpec but I'm getting a NPE (pasted at the end of this >> email) suggesting that this param is actually mandatory. >> Ultimately what I would like is to avoid pushing the job jar to the flink >> H/A storage since it's part of the image. >> I'm probably missing something regarding the use cases where jarURI can >> be omitted. >> >> Thanks! >> >> David. >> >> org.apache.flink.kubernetes.operator.exception.ReconciliationException: >> java.lang.NullPointerException >> at >> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:142) >> at >> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56) >> at >> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:145) >> at >> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:103) >> at >> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80) >> at >> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:102) >> at >> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) >> at >> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) >> at >> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) >> at >> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) >> at >> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> at java.base/java.lang.Thread.run(Thread.java:829) >> Caused by: java.lang.NullPointerException >> at >> org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:403) >> at >> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207) >> at >> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67) >> at >> org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(NativeFlinkService.java:66) >> at >> org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitApplicationCluster(AbstractFlinkService.java:184) >> at >> org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:182) >> at >> org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:59) >> at >> org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:115) >> at >> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:130) >> ... 13 more >> >
