Hi Dipak, Regarding question 1, I noticed from the logs that the method createBatchExecutionEnvironment from Beam is being used in your job. IIUC, this method utilizes Flink's DataSet API. If indeed the DataSet API is being used, the configuration option execution.batch-shuffle-mode will not take effect, and you should set the ExecutionMode to BATCH_FORCED, in Beam you can configure executionModeForBatch to BATCH_FORCED. And this limitation has been documented in the latest Flink version 1.19 documentation: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/#limitations-2 .
Best, Junrui Dipak Tandel <dipak.tande...@gmail.com> 于2024年3月29日周五 16:05写道： > Hi Everyone > > I am facing some issues while running the batch job on a Flink cluster > using Adaptive Batch Scheduler. I have deployed a flink cluster on > Kubernetes using the flink Kubernetes operator and submitted a job to the > cluster using Apache beam FlinkRunner. I am using Flink version 1.16. I > want to figure out two things. > > > > *1.* I am trying to run one batch job with an adaptive batch scheduler. > I referred to the elastic scaling document [1] and based on the document I > have added the below config to my FlinkDeployment. But the job is failing > with JobInitializationException: Could not start the JobMaster. I have > attached the job manager log below. *Can someone explain why the job is > failing despite using the recommended configuration?* > > jobmanager.scheduler: AdaptiveBatch > parallelism.default: "-1" > taskmanager.network.memory.buffers-per-channel: "0" > execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING" > > > > *2. *The document has listed some limitations of elastic scaling. One of > them is about the input file format. *My code is written in Apache Beam, > how to check, how the input is being read in Flink?* > > > - FileInputFormat sources are not supported: FileInputFormat sources > are not supported, including StreamExecutionEnvironment#readFile(...) > StreamExecutionEnvironment#readTextFile(...) and > StreamExecutionEnvironment#createInput(FileInputFormat, > ...). Users should use the new sources(FileSystem DataStream Connector > > <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/> > or FileSystem SQL Connector > > <https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>) > to read files when using the Adaptive Batch Scheduler. > > > Let me know if additional information is needed to debug the issue. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler > > Regards > Dipak > >
