Thank you both! I'll try to switch the scheduler to "AdaptiveBatchScheduler".
Best, Irakli ________________________________ From: Junrui Lee <jrlee....@gmail.com> Sent: 05 March 2024 03:50 To: user <user@flink.apache.org> Subject: Re: Batch mode execution Hello Irakli, The error is due to the fact that the Adaptive Scheduler doesn’t support batch jobs, as detailed in the Flink documentation[1]. When operating in reactive mode, Flink automatically decides the type of scheduler to use. For batch execution, the default scheduler is AdaptiveBatchScheduler, not AdaptiveScheduler as in the streaming case. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#limitations<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#limitations> Best regards, Junrui lorenzo.affetti.ververica.com<http://lorenzo.affetti.ververica.com> via user <user@flink.apache.org<mailto:user@flink.apache.org>> 于2024年3月4日周一 23:24写道： Hello Irakli and thank you for your question. I guess that somehow Flink enters the "reactive" mode while the adaptive scheduler is not configured. I would go with 2 options to isolate your issue: * Try with forcing the scheduling mode (https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler<https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/elastic_scaling/#adaptive-scheduler>) in your configuration: `jobmanager.scheduler: adaptive` * Let Flink decide for the runtime mode: if the source is bounded, you don't need `env.setRuntimeMode(RuntimeExecutionMode.BATCH)`, as Flink will understand that correctly. Can you try one of the two and see if that helps? For the rest: "running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy". Still, you could opt for a simple watermarking strategy and stay with the streaming mode (for example, 'BoundedOutOfOrcerness': https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector>). On Mar 4, 2024 at 15:54 +0100, irakli.keshel...@sony.com<mailto:irakli.keshel...@sony.com> <irakli.keshel...@sony.com<mailto:irakli.keshel...@sony.com>>, wrote: Hello, I have a Flink job which is processing bounded number of events. Initially, I was running the job in the "STREAMING" mode, but I realized that running it in the "BATCH" mode was better as I don't have to deal with the Watermark Strategy. The job is reading the data from the Kafka topic and was running fine in the "STREAMING" mode. I switched the job to the "BATCH" mode by setting "env.setRuntimeMode(RuntimeExecutionMode.BATCH)". I changed the Kafka Source to be bounded by setting ".setBounded(OffsetsInitializer.latest())" to the source builder and I updated the Watermark strategy to be "WatermarkStrategy.noWatermarks()". After making these changes and deploying the job I end up with following error: "java.lang.IllegalStateException","error.message":"Adaptive Scheduler is required for reactive mode". I couldn't find any documentation online which is connecting "Adaptive Scheduler" to the "BATCH" processing. Does anyone know where this error is coming from and how I can deal with it? Cheers, Irakli
