FileSource may cause akka.pattern.AskTimeoutException, and akka.ask.timeout not workable

2021-06-09 Thread
Hello all, Our team encounter *akka.pattern.AskTimeoutException *when start jobmanager. Base on the error message, we try to setup *akka.ask.timeout * and* web.timeout *to 360s, but both of them doesn't work. We guess the issue may cause by *FileSource.forRecordFileFormat.* The application will l

Re: SourceFunction cannot run in Batch Mode

2021-06-02 Thread
Sorry, there are some typos that may be misleading. The SourceFunction will be detected as* Streaming Mode.* 陳樺威 於 2021年6月3日 週四 下午1:29寫道: > Hi, > > Currently, we want to use batch execution mode [0] to consume historical > data and rebuild states for our streaming application. >

SourceFunction cannot run in Batch Mode

2021-06-02 Thread
Hi, Currently, we want to use batch execution mode [0] to consume historical data and rebuild states for our streaming application. The Flink app will be run on-demand and close after complete all the file processing. We implement a SourceFuntion [1] to consume bounded parquet files from GCS. Howe

Re: KafkaSource metrics

2021-05-24 Thread
ax - > Consumer lag > flink_taskmanager_job_task_operator_KafkaConsumer_commit_latency_max - > commit latency > > unsure if reactive mode makes any difference. > On Mon, May 24, 2021 at 7:44 PM 陳樺威 wrote: > >> Hello, >> >> Our team tries to test reacti

KafkaSource metrics

2021-05-24 Thread
Hello, Our team tries to test reactive mode and replace FlinkKafkaConsumer with the new KafkaSource. But we can’t find the KafkaSource metrics list. Does anyone have any idea? In our case, we want to know the Kafka consume delay and consume rate. Thanks, Oscar