Re: Kafka source with multiple partitions loses data during savepoint recovery

":2} {"@timestamp":"2022-03-18T03:54:25.083Z","@version":"1","message":"[Consumer clientId=KafkaSource--5609054382257236677-0, groupId=null] Seeking to offset 2 for partition stream--67362410-5","logger_name":"

Kafka source with multiple partitions loses data during savepoint recovery

Hi, I'm seeing an odd behavior for Kafka source where some records are dropped during recovery. My test set up is: Kafka source topic -> pass through flink job -> Kafka sink topic There are 10 partitions in the source & sink topics. Test Steps * Start the flink job, send 5 records (first batch)

Re: Exact-once processing when a job fails

hasn't >> completed). >> >> group-offsets does not help in your case. There actually is an option to >> commit offsets to Kafka during each checkpoint but Flink will also manage >> offsets in its own state. If there is no checkpoint then group offsets >> won'

Re: Exact-once processing when a job fails

hich is the default setting). The first > record will also be reprocessed but this is still valid because it is just > updating the result for the first record (which is the same as your > previous execution). > > Sharon Xie 于2022年1月5日周三 02:56写道： > >> Can someone help me unde

Exact-once processing when a job fails

Can someone help me understand how Flink deals with the following scenario? I have a job that reads from a source Kafka (starting-offset: latest) and writes to a sink Kafka with exactly-once execution. Let's say that I have 2 records in the source. The 1st one is processed without issue and the jo

Re: Unable to update logback configuration in Flink Native Kubernetes

I've faced the same issue before. I figured out that there is an internal configuration `$internal.deployment.config-dir` (code ) wh

Re: [External] : Timeout settings for Flink jobs?

t; On Fri, Oct 15, 2021 at 7:05 PM Fuyao Li wrote: > >> Hi Sharon, >> >> >> >> I think for DataStream API, you can override the isEndOfStream() method >> in the DeserializationSchema to control the input data source to end and >> thus end the workflow. &g

Re: Timeout settings for Flink jobs?

se case? Are you running a streaming job > or a batch job? For streaming jobs it is rare to have a time limit. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/rest_api/#jobs-jobid > > Sharon Xie 于2021年10月12日周二 上午3:42写道： > >> Hi there, >> &g

Timeout settings for Flink jobs?

Hi there, We have a use case where we want to terminate a job when a time limit is reached. Is there a Flink setting that we can use for this use case? Thanks, Sharon

Re: New session mode job manager deployment rejected existing task managers

Actually we figured it out. We need to configure High Availability mode to recover jobs during new kubernetes deployment. On Tue, Oct 5, 2021 at 11:39 AM Sharon Xie wrote: > Hi, > > I'm currently running Flink 1.13.2 using kubernetes session mode - native > kubernetes. When

New session mode job manager deployment rejected existing task managers

Hi, I'm currently running Flink 1.13.2 using kubernetes session mode - native kubernetes. When I update the job manager deployment through `kubectl apply flink-jobmanager-deployment.yaml`, a new job manager pod is created. I'd expect all the task manager pods will re-register with the new JM pod.