Re: [Question] Bundle finalization callback

Got it. The ack deadline is configurable on the consumer but this gave me some new ideas. Thanks! Johanna On Sun, Oct 15, 2023, 22:51 Robert Burke wrote: > I would recommend avoiding over fitting to a specific runner, but given > the constraints, I'd say being eager about self checkpointing and

Re: [Question] Bundle finalization callback

I would recommend avoiding over fitting to a specific runner, but given the constraints, I'd say being eager about self checkpointing and ensuring fine grain splits. This will allow runners to schedule more bundles in parallel if they are able, and provide independence between them. Part of the is

Re: [Question] Bundle finalization callback

Okay I see, thank you for your quick reply! I'll have a look into that file. Do you have an idea of on which interval I could expect the Dataflow runner to initiate the finalization? Thinking of the case where I have a message ack deadline of e.g. 30s and a continuous stream of messages that keeps

Re: [Question] Bundle finalization callback

Hi! Nswers inline. On Sun, Oct 15, 2023, 11:48 AM Johanna Öjeling via dev wrote: > Hi, > > I'm working on a native streaming IO connector for the Go SDK to enable > reads and writes from/to NATS (#29000 > ) and would like to better > understand how bu