Re: Sink to database with multiple statements in a transaction, and with retry

2023-01-05 Thread Yoni Gibbs
Also, it seems that sinks have to be synchronous, I think? (Unless there's some async equivalent of SinkFunction​?) Assuming they're synchronous, if I do the retry strategy manually in the implementation of SinkFunction.invoke​ that means I'll be blocking that thread while waiting to do a retry

Sink to database with multiple statements in a transaction, and with retry

2023-01-05 Thread Yoni Gibbs
I want to sink some data to a database, but the data needs to go into multiple tables, in a single transaction. Am I right in saying that I cannot use the JDBC Connector for this as it only handles single SQL statements? Assuming that's right, I believe that I need to write a custom sink, so I n

Async IO & Retry: how to get error details

2023-01-04 Thread Yoni Gibbs
I'm following the documentation here, getting async IO to retry on failure. What I want is for the async action to be attempted, say, three times, and then give up and continue processing f

Re: AsyncDataStream: Retries keep executing after timeout

2022-12-15 Thread Yoni Gibbs
pretty new to Flink, so I might well just have misunderstood something or set something up incorrectly. Thanks in advance, Yoni. From: Lincoln Lee Sent: 14 December 2022 13:51 To: Yoni Gibbs Cc: user@flink.apache.org Subject: Re: AsyncDataStream: Retries keep exe

AsyncDataStream: Retries keep executing after timeout

2022-12-13 Thread Yoni Gibbs
Hi, I've got a Kinesis consumer which reacts to each record by doing some async work using an implementation of RichAsyncFunction. I'm adding a retry strategy. After x failed attempts I want this to time out and give up by returning no data (i.e. not be treated as a failure). Here is a cut dow