Re: Is outputting from components other than sinks or side outputs a no-no ?

2020-07-27 Thread Tom Fennelly
ve to take all those potentials into account > which significantly complicates the code > > On 2020/07/27 07:45:27, Tom Fennelly wrote: > > Thank you David. > > > > In the case we have in mind it should only happen literally on the very > > rare Exception i.e. in som

Re: Is outputting from components other than sinks or side outputs a no-no ?

2020-07-27 Thread Tom Fennelly
dev/stream/operators/asyncio.html> > > Best, > David > > On Sun, Jul 26, 2020 at 11:08 AM Tom Fennelly > wrote: > >> Hi. >> >> What are the negative side effects of (for example) a filter function >> occasionally making a call out to a DB ? Is this a big no-no and should all >> outputs be done through sinks and side outputs, no exceptions ? >> >> Regards, >> >> Tom. >> >

Is outputting from components other than sinks or side outputs a no-no ?

2020-07-26 Thread Tom Fennelly
Hi. What are the negative side effects of (for example) a filter function occasionally making a call out to a DB ? Is this a big no-no and should all outputs be done through sinks and side outputs, no exceptions ? Regards, Tom.

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-24 Thread Tom Fennelly
Thanks Robert. On Fri, Jul 24, 2020 at 2:32 PM Robert Metzger wrote: > Hey Tom, > > I'm not aware of any patterns for this problem, sorry. Intuitively, I > would send dead letters to a separate Kafka topic. > > Best, > Robert > > > On Wed, Jul 22, 20

Re: Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Tom Fennelly
; > Thanks, > > Chen > > > > > > > > *From: *Eleanore Jin > *Sent: *Wednesday, July 22, 2020 9:25 AM > *To: *Tom Fennelly > *Cc: *user > *Subject: *Re: Recommended pattern for implementing a DLQ with Flink+Kafka > > > > +1 we have a similar us

Recommended pattern for implementing a DLQ with Flink+Kafka

2020-07-22 Thread Tom Fennelly
Hi. I've been searching blogs etc trying to see if there are established patterns/mechanisms for reprocessing of failed messages via something like a DLQ. I've read about using checkpointing and restarting tasks (not what we want because we want to keep processing forward) and then also how some u