Re: How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Leon Xu
Thanks Austin. Will take a look at the AsyncIO. Looks like a pretty cool feature. On Fri, Feb 10, 2023 at 1:31 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > It's been a while, but I think I've done something similar before with > Async I/O [1] and batching records with a window. >

Re: How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Austin Cawley-Edwards
It's been a while, but I think I've done something similar before with Async I/O [1] and batching records with a window. This was years ago, so no idea if this was/is good practice, but essentially it was: -> Window by batch size (with a timeout trigger to maintain some SLA) -> Process function t

Re: How to process mini-batch events in Flink with Datastream API

2023-02-10 Thread Leon Xu
I wonder if windows will be the solution when it comes to datastream API. On Fri, Feb 10, 2023 at 12:07 PM Leon Xu wrote: > Hi Flink Users, > > We wanted to use Flink to run a decoration pipeline, where we would like > to make calls to some external service to fetch data and alter the event in >