Re: The best way to read historical data in a stream

2021-07-07 Thread Lasse Nedergaard
Hi Arvid Thanks for your input. My first implemention was iterations but I had a challenges to match up the returned rows with the original input so the current implementation use Async IO where I attach the found rows with the input. It makes it easier downstream. I just have to test if I can

Re: The best way to read historical data in a stream

2021-07-07 Thread Arvid Heise
Hi Lasse, That's a tough question. The real Kappa way would be to load the full database as a 2. input into the job and use joins. But I'm assuming that you can't or don't want to do that. 1. Can work if you use a windowing operator before and only trigger one or few async IO calls per window bat

The best way to read historical data in a stream

2021-07-05 Thread Lasse Nedergaard
Hi I’m looking for some advice for the “right” way to load historical data into a stream. The case is as follow. I have a stream, sometimes I need to match the current live stream data up against data stored in database, let say elasticsearch, I generate a side output with the query informat