Re: The best way to read historical data in a stream

2021-07-07 Thread Lasse Nedergaard
Hi Arvid Thanks for your input. My first implemention was iterations but I had a challenges to match up the returned rows with the original input so the current implementation use Async IO where I attach the found rows with the input. It makes it easier downstream. I just have to test if I can

Re: The best way to read historical data in a stream

2021-07-07 Thread Arvid Heise
Hi Lasse, That's a tough question. The real Kappa way would be to load the full database as a 2. input into the job and use joins. But I'm assuming that you can't or don't want to do that. 1. Can work if you use a windowing operator before and only trigger one or few async IO calls per window bat