Hi Maciej, thanks for joining. I answer your comments below.
>
> the idea is quite interesting - although maintaining some coordination to
> be able to handle checkpoints would probably pretty tricky. Did you figure
> out how to handle proper distribution of tasks between TMs? As far as I
> unders
Hi Krzysiek,
the idea is quite interesting - although maintaining some coordination
to be able to handle checkpoints would probably pretty tricky. Did you
figure out how to handle proper distribution of tasks between TMs? As
far as I understand you have to guarantee that all sources reading fr
I want to do a bit different hacky PoC:
* I will write a sink, that caches the results in "JVM global" memory. Then
I will write a source, that reads this cache.
* I will launch one job, that reads from Kafka source, shuffles the data to
the desired partitioning and then sinks to that cache.
* Then
I saw that requirement but I'm not sure if you really need to modify the
query at runtime.
Unless you need reprocessing for newly added rules, I'd probably just
cancel with savepoint and restart the application with the new rules. Of
course, it depends on the rules themselves and how much state th
Hello Arvid,
Thanks for joining to the thread!
First, did you take into consideration that I would like to dynamically add
queries on the same source? That means first define one query, later the
day add another one , then another one, and so on. A Week later kill one of
those, start yet another on
Hi Krzysztof,
from my past experience as data engineer, I can safely say that users often
underestimate the optimization potential and techniques of the used
systems. I implemented a similar thing in the past, where I parsed up to
500 rules reading from up to 10 data sources.
The basic idea was to
