Hi!
In that case just put these two SQL queries into one job and use the same
table as their source. The source task will be automatically reused
(actually controlled by a configuration [1] but it is enabled by default)
and the records will be only read once from Kafka and will be replicated
insid
So let's assume if we are reading data from a topic named A and performing
aggregation on 4 hour window and a different pipeline which does same
aggregation for 6 hour window, so in this case we are reading same data
couple of times and processing it for different intervals, can we do it
using just
Hi!
I wanted to explore if we can avoid reading multiple times and read only
> once
>
Could you elaborate more on the reason you need this? If both of your
queries need full data from that topic you'll have to read the whole topic
many times. If each query only need to consume a specific portion
Hi Team,
We have various SQL queries where we are querying the same kafka topic but
both of the queries are completely different. I wanted to explore if we can
avoid reading multiple times and read only once and perform different
processing according to the query and dump data accordingly.
Thanks
