Hi! In that case just put these two SQL queries into one job and use the same table as their source. The source task will be automatically reused (actually controlled by a configuration [1] but it is enabled by default) and the records will be only read once from Kafka and will be replicated inside Flink if necessary.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-optimizer-reuse-source-enabled Surendra Lalwani <surendra.lalw...@swiggy.in> 于2022年5月17日周二 15:52写道： > So let's assume if we are reading data from a topic named A and performing > aggregation on 4 hour window and a different pipeline which does same > aggregation for 6 hour window, so in this case we are reading same data > couple of times and processing it for different intervals, can we do it > using just a single read. > > Thanks and Regards , > Surendra Lalwani > > > On Tue, May 17, 2022 at 1:19 PM Caizhi Weng <tsreape...@gmail.com> wrote: > >> Hi! >> >> I wanted to explore if we can avoid reading multiple times and read only >>> once >>> >> >> Could you elaborate more on the reason you need this? If both of your >> queries need full data from that topic you'll have to read the whole topic >> many times. If each query only need to consume a specific portion of that >> topic I would suggest splitting it into multiple topics. >> >> >> Surendra Lalwani <surendra.lalw...@swiggy.in> 于2022年5月17日周二 15:44写道： >> >>> Hi Team, >>> >>> We have various SQL queries where we are querying the same kafka topic >>> but both of the queries are completely different. I wanted to explore if we >>> can avoid reading multiple times and read only once and perform different >>> processing according to the query and dump data accordingly. >>> >>> Thanks and Regards , >>> Surendra Lalwani >>> >>> >>> ------------------------------ >>> IMPORTANT NOTICE: This e-mail, including any attachments, may contain >>> confidential information and is intended only for the addressee(s) named >>> above. If you are not the intended recipient(s), you should not >>> disseminate, distribute, or copy this e-mail. Please notify the sender by >>> reply e-mail immediately if you have received this e-mail in error and >>> permanently delete all copies of the original message from your system. >>> E-mail transmission cannot be guaranteed to be secure as it could be >>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or >>> contain viruses. Company accepts no liability for any damage or loss of >>> confidential information caused by this email or due to any virus >>> transmitted by this email or otherwise. >> >> > > ------------------------------ > IMPORTANT NOTICE: This e-mail, including any attachments, may contain > confidential information and is intended only for the addressee(s) named > above. If you are not the intended recipient(s), you should not > disseminate, distribute, or copy this e-mail. Please notify the sender by > reply e-mail immediately if you have received this e-mail in error and > permanently delete all copies of the original message from your system. > E-mail transmission cannot be guaranteed to be secure as it could be > intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. Company accepts no liability for any damage or loss of > confidential information caused by this email or due to any virus > transmitted by this email or otherwise.
