Hi Xuyang,
Thanks for the reply!
I don't have any dataset to share with you at this time.
The last block in the Flink pipeline performs 2 functions - temporal join( 2nd
temporal join) and writing data into the sink topic.
This is what Flink SQL code looks like -
SELECT * FROM
TABLE1 T1
LEFT JOIN TABLE2 FOR SYSTEM_TIME AS OF T1.timestamp as T2 -- Temporal
join
ON *JOIN CONDITION*
LEFT JOIN TABLE3 FOR SYSTEM_TIME AS OF T1.timestamp AS T3 -- Temporal
join
ON *JOIN CONDITION*
Thanks,
Yad
________________________________
From: Xuyang <xyzhong...@163.com>
Sent: Friday, December 15, 2023 9:33 AM
To: user@flink.apache.org <user@flink.apache.org>
Subject: Re:Re: Event stuck in the Flink operator
This Message Is From an Untrusted Sender
"Caution" You have not previously corresponded with this sender. If you do not
recognize this send, verify their identity offline. If the message appears
suspicious, please click the "Report Phish" button in your Outlook client.
Hi, Yad.
Can you share the smallest set of sql that can reproduce this problem?
BTW, the last flink operator you mean is the sink with kafka connector?
--
Best！
Xuyang
在 2023-12-15 04:38:21，"Alex Cruise" <a...@cluonflux.com> 写道：
Can you share your precise join semantics?
I don't know about Flink SQL offhand, but here are a couple ways to do this
when you're using the DataStream API:
* use the Session Window
join<https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/joining/*session-window-join__;Iw!!KtM2tloZCg!uiRCPr9yeK3PN4HKfOdb19KvOLh7poLXLzIj4OT8a6vIDR_K5ns3OqUc_4CxFkxhOJoH2zfpV_RN_oFRnT8JX84$>
type, which automatically closes a window after a configurable delay since the
last record
* if you're using a ProcessFunction (very low-level), you can set a
timer<https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/process_function/*timers__;Iw!!KtM2tloZCg!uiRCPr9yeK3PN4HKfOdb19KvOLh7poLXLzIj4OT8a6vIDR_K5ns3OqUc_4CxFkxhOJoH2zfpV_RN_oFReMavaGE$>
if you need to react to the non-arrival of data. Just make sure you cancel it
when data finally does arrive. :)
-0xe1a
On Thu, Dec 14, 2023 at 6:36 AM T, Yadhunath
<yadhunat...@boarshead.com<mailto:yadhunat...@boarshead.com>> wrote:
Hi,
I am using Flink version 1.16 and I have a streaming job that uses PyFlinkSQL
API.
Whenever a new streaming event comes in it is not getting processed in the last
Flink operator ( it performs temporal join along with writing data into Kafka
topic) and it will be only pushed to Kafka on the arrival of the next streaming
event. It is like the last operator needs an event to process the previous
event. Did anyone experience a similar issue?
Really appreciate if someone could advise a solution for this.
Please let me know if you require more input.
Thanks,
Yad
