Hi Tymur,
1. Why do you want a remote function to call an embedded function in this case?
To Implement an Async IO call.
https://nightlies.apache.org/flink/flink-statefun-docs-master/docs/sdk/flink-datastream/#completing-async-requests
2. Do you have separate outgoing Kafka messages or join them per record?
1 DB query will fetch Millions records which will be aggregated into
multiple separate events/messages to be published to Kafka.
3. What is explicit acknowledgment at the end for?
As 1 Ingress event may have > 500 IDs. So once entire events has been
processed ( i.e. DB query is completed for all 500 IDs ) we have to send a
Trigger Event for Down-Stream consumers.
Also If there is better way to code this problem do share some pointers.
Regards,
Himanshu
________________________________
From: Tymur Yarosh <ti.yar...@gmail.com>
Sent: Tuesday, August 16, 2022 2:37 PM
To: user@flink.apache.org <user@flink.apache.org>; Himanshu Sareen
<himanshusar...@outlook.com>
Subject: Re: Flink-Statefun : Design Solution advice ( ASYNC IO)
Hi,
Just a few questions:
1. Why do you want a remote function to call an embedded function in this case?
2. Do you have separate outgoing Kafka messages or join them per record?
3. What is explicit acknowledgment at the end for?
Best,
Tymur Yarosh
16 серп. 2022 р., 11:48 +0300, Himanshu Sareen <himanshusar...@outlook.com>,
писав(-ла):
Team,
I'm solving a use-case and needs advice/suggestions if Flink is right choice
for solution.
1. Ingress - Kafka events/messages consist of multiple IDs.
2. Task - For each ID in a Kafka message query Cassandra DB ( asynchronously)
to fetch records. Prepare multiple small messages out of the records received.
3. Egress - Emit all the small messages/event to Kafka.
Ingress Event may contain more than 500 IDs
Each ID may result into Millions of Records from Cassandra DB.
My Design Approach
1. Remote Stateful Function listens to Ingress.
2. Invokes Embedded function ( which implements ASYNC IO ) for each ID.
3. Once records are available from Embedded Function, process them and emits
multiple events to Kafka.
4. Send back an acknowledgement to calling remote Function.
Please share suggestions or advice.
Note - I'm unable to find a good example for embedded and remote Function
interaction.
Regards
