Hi, Just a few questions: 1. Why do you want a remote function to call an embedded function in this case? 2. Do you have separate outgoing Kafka messages or join them per record? 3. What is explicit acknowledgment at the end for?
Best, Tymur Yarosh 16 серп. 2022 р., 11:48 +0300, Himanshu Sareen <himanshusar...@outlook.com>, писав(-ла): > Team, > > I'm solving a use-case and needs advice/suggestions if Flink is right choice > for solution. > > 1. Ingress - Kafka events/messages consist of multiple IDs. > 2. Task - For each ID in a Kafka message query Cassandra DB ( asynchronously) > to fetch records. Prepare multiple small messages out of the records received. > 3. Egress - Emit all the small messages/event to Kafka. > > Ingress Event may contain more than 500 IDs > Each ID may result into Millions of Records from Cassandra DB. > > > My Design Approach > > 1. Remote Stateful Function listens to Ingress. > 2. Invokes Embedded function ( which implements ASYNC IO ) for each ID. > 3. Once records are available from Embedded Function, process them and emits > multiple events to Kafka. > 4. Send back an acknowledgement to calling remote Function. > > Please share suggestions or advice. > > Note - I'm unable to find a good example for embedded and remote Function > interaction. > > Regards
