Hi all, I need to send a request to an external web service and then store the response in a DB table, and I am wondering how people have approached this or similar problems in the past.
The flow is: Kafka source (msgs only every few seconds) => filter/map operators => result sent to web service (which updates state in that system) => response stored in DB. Initially I was thinking of just creating a custom sink which basically: Sends request to webservice => Get response containing external key => Save key into DB This feels to me like basically smashing together 2 separate sinks into 1, and I am not sure if that is a good design or not. Another option would be to create a RichMapFunction (possibly async function) which does the web service call. My map function can then just return the response which I can then feed into a standard DB sink. However, with this approach it feels strange to update an external system in a map() function, but maybe that's ok? Also, I presume to make my map function idempotent I would need to store some state (I can key the messages and use a ValueState) so I don't do duplicate web service calls if there is a failure? Thoughts? Thanks in advance.
