Hello,
I am currently using Dynamodb as a persistent store in my application, and in process of transforming the application into streaming pipeline utilizing Flink. Pipeline is simple and stateless, consume data from Kafka, apply some transformation and write records to Dynamodb. I would want to collect records in buffer and batch write them instead of writing one at a time. Also want to put retry and exponential back off for unprocessed records in batch. My first question , is Dynamodb a suitable Sink? I googled and searched on mailing list archive, but couldn’t find anyone having that requirement. What are some of the guidelines around implementing a fault tolerant Sink with at least once guarantees? I am looking at ElasticSearch implementation and it does have the capabilities that I want, Could I just try implementing Dynamodb Sink similar way? Basically, implement Checkpointed interface, Collect records in buffer in invoke method, flush buffer and checkEerrors in snapshot method, Flush method will call Dynamodb’s batchwriteitem and if there are any unprocessed records add them back in buffer. FailureHandler will attempt to flush buffer again or do nothing or fail the sink based on failure handling policy. Please let me know if am in right direct Thank You
