[
https://issues.apache.org/jira/browse/KAFKA-12475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306615#comment-17306615
]
A. Sophie Blee-Goldman commented on KAFKA-12475:
------------------------------------------------
Well, many remote storage backends support transactions of some kind, no? I'm
not saying we have perfect support for this at the moment (clearly EOS + remote
storage has many hurdles to get past), but it should be possible to layer in
some kind of support for remote state stores as a first class citizen. Not
necessarily on the near-term roadmap of course, but as we continue to solidify
EOS and move towards it as the standard then it's good to have some plans in
the back pocket for these special cases.
Maybe I'm secretly hoping there is an advanced user who heavily relies on
remote stores and will want to pick this up for us :P
> Kafka Streams breaks EOS with remote state stores
> -------------------------------------------------
>
> Key: KAFKA-12475
> URL: https://issues.apache.org/jira/browse/KAFKA-12475
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
> Labels: needs-kip
>
> Currently in Kafka Streams, exactly-once semantics (EOS) require that the
> state stores be completely erased and restored from the changelog from
> scratch in case of an error. This erasure is implemented by closing the state
> store and then simply wiping out the local state directory. This works fine
> for the two store implementations provided OOTB, in-memory and rocksdb, but
> fails when the application includes a custom StateStore based on remote
> storage, such as Redis. In this case Streams will fail to erase any of the
> data before reinserting data from the changelog, resulting in possible
> duplicates and breaking the guarantee of EOS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
