Hi Guozhang,
Thanks for the KIP! This is an important feature for Kafka Streams and will
help to unlock a bunch of use cases.
I have some concerns/questions:
1. Producer per task: I'm worried about the overhead this is going to
put on both the streams app and the Kafka Brokers. You can easily imagine
an app consuming thousands of partitions. What load will this put on the
brokers? Am i correct in assuming that there will be metadata requests per
Producer? The memory overhead in the streams app will also increase fairly
significantly. Should we adjust ProducerConfig.BUFFER_MEMORY_CONFIG?
2. State Store recovery: As we already know, restoring the entire
changelog can take an extremely long time. Even with a fairly small dataset
and an inappropriately tuned segment size, this can take way too long. My
concern is that failures happen and then recovery takes "forever" and we
end up in a situation where we need to change the max.poll.interval to be
some very large number or else we end up in "rebalance hell". I don't think
this provides a very good user experience. You mention RocksDB
checkpointing in the doc - should we explore this idea some more? i.e.,
understand the penalty for checkpointing. Maybe checkpoint every *n*
commits?
3. What does EoS mean for Caching? If we set the commit interval to
100ms then the cache is not going to be very effective. Should it just be
disabled?
Thanks,
Damian
On Tue, 28 Feb 2017 at 21:54 Guozhang Wang <wangg...@gmail.com> wrote:
> Hi all,
>
> I have just created KIP-129 to leverage KIP-98 in Kafka Streams and provide
> exactly-once processing semantics:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics
>
> This KIP enables Streams users to optionally turn on exactly-once
> processing semantics without changing their app code at all by leveraging
> the transactional messaging features provided in KIP-98.
>
> The above wiki page provides a high-level view of the proposed changes,
> while detailed implementation design can be found in this Google doc:
>
>
> https://docs.google.com/document/d/1pGZ8xtOOyGwDYgH5vA6h19zOMMaduFK1DAB8_gBYA2c
>
> We would love to hear your comments and suggestions.
>
> Thanks,
> -- Guozhang
>
