On Sat, Jan 6, 2018 at 5:57 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Thu, Jan 4, 2018, at 10:37, Jun Rao wrote: >> >> 4. The process to mark partition as dirty requires updating every fetch >> session having the partition. This may add some overhead. An alternative >> approach is to check the difference btw cached fetch offset and HW (or LEO) >> when serving the fetch request. > > That's a good point. The caching approach avoids needing to update every > fetch session when one of those numbers changes. I think an even more > important advantage is that it's simpler to implement -- we don't have to > worry about forgetting to update a fetch session when one of those numbers > changes. The disadvantage is some extra memory consumption per partition per > fetch session. > > I think the advantage, especially in terms of simplicity, might override the > memory concern. My initial implementation uses the caching approach. I will > update the KIP once I have this working. >
We're very interested in this KIP because it might improve one of our topic-heavy clusters. I have a stress-test generating topics across a number of kafka brokers; if you'd like early and quick feedback on your implementation let me know! The discussion thread is very long, so hopefully I'm not asking something that was asked before: does Kafka already expose `FetchRequest` size for monitoring purposes? It might improve the KIP to track the before-and-after behavior. -Andrey
