Hi all, I have updated the KIP to reflect changes from the discussion in this thread. In particular, the KIP now recommends adding a configuration that allows disabling any kind of message down-conversion for client `FetchRequest`, instead of having to specify the minimum compatibility.
I also added a note about Jun's idea of down-converting only the first message batch of the first topic-partition. I think this is a bit tricky to implement, and given that the maximum fetch size for a topic-partition is capped at 1MB by default, it should probably be acceptable to take the cost of down-converting the entire partition upfront. Jun, let me know what you think. Any other suggestions / feedback are welcome. Thanks, Dhruvil On Tue, Apr 17, 2018 at 4:21 PM, Dhruvil Shah <dhru...@confluent.io> wrote: > Hi Jun, > > Yes, that is true. Ideally, we should be able to down-convert only the > first message batch in the request handling thread and delay everything > else till the network thread. I have not thought through all the details of > how we could do this but at first glance this seems tricky to implement, > given that `FetchResponse.PartitionData` for the first partition will be a > combination of `MemoryRecords` and `LazyDownConvertedRecords`. I will think > about this a bit more to see if I can come up with a clean abstraction, and > will also add it to the KIP. > > Thanks, > Dhruvil > > On Mon, Apr 16, 2018 at 6:07 PM, Jun Rao <j...@confluent.io> wrote: > >> Hi, Dhruvil, >> >> Thanks for the KIP. Looks good me to overall. Just one comment below. >> >> "To prevent this from happening, we will not delay down-conversion of the >> first partition in the response. We will down-convert all messages of the >> first partition in the I/O thread (like we do today), and only delay >> down-conversion for subsequent partitions." It seems that we can further >> optimize this by only down-converting the first message set in the first >> partition in the request handling threads? >> >> Jun >> >> >> On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah <dhru...@confluent.io> >> wrote: >> >> > Hi, >> > >> > I created a KIP to help mitigate out of memory issues during >> > down-conversion. The KIP proposes introducing a configuration that can >> > prevent down-conversions altogether, and also describes a design for >> > efficient memory usage for down-conversion. >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > 283%3A+Efficient+Memory+Usage+for+Down-Conversion >> > >> > Suggestions and feedback are welcome! >> > >> > Thanks, >> > Dhruvil >> > >> > >
