On Fri, Jan 8, 2021 at 8:27 AM Enrico Olivelli <eolive...@gmail.com> wrote: > > Hi Matteo, > in this comment you are talking about an issue you saw when WQ is greater > that AQ > https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246 > > IIUC you are saying that if one bookie is slow the client continues to > accumulate references to the entries that still have not received the > confirmation from it. > I think that this is correct. > > Have you seen problems in production related to this scenario ? > Can you tell more about them ?
Yes, for simplicity, assume e=3, w=3, a=2. If one bookie is slow (not down, just slow), the BK client will the acks to the user that the entries are written after the first 2 acks. In the meantime, it will keep waiting for the 3rd bookie to respond. If the bookie responds within the timeout, the entries can now be dropped from memory, otherwise the write will timeout internally and it will get replayed to a new bookie. In both cases, the amount of memory used in the client will max at "throughput" * "timeout". This can be a large amount of memory and easily cause OOM errors. Part of the problem is that it cannot be solved from outside the BK client, since there's no visibility on what entries have 2 or 3 acks and therefore it's not possible to apply backpressure. Instead, there should be a backpressure mechanism in the BK client itself to prevent this kind of issue. One possibility there could be to use the same approach as described in https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits, giving a max memory limit per BK client instance and throttling everything after the quota is reached. Matteo
