[
https://issues.apache.org/jira/browse/HTTPCORE-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17618160#comment-17618160
]
John Leacox edited comment on HTTPCORE-726 at 10/15/22 7:39 PM:
----------------------------------------------------------------
[~olegk] The intent of the change-set is to report the capacity based on the
number of bytes read out of the buffer. The result of that is a reduced number
of buffer expansions and heap allocations. Technically it could maybe still
grow unbounded, but I haven't seen it grow even close to how big it was getting
before. I think, but I could be misunderstanding something, this change to the
way capacity is reported is also more correct; The original implementation
results in the {{CapacityWindow}} growing huge and way out of proportion to the
actual buffer size.
Separately from that, the piece I'm still not understanding is how
{{CapacityWindow}} in {{AbstractHttp1StreamDuplexer}} is intended to control
when the more bytes are read off the socket. Is it odd that it's setting
{{OP_READ}} even though the window is still negative after being incremented?
was (Author: johnlcox):
[~olegk] The intent of the change-set is to report the capacity based on the
number of bytes read out of the buffer. The result of that is a reduced number
of buffer expansions and heap allocations. Technically it could maybe still
grow unbounded, but I haven't seen it grow even close to how big it was getting
before. I think, but I could be misunderstanding something, this change to the
way capacity is reported is also more correct; The original implementation
results in the {{CapacityWindow}} growing huge and way out of proportion to the
actual buffer size.
Separately from that, the piece I'm still not understanding is how
{{CapacityWindow in {{AbstractHttp1StreamDuplexer}}}} is intended to control
when the more bytes are read off the socket. Is it odd that it's setting
{{{{{}OP_READ{}}}}} even though the window is still negative after being
incremented?
> SharedInputBuffer expands unbounded on large responses
> ------------------------------------------------------
>
> Key: HTTPCORE-726
> URL: https://issues.apache.org/jira/browse/HTTPCORE-726
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore
> Affects Versions: 5.1.3
> Environment: MacOS 12, Debian 9, Debian 10
> Reporter: John Leacox
> Priority: Major
> Fix For: 5.1.5, 5.2-beta3
>
> Attachments: SharedInputBuffer_Unbounded_Growth_Example.patch,
> SharedInputBuffer_Unbounded_Growth_Possible_Fix.patch
>
>
> Large responses can result in the {{SharedInputBuffer}} expanding unbounded,
> but with small intervals. Each expansion is the minimum 1024 byte aligned
> size bigger than the current buffer that is filling it beyond its current
> capacity. This leads to a ton of heap allocations and garbage collection each
> time is expands.
>
> It looks like the capacity updates to {{CapacityChannel}} are reporting
> current capacity rather than deltas/increments, so that even when small reads
> occur, the whole capacity is reported again to the channel and the capacity
> window also grows unbounded.
>
> I've attached a patch that includes some {{println}} statements for the
> capacity window size and for buffer being expanded along with the integration
> test {{testLargeResponseConsumer}} that shows the buffer and window growing
> quite large (the test passes, but the println statements show how large they
> are growing).
>
> For a fix I was thinking of updating the capacity tracking to be similar to
> {{ReactiveDataConsumer}} in tracking the number of bytes read and then
> reporting that delta to {{CapacityChannel}}. This does seem to prevent the
> buffer from expanding to huge sizes, but it still expands more than I would
> expect because after each read from the {{SharedInputBuffer}}. This remaining
> expansion appears to be from {{CapacityWindow}} always setting the IO
> sessions with {{OP_READ}} even when the window is still negative after the
> update.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
