[
https://issues.apache.org/jira/browse/IGNITE-28673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102108#comment-18102108
]
Anton Vinogradov commented on IGNITE-28673:
-------------------------------------------
I looked into this while Aleksandr was away, and I am handing the ticket back
with what I found. No code from me, only analysis.
h3. The reproducer fails on clean master
Runs of {{zkTest#testConcurrentUpdatesAndQueryStartAtomicCacheGroup}} on one
machine (12 cores, heap 9.6G):
|| build || result ||
| master, unchanged | fails, 317.7 s |
| master + the change from this ticket | fails, 318.8 s |
| same test on TCP discovery | passes, 45.4 s |
The change does not introduce the failure. ZooKeeper runs fail the same way
without it.
h3. On a large heap the symptom is not OOM
No {{OutOfMemoryError}} in these runs. Instead:
* the client node received 15 534 790 messages (2.27 GB) in about 170 seconds;
* the pause detector logged 131 long JVM pauses, the longest 3.7 s;
* the client lost its ZooKeeper session ("have not heard from server in
14347ms") and the coordinator evicted it: {{Node FAILED ... client=true}};
* after that the test stays on "Iteration: 0" until the 300 s limit.
This is the same overload the heap dump shows. With a smaller heap it ends in
OOM; with a large heap the GC pauses kill the ZooKeeper session first.
h3. Sending the start result under another pool does not help
I tried sending *{{ContinuousRoutineStartResultMessage}}* with
{{MANAGEMENT_POOL}} instead of {{SYSTEM_POOL}}. The test still failed (362.8
s). Two reasons, and both matter for the design:
# {{connectionsPerNode}} is 1 by default
({{TcpCommunicationSpi.DFLT_CONN_PER_NODE}}), so all messages to a node share
one connection. The start result waits behind the notifications on the wire,
not in the executor queue. The pool decides who runs a message after it
arrives, not when it arrives.
# On this reproducer the client dies before the ordering matters, so it cannot
confirm or reject a fix that only changes priority. A lighter load is needed to
test that part.
h3. What is underneath
Nothing slows the producer down: {{messageQueueLimit}} is 0 by default
({{TcpCommunicationSpi.DFLT_MSG_QUEUE_LIMIT}} =
{{GridNioServer.DFLT_SEND_QUEUE_LIMIT}} = unlimited). The server sends
notifications as fast as it creates them, and a subscriber that cannot keep up
only accumulates them. Moving the start completion off discovery makes this
visible, but the overload is older than this ticket. I filed IGNITE-28955 for
it and linked it here.
> Remove mutable path from continuous routine start discovery
> -----------------------------------------------------------
>
> Key: IGNITE-28673
> URL: https://issues.apache.org/jira/browse/IGNITE-28673
> Project: Ignite
> Issue Type: Task
> Reporter: Aleksandr Chesnokov
> Assignee: Aleksandr Chesnokov
> Priority: Major
> Labels: IEP-132, ise, wire-format
> Fix For: 2.19
>
> Attachments: patch.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Remove the mutable continuous routine start discovery path and keep a single
> immutable-style flow for starting routines. This should simplify
> *{{GridContinuousProcessor}}* by removing duplicated start messages, mutable
> ack handling, and special branching between mutable and immutable discovery
> modes
> During validation,
> *{{{}CacheContinuousQueryConcurrentPartitionUpdateTest{}}}#concurrentUpdatesAndQueryStart*
> exposed a performance issue. After replacing the discovery ack with
> {*}{{ContinuousRoutineStartResultMessage}}{*}, CQ start completion started
> using the same *{{TOPIC_CONTINUOUS}}* communication path as regular CQ
> notifications
> Under heavy update load, CQ notifications may create a large communication
> backlog and delay {*}{{ContinuousRoutineStartResultMessage}}{*}. JFR and heap
> dump analysis showed millions of retained *{{GridContinuousMessage}}* and
> *{{CacheContinuousQueryEntry}}* objects and only one waiting
> *{{ContinuousRoutineStartResultMessage}}*
> As a result, _{{cache.query(qry)}}_ may hang for a long time while waiting
> for *{{ContinuousRoutineStartResultMessage}}* from the communication queue
> instead of discovery
> UPD: The same problem with zookeeper and immutable path.
> * Download and apply patch from attachments
> * Run zkTest#testConcurrentUpdatesAndQueryStartAtomicCacheGroup
> * See OOM
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
