[
https://issues.apache.org/jira/browse/IGNITE-28673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102105#comment-18102105
]
Anton Vinogradov commented on IGNITE-28673:
-------------------------------------------
[~chesnokoff] I picked this up while you are away - will sync with you when you
are back, and the credit for the analysis below is yours.
Two findings from porting your PR #13133 onto current master.
*The first of your three commits is already in master.*
{{StartRoutineDiscoveryMessageV2}} is merged into
{{StartRoutineDiscoveryMessage}} - it came in through IGNITE-28477. Only the
removal of the mutable path is still needed.
*The hang is not caused by your change.* I ran the ZooKeeper reproducer from
your attachment on clean master, with none of your commits applied: it times
out at 317.7 s against the 300 s limit - the same as with your change (318.8
s). Under TCP discovery the same test class passes in 45 s either way.
That matches what you wrote in the UPD: the problem is on the immutable path,
and ZooKeeper already uses it, because {{immutableDiscoCustomMsg =
!ctx.discovery().mutableCustomMessages()}} and {{ZookeeperDiscoverySpi}}
carries no {{@DiscoverySpiMutableCustomMessageSupport}}.
So this is a live bug in master rather than a regression of this ticket, and
this ticket is what would spread it from ZooKeeper to everyone. I filed it
separately as IGNITE-28955 and linked it as a blocker; taking that one first.
> Remove mutable path from continuous routine start discovery
> -----------------------------------------------------------
>
> Key: IGNITE-28673
> URL: https://issues.apache.org/jira/browse/IGNITE-28673
> Project: Ignite
> Issue Type: Task
> Reporter: Aleksandr Chesnokov
> Assignee: Anton Vinogradov
> Priority: Major
> Labels: IEP-132, ise, wire-format
> Fix For: 2.19
>
> Attachments: patch.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Remove the mutable continuous routine start discovery path and keep a single
> immutable-style flow for starting routines. This should simplify
> *{{GridContinuousProcessor}}* by removing duplicated start messages, mutable
> ack handling, and special branching between mutable and immutable discovery
> modes
> During validation,
> *{{{}CacheContinuousQueryConcurrentPartitionUpdateTest{}}}#concurrentUpdatesAndQueryStart*
> exposed a performance issue. After replacing the discovery ack with
> {*}{{ContinuousRoutineStartResultMessage}}{*}, CQ start completion started
> using the same *{{TOPIC_CONTINUOUS}}* communication path as regular CQ
> notifications
> Under heavy update load, CQ notifications may create a large communication
> backlog and delay {*}{{ContinuousRoutineStartResultMessage}}{*}. JFR and heap
> dump analysis showed millions of retained *{{GridContinuousMessage}}* and
> *{{CacheContinuousQueryEntry}}* objects and only one waiting
> *{{ContinuousRoutineStartResultMessage}}*
> As a result, _{{cache.query(qry)}}_ may hang for a long time while waiting
> for *{{ContinuousRoutineStartResultMessage}}* from the communication queue
> instead of discovery
> UPD: The same problem with zookeeper and immutable path.
> * Download and apply patch from attachments
> * Run zkTest#testConcurrentUpdatesAndQueryStartAtomicCacheGroup
> * See OOM
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
