[
https://issues.apache.org/jira/browse/IGNITE-28955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Vinogradov updated IGNITE-28955:
--------------------------------------
Description:
h3. What happens
A continuous query subscriber can be flooded with notifications. Nothing slows
the sender down: {{messageQueueLimit}} is 0 by default
({{TcpCommunicationSpi.DFLT_MSG_QUEUE_LIMIT}} =
{{GridNioServer.DFLT_SEND_QUEUE_LIMIT}}), which means unlimited. The server
sends a notification as soon as it creates one, and a subscriber that cannot
keep up only accumulates them on the heap.
h3. How it shows up
*{{CacheContinuousQueryConcurrentPartitionUpdateTest}}#concurrentUpdatesAndQueryStart*
runs 10 threads that update 3 caches in a loop until the listener has counted
enough events. With ZooKeeper discovery the client falls behind. One run on 12
cores and a 9.6G heap:
* the client received 15 534 790 messages (2.27 GB) in about 170 seconds;
* the pause detector logged 131 long JVM pauses, the longest 3.7 s;
* the client lost its ZooKeeper session and the coordinator evicted it.
With a smaller heap the same load ends in {{OutOfMemoryError}}, with millions
of retained *{{GridContinuousMessage}}* and *{{CacheContinuousQueryEntry}}* —
see the heap dump in IGNITE-28673. Both endings are the same overload.
h3. Why it matters beyond this test
IGNITE-28673 moves the completion of a continuous routine start off discovery
and onto the communication path, where it shares one connection with the
notifications ({{connectionsPerNode}} is 1 by default). Under this overload the
start result waits behind them. While the subscriber can be flooded, a fix
there that only changes message priority cannot be verified: the subscriber
dies before ordering starts to matter.
h3. Open question
Ignite already has {{slowClientQueueLimit}} for subscribers that cannot keep
up, but it is off by default
({{TcpCommunicationConfiguration#slowClientQueueLimit}} is 0). Whether the
answer is a different default, back pressure inside the continuous query path,
or both, needs a decision.
was:
Starting a continuous query can hang for minutes on a cluster that uses
ZooKeeper discovery while the same caches are under a heavy update load.
h3. Reproduction
Add a subclass of {{CacheContinuousQueryConcurrentPartitionUpdateTest}} that
runs under ZooKeeper discovery (the patch attached to IGNITE-28673 does exactly
that) and run {{testConcurrentUpdatesAndQueryStartAtomicCacheGroup}}.
On current master it does not finish: 318 s against the 300 s test timeout. TCP
discovery is not affected - the same test class passes in 45 s.
h3. Root cause
The message that completes the routine start and the continuous query
notifications share a topic and a pool:
{code:java}
// GridContinuousProcessor, sending the start result
ctx.io().sendToGridTopic(node, TOPIC_CONTINUOUS, new
ContinuousRoutineStartResultMessage(...), SYSTEM_POOL);
// GridContinuousProcessor, sending CQ notifications
ctx.io().sendToGridTopic(node, TOPIC_CONTINUOUS, msg, SYSTEM_POOL, ackC);
{code}
Under a heavy update load the notifications fill the queue, and the single
{{ContinuousRoutineStartResultMessage}} waits behind them - head-of-line
blocking. {{cache.query(qry)}} waits for that message, so the caller sees a
hang. The analysis in IGNITE-28673 shows the same picture from a heap dump:
millions of retained {{GridContinuousMessage}} and
{{CacheContinuousQueryEntry}} objects and one waiting
{{ContinuousRoutineStartResultMessage}}.
h3. Why only ZooKeeper today
{{GridContinuousProcessor}} picks the path by {{immutableDiscoCustomMsg =
!ctx.discovery().mutableCustomMessages()}}, which reads the
{{@DiscoverySpiMutableCustomMessageSupport}} annotation of the configured SPI.
{{TcpDiscoverySpi}} has it, so the routine start is acknowledged over discovery
and never touches the communication queue. {{ZookeeperDiscoverySpi}} does not,
so it already uses the communication path - and hits this.
h3. Why it matters beyond ZooKeeper
IGNITE-28673 removes the mutable path and makes the communication one the only
path. Merging it without fixing this would extend the hang to TCP discovery,
that is, to everyone. So this blocks IGNITE-28673.
Summary: Continuous query notifications can overload the subscriber
(was: Continuous query start hangs under load with ZooKeeper discovery)
> Continuous query notifications can overload the subscriber
> ----------------------------------------------------------
>
> Key: IGNITE-28955
> URL: https://issues.apache.org/jira/browse/IGNITE-28955
> Project: Ignite
> Issue Type: Bug
> Reporter: Anton Vinogradov
> Priority: Major
> Labels: ise
> Fix For: 2.19
>
>
> h3. What happens
> A continuous query subscriber can be flooded with notifications. Nothing
> slows the sender down: {{messageQueueLimit}} is 0 by default
> ({{TcpCommunicationSpi.DFLT_MSG_QUEUE_LIMIT}} =
> {{GridNioServer.DFLT_SEND_QUEUE_LIMIT}}), which means unlimited. The server
> sends a notification as soon as it creates one, and a subscriber that cannot
> keep up only accumulates them on the heap.
> h3. How it shows up
> *{{CacheContinuousQueryConcurrentPartitionUpdateTest}}#concurrentUpdatesAndQueryStart*
> runs 10 threads that update 3 caches in a loop until the listener has
> counted enough events. With ZooKeeper discovery the client falls behind. One
> run on 12 cores and a 9.6G heap:
> * the client received 15 534 790 messages (2.27 GB) in about 170 seconds;
> * the pause detector logged 131 long JVM pauses, the longest 3.7 s;
> * the client lost its ZooKeeper session and the coordinator evicted it.
> With a smaller heap the same load ends in {{OutOfMemoryError}}, with millions
> of retained *{{GridContinuousMessage}}* and *{{CacheContinuousQueryEntry}}* —
> see the heap dump in IGNITE-28673. Both endings are the same overload.
> h3. Why it matters beyond this test
> IGNITE-28673 moves the completion of a continuous routine start off discovery
> and onto the communication path, where it shares one connection with the
> notifications ({{connectionsPerNode}} is 1 by default). Under this overload
> the start result waits behind them. While the subscriber can be flooded, a
> fix there that only changes message priority cannot be verified: the
> subscriber dies before ordering starts to matter.
> h3. Open question
> Ignite already has {{slowClientQueueLimit}} for subscribers that cannot keep
> up, but it is off by default
> ({{TcpCommunicationConfiguration#slowClientQueueLimit}} is 0). Whether the
> answer is a different default, back pressure inside the continuous query
> path, or both, needs a decision.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
