SEPURI-SAI-KRISHNA commented on issue #13137:
URL: https://github.com/apache/seatunnel/issues/13137#issuecomment-6682629726

   I reproduced this on current `dev` (`df59bb4cd1`) without MySQL. In this 
reproduction the extra instances are not retained task group contexts. I do not 
plan to open a PR.
   
   Job: FakeSource (`parallelism = 5`, `split.num = 5`, `row.num = 100000000`, 
`split.read-interval = 100`) -> Sql -> Console, streaming, realtime 
observability on, default `queue-type`, JDK 11. `subtaskCount` after each 
same-ID savepoint stop and restore, no failover:
   
   ```
             start  1st  2nd  3rd  4th
   1 node    5      6    7    9    9
   1 node    5      8    8    9    9
   2 nodes   5      7    9    9    9
   2 nodes   5      5    6    6    -
   ```
   
   Row 3 shows the same `7 / 14336` as the report. Row 2 had logging added. At 
every probe the task mapping had 5 queue groups and a class histogram showed 6 
live `TaskGroupContext` objects in the cluster: those 5 plus the enumerator.
   
   **Cause.** `getQueueCache` registers the queue counters on the metrics 
context of the task that reaches it first. A group's two tasks initialize on 
separate worker threads, so the winner can change per attempt. 
`SeaTunnelTask.init()` reloads each task's context from the metrics snapshot 
store, which `removeMetricsContext` keeps on a savepoint end. If the other task 
wins after a restore, the earlier owner still reports its old counters, so the 
group has two rows for one queue. No worker move is needed.
   
   I logged the registrations and the rows seen by `RawAggConsumer.consumeLong` 
on 1 node through 5 restores. The number of tasks that had ever registered 
matched `subtaskCount` in all 6 attempts (5, 6, 7, 7, 9, 9). Size rows of tasks 
that did not register in that attempt stayed at 0, so `queueFillRatio` and 
`bpRatio` read low: 5/7 of their value when the count is 7.
   
   **Possible fix.** A throwaway prototype removes `IntermediateQueueSize*`, 
`IntermediateQueuePutBlockedNs#*` and `IntermediateQueueCapacity#*` in 
`getOrCreateMetricsContext`, since that queue no longer exists. One run on 2 
nodes stayed at `5 / 10240` for all 7 restores. Not a tested patch. 
`--get_running_job_metrics` merges job history and still lists the old rows, so 
that view needs its own decision.
   
   **Limits.** I did not run MySQL CDC or its idle reader close, so I cannot 
say this explains both extra instances in the report. The disruptor queue 
registers the same way, but I did not run it.
   
   Client arguments:
   
   ```
   -c job.conf --async --set-job-id 13137
   -s 13137
   -c job.conf -r 13137 --async
   ```
   
   I can post the config and raw responses if useful.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to