nzw921rx opened a new issue, #13137:
URL: https://github.com/apache/seatunnel/issues/13137

   ### Search before asking
   
   - [x] I searched existing issues and pull requests and found no report of 
this specific savepoint-restore queue-instance mismatch.
   
   ### What happened
   
   After a streaming MySQL CDC job was stopped with a savepoint and restored 
using the **same job ID**, the realtime edge API reported more queue instances 
than the current physical task mapping contained.
   
   The job captures seven tables into a JDBC MultiTableSink. Configured 
parallelism is 5 and incremental parallelism is 1. Parallelism was not changed 
during restore.
   
   **Current task mapping:**
   
   - TaskGroup IDs `2, 3, 4, 5, 6`: five queue-bearing groups. Each contains 
one `SourceSeaTunnelTask` and one `TransformSeaTunnelTask`, sharing the 
intermediate queue within that group.
   - TaskGroup ID `1`: one `SourceSplitEnumeratorTask`, which does not 
contribute an instance of the Sink input queue.
   - Worker addresses and runtime task IDs are omitted.
   
   **Direct engine response for queue -9, targeting Sink vertex 4**, bypassing 
the custom Prometheus exporter and Grafana (sanitized excerpt; timestamps 
omitted):
   
   ```json
   {
     "bucketMs": 5000,
     "edges": [{
       "queueId": -9,
       "targetVertexId": 4,
       "points": [{
         "queueSize": 2048,
         "subtaskCount": 7,
         "queueFillRatio": 0.14285714285714285,
         "queueCapacity": 14336,
         "emitBlockedNs": 4507915406,
         "bpRatio": 0.12879758302857142
       }]
     }]
   }
   ```
   
   The instance count and capacity persisted across the returned buckets. Five 
default-capacity queues would total `5 * 2048 = 10240`; the response instead 
represents `7 * 2048 = 14336`. The extra two contributions have not yet been 
identified.
   
   The ratio `2048 / 14336 = 14.2857%` is internally consistent. **The reported 
problem is the instance-count/aggregation-scope mismatch**, not aggregate 
capacity exceeding a single queue's capacity. One active incremental reader 
does not imply that other planned tasks or metric contexts disappear.
   
   **Observed sequence / reproduction outline** (not yet reduced to a 
standalone reproducer):
   
   1. Run a streaming MySQL CDC -> Transform -> JDBC MultiTableSink job with 
seven captured tables, parallelism 5 and realtime observability enabled.
   2. Let it reach the incremental binlog phase (`incremental.parallelism=1`).
   3. Stop with a savepoint and restore under the same job ID, keeping 
parallelism unchanged.
   4. Compare the realtime edge API with the current physical task mapping 
using the diagnostic requests below.
   5. In the observed run, the mapping lists five queue-bearing groups, while 
the edge reports `subtaskCount=7` and `queueCapacity=14336`.
   
   **Expected:** Realtime queue counts, capacity and ratio denominators should 
use a consistent, explainable set of physical instances for the current 
execution. If historical instances are retained intentionally, distinguish them 
from current-execution realtime statistics.
   
   **Investigation pointers; root cause is not confirmed:** 
`TaskExecutionService.provideDynamicMetrics()` includes both 
`finishedExecutionContexts` and `executionContexts`. The realtime collection 
filters by job ID and metric prefix, then aggregates by job/queue ID without 
retaining the contributing worker/task-group identity. Retained contexts after 
same-ID restore are a possibility to investigate, not an established cause. 
Task thread/resource leakage and data-correctness impact have not been 
established.
   
   Related: #11495 addresses metric reset after failed-pipeline recovery. This 
report concerns live queue-instance counts after same-ID savepoint resume; 
whether the causes overlap remains to be checked.
   
   ### SeaTunnel Version
   
   3.1.0
   
   ### SeaTunnel Config
   
   Relevant settings only. Connection credentials, endpoints and business table 
names are omitted. Seven source tables feed a JDBC MultiTableSink.
   
   ```hocon
   env {
     job.mode = "STREAMING"
     parallelism = 5
     engine {
       observability {
         enabled = true
         bucket_ms = 5000
         retention_minutes = 3
       }
     }
   }
   source {
     MySQL-CDC {
       parallelism = 5
       incremental.parallelism = 1
       # Connection and table settings omitted.
     }
   }
   ```
   
   ### Running Command
   
   The exact submit/resume command was not captured. The job was stopped with a 
savepoint and resumed under the same job ID. These read-only requests obtained 
the evidence; replace placeholders:
   
   ```bash
   curl -fsS 
"http://<master-host>:8080/metrics/realtime/jobs/<job-id>/edges?windowMs=30000"
   curl -fsS 
"http://<master-host>:5801/hazelcast/rest/maps/trace/task-mapping/<job-id>"
   ```
   
   ### Error Exception
   
   No exception is required to observe the mismatch. The job remains RUNNING. 
The observed problem is in monitoring statistics; data-correctness impact has 
not been established.
   
   ### Zeta or Flink or Spark Version
   
   Zeta engine bundled with SeaTunnel 3.1.0.
   
   ### Java or Scala Version
   
   Not collected.
   
   ### Screenshots
   
   Omitted to avoid exposing infrastructure and business identifiers. Sanitized 
numeric evidence is included above.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to