nzw921rx opened a new issue, #13137:
URL: https://github.com/apache/seatunnel/issues/13137
### Search before asking
- [x] I searched existing issues and pull requests and found no report of
this specific savepoint-restore queue-instance mismatch.
### What happened
After a streaming MySQL CDC job was stopped with a savepoint and restored
using the **same job ID**, the realtime edge API reported more queue instances
than the current physical task mapping contained.
The job captures seven tables into a JDBC MultiTableSink. Configured
parallelism is 5 and incremental parallelism is 1. Parallelism was not changed
during restore.
**Current task mapping:**
- TaskGroup IDs `2, 3, 4, 5, 6`: five queue-bearing groups. Each contains
one `SourceSeaTunnelTask` and one `TransformSeaTunnelTask`, sharing the
intermediate queue within that group.
- TaskGroup ID `1`: one `SourceSplitEnumeratorTask`, which does not
contribute an instance of the Sink input queue.
- Worker addresses and runtime task IDs are omitted.
**Direct engine response for queue -9, targeting Sink vertex 4**, bypassing
the custom Prometheus exporter and Grafana (sanitized excerpt; timestamps
omitted):
```json
{
"bucketMs": 5000,
"edges": [{
"queueId": -9,
"targetVertexId": 4,
"points": [{
"queueSize": 2048,
"subtaskCount": 7,
"queueFillRatio": 0.14285714285714285,
"queueCapacity": 14336,
"emitBlockedNs": 4507915406,
"bpRatio": 0.12879758302857142
}]
}]
}
```
The instance count and capacity persisted across the returned buckets. Five
default-capacity queues would total `5 * 2048 = 10240`; the response instead
represents `7 * 2048 = 14336`. The extra two contributions have not yet been
identified.
The ratio `2048 / 14336 = 14.2857%` is internally consistent. **The reported
problem is the instance-count/aggregation-scope mismatch**, not aggregate
capacity exceeding a single queue's capacity. One active incremental reader
does not imply that other planned tasks or metric contexts disappear.
**Observed sequence / reproduction outline** (not yet reduced to a
standalone reproducer):
1. Run a streaming MySQL CDC -> Transform -> JDBC MultiTableSink job with
seven captured tables, parallelism 5 and realtime observability enabled.
2. Let it reach the incremental binlog phase (`incremental.parallelism=1`).
3. Stop with a savepoint and restore under the same job ID, keeping
parallelism unchanged.
4. Compare the realtime edge API with the current physical task mapping
using the diagnostic requests below.
5. In the observed run, the mapping lists five queue-bearing groups, while
the edge reports `subtaskCount=7` and `queueCapacity=14336`.
**Expected:** Realtime queue counts, capacity and ratio denominators should
use a consistent, explainable set of physical instances for the current
execution. If historical instances are retained intentionally, distinguish them
from current-execution realtime statistics.
**Investigation pointers; root cause is not confirmed:**
`TaskExecutionService.provideDynamicMetrics()` includes both
`finishedExecutionContexts` and `executionContexts`. The realtime collection
filters by job ID and metric prefix, then aggregates by job/queue ID without
retaining the contributing worker/task-group identity. Retained contexts after
same-ID restore are a possibility to investigate, not an established cause.
Task thread/resource leakage and data-correctness impact have not been
established.
Related: #11495 addresses metric reset after failed-pipeline recovery. This
report concerns live queue-instance counts after same-ID savepoint resume;
whether the causes overlap remains to be checked.
### SeaTunnel Version
3.1.0
### SeaTunnel Config
Relevant settings only. Connection credentials, endpoints and business table
names are omitted. Seven source tables feed a JDBC MultiTableSink.
```hocon
env {
job.mode = "STREAMING"
parallelism = 5
engine {
observability {
enabled = true
bucket_ms = 5000
retention_minutes = 3
}
}
}
source {
MySQL-CDC {
parallelism = 5
incremental.parallelism = 1
# Connection and table settings omitted.
}
}
```
### Running Command
The exact submit/resume command was not captured. The job was stopped with a
savepoint and resumed under the same job ID. These read-only requests obtained
the evidence; replace placeholders:
```bash
curl -fsS
"http://<master-host>:8080/metrics/realtime/jobs/<job-id>/edges?windowMs=30000"
curl -fsS
"http://<master-host>:5801/hazelcast/rest/maps/trace/task-mapping/<job-id>"
```
### Error Exception
No exception is required to observe the mismatch. The job remains RUNNING.
The observed problem is in monitoring statistics; data-correctness impact has
not been established.
### Zeta or Flink or Spark Version
Zeta engine bundled with SeaTunnel 3.1.0.
### Java or Scala Version
Not collected.
### Screenshots
Omitted to avoid exposing infrastructure and business identifiers. Sanitized
numeric evidence is included above.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]