stinger1206 opened a new issue, #114:
URL: https://github.com/apache/doris-kafka-connector/issues/114

   # [Issue] Writer selection by string prefix + MAX offset aggregation can 
lose/stall records
   
   ## Summary
   
   Two related defects in `DorisDefaultSinkService` (26.0.0, 26.1.0, master):
   
   ### 1. String-prefix writer selection aggregates unrelated topic-partitions
   
   Writers are keyed `topic_partition[_table]` and selected with
   `entry.getKey().startsWith(topic + "_" + partition)`. Because one key can be 
a
   string-prefix of another, this matches the wrong writers:
   
   - `topic_test` partition 1 (`topic_test_1...`) also matches partition 10's 
writers
     (`topic_test_10...`);
   - topic `X` partition 1 collides with topic `X_1` partition 0.
   
   `getOffset()` then aggregates unrelated partitions — reporting a buffered 
writer's offset as
   another partition's committed offset (stalled or wrong commits), and 
`commit()` commits
   another partition's writers.
   
   ### 2. MAX offset aggregation loses buffered records (multi-table routing)
   
   With `record.tablename.field`, one topic-partition routes to multiple 
tables, one writer
   each. `getOffset()` returned `max(writer.getOffset())` — safe only for the 
most advanced
   writer. Records still buffered in a lagging writer are below that offset; 
after the offset
   is committed and the process crashes, they are never re-consumed.
   
   ```text
   offset 100 -> table B (still buffered in writer B)
   offset 101 -> table A (flushed + committed in writer A)
   getOffset() = MAX = 102  ->  committed  ->  crash  ->  record 100 for table 
B is lost
   ```
   
   ## Impact
   
   - Defect 1: any deployment whose topic/partition names can prefix-collide 
(partition 1 vs
     10 of the same topic is enough). Wrong offsets for both `getOffset` and 
`commit`.
   - Defect 2: deployments using `record.tablename.field`. Silent record loss 
on crash.
   - A naive MAX->MIN replacement is not a fix: an idle table writer would 
stall the
     topic-partition's committed offset indefinitely.
   
   ## Suggested fix
   
   1. Select writers by exact (topic, partition) identity — e.g. a structured 
writer key with
      `equals`/`hashCode` — not string-prefix matching.
   2. Report a contiguous safe watermark per topic-partition:
      - if any writer still buffers records: the minimum first-buffered offset 
across writers;
      - else: the max committed offset across writers.
   
   `DorisSinkTask.preCommit` calls `commit()` (durable flush/2PC, throws on 
failure) before
   `getOffset()`, so "empty buffer + committed offset" at getOffset time means 
durably loaded;
   see the PR for the full lifecycle analysis.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to