stinger1206 opened a new pull request, #113:
URL: https://github.com/apache/doris-kafka-connector/pull/113

   
   Fixes #111
   
   ## Problem
   
   In combined-flush mode a background stream load runs on a single executor 
thread
   (`AsyncDorisStreamLoad.LoadAsyncExecutor`). When a load fails (BE 
restarting, connection
   refused, a timeout, a failed Doris load result), the failure is dropped 
together with its
   data, and the next offset commit runs past the lost records:
   
   1. `LoadAsyncExecutor.run()` catches the failure, stores it in `exception`, 
clears
      `flushQueue` and exits. The failed `RecordBuffer` is gone from memory — 
the writer
      already swapped its own buffer when it enqueued it 
(`AsyncStreamLoadWriter.flushBuffer`).
   2. The next `put()` calls `insert()`, which throws from 
`checkFlushException()`.
      `DorisSinkTask.put()` catches it, calls `sink.init()` — which restarts 
the executor and
      resets `exception` to null — and throws `RetriableException`.
   3. Kafka Connect re-delivers only the batch of that `put()`. The records of 
the failed
      buffer came from earlier `put()` calls and are never re-delivered.
   4. The next `preCommit()` flushes the new buffer, and 
`DorisCombinedSinkService.getOffset()`
      returns the offsets from `topicPartitionOffset` — the last offsets *seen* 
by `insert()`,
      not confirmed by Doris. The commit succeeds past the lost records.
   
   The reporter's reproduction (issue #111) lost 29-39% of all rows this way 
while the tasks
   stayed RUNNING.
   
   ## Change
   
   `AsyncDorisStreamLoad` now latches the failure for the lifetime of the 
object:
   
   - `LoadAsyncExecutor.run()` sets a `loadFailed` latch in its catch blocks, 
next to the
     existing `exception` reference and the queue clear.
   - `checkFlushException()` throws not only while `exception` is set, but also 
while the latch
     is set. `start()` keeps restarting the executor thread and keeps clearing 
`exception`
     (unchanged), but it cannot clear the latch.
   
   Consequences, in order of the four steps above:
   
   - After the failure, every `insert()`, `flush()` and `forceFlush()` on the 
affected writer
     throws `DorisException` (RuntimeException), including across executor 
restarts.
   - `DorisSinkTask.put()` retries stay `RetriableException` until 
`max.retries` is exhausted;
     then the original exception escapes `put()` and the task fails. Kafka 
Connect does not
     restart a failed task by itself — the task stays FAILED until it is 
restarted (manually
     or by external automation). On restart, new writers are created and 
consumption resumes
     from the last committed offsets.
   - While the task is still running, `preCommit()` can never return the unsafe 
offsets: it
     calls `sink.commit(offsets)` first, which calls `commitFlush()` → 
`forceFlush()` →
     `checkFlushException()` and throws before `getOffset()` is consulted. 
Kafka Connect's
     `WorkerSinkTask.commitOffsets()` catches such an exception, skips the 
commit and rewinds
     to the last committed offsets; it does not fail the task — the task keeps 
running and
     keeps failing on `put()` and `preCommit()` until the `put()` retries are 
exhausted.
   
   Two different restarts are involved, and only the second one recovers:
   
   - **Executor restart** (`DorisSinkTask.put()` → `sink.init()` →
     `AsyncStreamLoadWriter.start()` → `AsyncDorisStreamLoad.start()`): 
re-`execute`s the
     executor runnable and clears `exception`, but **cannot clear the 
`loadFailed` latch**.
     It keeps the connector failing on every redelivered `put()` and on every 
`preCommit()`;
     it is not a recovery.
   - **Kafka Connect task-instance restart** (after the failed task is 
restarted — manually
     or by external automation — or after a rebalance): creates new writers and 
therefore new
     `AsyncDorisStreamLoad` instances with fresh latches, and resumes 
consumption from the
     last committed offsets — this is the recovery, and it re-delivers 
everything since the
     last commit.
   
   A failed load does not prove the records were not written: if the HTTP 
response was lost
   (timeout, connection cut), the load outcome is unknown and the re-delivered 
records may
   already be in Doris. Recovery is therefore at-least-once; with a primary-key 
table and
   Debezium-style upserts, re-delivery is idempotent (duplicate rows are 
otherwise possible
   for tables without a unique key).
   
   `forceFlush()` (the `preCommit` barrier) also cannot miss an in-flight 
failure:
   `waitAsyncLoadFinish()` enqueues two eof markers into the capacity-1 flush 
queue served by
   a single sequential executor thread, so the second marker's `put` cannot 
complete until
   the executor has polled the first — which happens only after `load()` 
returned. If that
   load failed, `exception`/`loadFailed` are set before the blocked `put` 
unblocks, and the
   `checkFlushException()` right after it throws. Covered by
   `forceFlushWaitsForInFlightLoadAndSurfacesItsFailure`, which pins the HTTP 
request with a
   `CountDownLatch`, calls `forceFlush()` on a separate thread, and asserts it 
neither
   returns while the request is in flight nor returns normally after the 
request fails.
   
   This is deliberately the "fail the commit/task" variant of the fix suggested 
in #111: the
   failed buffer is not re-sent (its records are no longer available for 
retry), so the only
   safe recovery is replaying from the last committed offsets.
   
   ## Relation to PR #112
   
   PR #112 makes every unexpected HTTP response (non-200, null entity, empty 
body) fail the
   load in the first place. This PR is independent of it: it handles every 
failure that does
   throw out of the load path (connection refused, timeout, Doris 
`Failed`/`Cancelled` JSON).
   With both PRs, the proxy-503 scenario of #110 also reaches this latch 
instead of returning
   normally.
   
   ## Tests
   
   - `TestAsyncStreamLoadFailureLatch` (5 tests):
     - a Doris `Failed` load result latches: the failure surfaces on 
`forceFlush()`, and
       survives `start()` (the `sink.init()` executor restart) for 
`checkException()`,
       `flush()` and `forceFlush()`;
     - the same for an `IOException` from the HTTP client (connection refused 
while the BE
       restarts);
     - `forceFlush()` cannot return while a load is still in flight, and 
surfaces that load's
       failure when it completes: the HTTP request is pinned by a 
`CountDownLatch`, and a
       separate thread calling `forceFlush()` stays blocked until the request 
fails, then
       receives the `DorisException`;
     - a normal `close()` interrupting an idle executor poll does not latch;
     - successful loads do not latch: two full flush cycles succeed, and a 
restart is
       harmless.
   - `TestCombineFlushFailedLoadCommits` (2 tests, real 
`DorisCombinedSinkService` /
     `DorisSinkTask`, only the HTTP layer mocked via the injectable-httpClient 
constructor):
     - service level: after a failed load, `commit()` throws, and keeps 
throwing across
       `init()` and further inserts, while `getOffset()` still reports the 
last-seen offset —
       reachable only through a successful `commit()`, which `preCommit` runs 
first;
     - task level: `put()` buffers, `preCommit()` throws, each re-delivered 
`put()` throws
       `RetriableException` until `max.retries` (2 in the test) is exhausted, 
then the
       original `DorisException` escapes (the task fails; on restart, 
consumption resumes
       from the last committed offsets), and `preCommit()` still never returns 
offsets.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to