RockteMQ-AI commented on issue #546: URL: https://github.com/apache/rocketmq-connect/issues/546#issuecomment-5235134230
**Issue Evaluation** Category: `bug` | Status: **Confirmed** The data loss occurs because `RocketMQSourceTask` puts consumed messages into an in-memory `blockingQueue` before they are checkpointed. If the connector crashes or restarts, all messages in the queue that haven't been checkpointed are lost. **Root Cause:** The offset is committed after messages are queued, but the queue itself is not persisted. On restart, the last committed offset causes re-reading from that point, but messages already consumed and queued (but not yet delivered to the sink) are lost. **Impact:** Violates at-least-once delivery guarantee. Data loss on connector restart. **Severity:** High — data integrity issue in production. **Suggested approaches:** 1. Implement write-ahead logging for the queue 2. Use synchronous offset commits (commit only after sink acknowledges) 3. Implement exactly-once semantics via transactional offset management --- *Automated evaluation by github-manager* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
