seokjin0414 opened a new pull request, #3099: URL: https://github.com/apache/iggy/pull/3099
## Summary Closes #2872 - Add integration test for the message deduplication pipeline (7-step scenario) - Fix server panic when all messages in a batch are duplicates - Fix partition offset calculation after dedup removes mid-batch messages - Fix deduplicator not being created for lazily-initialized partitions ### Bug fixes **Empty batch panic** (`messages.rs`): After `prepare_for_persistence()` removes all duplicate messages, subsequent `.unwrap()` calls on `first_timestamp()`, `last_timestamp()`, `last_offset()` panic. Added empty batch guard. **Offset calculation** (`messages.rs`): `last_offset` was computed as `current_offset + count - 1`, which doesn't account for offset gaps created by dedup removal. Changed to use `segment.end_offset` (the actual last offset from batch). **Deduplicator not created** (`partitions.rs`): `init_partition_inner()` hardcoded `None` for the `message_deduplicator` parameter. Added `create_message_deduplicator()` call matching the bootstrap path. ### Integration test scenario | Step | Description | Validates | |------|-------------|-----------| | 1 | Send 10 messages with id=0 (auto UUID) | All pass through with unique IDs | | 2 | Send 10 messages with explicit IDs 1-10 | Normal dedup registration | | 3 | Re-send IDs 1-10 with different payload | Duplicates rejected, original payload preserved | | 4 | Send all-duplicate batch | No server crash, count unchanged | | 5 | Send mixed batch (IDs 6-15) | Only new IDs 11-15 accepted | | 6 | Verify offsets | Monotonically increasing after dedup | | 7 | Wait for TTL expiry, re-send IDs 1-10 | Previously seen IDs accepted again | ## Test plan - [x] `cargo fmt --all -- --check` - [x] `cargo clippy -p server -p integration --all-targets -- -D warnings` - [ ] `cargo test -p integration --test mod -- message_deduplication` (CI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
