[
https://issues.apache.org/jira/browse/KAFKA-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832561#comment-17832561
]
Kamal Chandraprakash edited comment on KAFKA-16414 at 3/31/24 9:25 AM:
-----------------------------------------------------------------------
My suggestion is to keep the existing behavior for {{retention.bytes}} config.
Assume that the user configured the {{retention.bytes}} = 1 byte and default
{{segment.bytes}} = 1 GB. And, there is one record batch in the active segment
with total-size = 5 KB. If we rotate the active segment and delete it, then all
the data for that partition will be removed. Shouldn't we have to maintain the
one byte of data which is the last batch? Handling this case will be tricky, as
we have to split the active segment into two, keep only the segment with
last-batch, and it will also exceed the user-configured retention bytes.
For {{{}retention.ms{}}}, the behavior is correct since all the records/batches
in that segments are older than the user-configured retention time, so we mark
the active segment as eligible for deletion.
was (Author: ckamal):
My suggestion is to keep the existing behavior for {{retention.bytes}} config.
Assume that the user configured the {{retention.bytes}} = 1 byte and default
{{segment.bytes}} = 1 GB. And, there is one record batch in the active segment
with total-size = 5 KB. If we rotate the active segment and delete it, then all
the data for that partition will be removed. Shouldn't we have to maintain the
one byte of data which is the last batch? Handling this case will be tricky, as
we have to split the active segment into two.
For {{{}retention.ms{}}}, the behavior is correct since all the records/batches
in that segments are older than the user-configured retention time, so we mark
the active segment as eligible for deletion.
> Inconsistent active segment expiration behavior between retention.ms and
> retention.bytes
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-16414
> URL: https://issues.apache.org/jira/browse/KAFKA-16414
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 3.6.1
> Reporter: Kuan Po Tseng
> Assignee: Kuan Po Tseng
> Priority: Major
>
> This is a follow up issue on KAFKA-16385.
> Currently, there's a difference between how retention.ms and retention.bytes
> handle active segment expiration:
> - retention.ms always expire active segment when max segment timestamp
> matches the condition.
> - retention.bytes only expire active segment when retention.bytes is
> configured to zero.
> The behavior should be either rotate active segments for both retention
> configurations or none at all.
> For more details, see
> https://issues.apache.org/jira/browse/KAFKA-16385?focusedCommentId=17829682&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17829682
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
