Hello all, The metadata blindness and consumer downtime windows are inherently unpredictable, making a fixed by_duration approach unviable. No static duration can accommodate both extremes:
- Short durations fail during extended downtimes, leading to the exact silent data loss this KIP aims to prevent. - Long durations (calibrated for worst-case scenarios) cause routine restarts to trigger unnecessary mass replays, undermining the latest offset reset policy. This is why the proposed solution uses creation time. It provides a discrete, deterministic signal—verifying whether a partition is actually newer than the group, rather than relying on an arbitrary time window that has no correlation with the consumer’s actual downtime. Best Regards, Jiunn-Yang > Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： > > Hi all, > > I'd like to revisit Jun previous comment about using by_duration=5s. I > realize now that having a small backlog is actually an acceptable trade-off, > especially since the latest policy inherently brings a tiny backlog to > starting consumers anyway due to the time gap between fetching the offset and > actually consuming. > > If we bring the simple by_duration=5s solution back to the KIP, I'd like to > discuss a further step: Should we just change the default policy from latest > to by_duration=5s specifically for the new consumer and share consumer? > > Best, > Chia-Ping > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: >> hi Jun >> >> The most important part of this story is how users should expect the data >> they can see when using the latest or by_duration policy with expanded >> partitions. >> >> Yes, the by_duration policy can minimize data loss, but it is >> non-deterministic, which means users will either read too many historical >> records from existing partitions or lose some records from expanded >> partitions. >> >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to >> understand, and that is why I preferred having a whole new policy based >> entirely on group creation time (KIP-1282) >> >> Best, >> Chia-Ping >> >> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： >> >>> Hi, Chia-Ping and Jiunn-Yang, >>> >>> Thanks for the reply. I am still trying to understand the value of the new >>> configs with the KIP. >>> >>> The motivation of the KIP is that a user doesn't want to miss the data if >>> the backlog is small. The backlog of the existing partition is easy to >>> understand because it relates to retention time. The backlog for the new >>> partition is a bit subtle to understand since it depends on the metadata >>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs to >>> understand the metadata refresh delay on the consumer side and use it to >>> set the config. >>> >>> Now, let's consider the alternative: setting the same value for the >>> existing by_duration policy. The KIP lists three issues with this approach. >>> 1. It computes the seek target client-side as now() - duration, which >>> introduces clock skew across consumers and forces operators to choose >>> overly large durations, causing unnecessary reprocessing. >>> 2. The target timestamp is recomputed on each retry, so failed >>> ListOffsetsRequest retries can shift the target forward and potentially >>> miss records produced between attempts. >>> 3. It applies uniformly to all partitions without committed offsets, and >>> cannot distinguish newly expanded partitions from long-existing partitions >>> newly assigned to the group, leading to unnecessary replay. >>> >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to >>> the metadata refresh delay. We could also consider improving the >>> implementation. For issue 3, the metadata refresh delay is typically low >>> (in the order of minutes with the classic consumer and tens of seconds with >>> the new consumer). If a user is ok with reading that much backlog for new >>> partitions, it seems they will be ok doing the same for existing >>> partitions. >>> >>> So, instead of introducing a new config, could we just reuse the existing >>> config with better documentation and/or implementation? >>> >>> Jun >>> >>> >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: >>> >>>> Hello Jun, >>>> >>>> You're right that group creation time is the more intuitive answer at >>>> first glance, >>>> the KIP's own motivation talks about partitions that "predate the group" >>>> vs partitions >>>> "created during group runtime," which directly points to a >>> group-lifecycle >>>> classifier. >>>> I'd like to walk through why we landed on partition age, and the >>>> trade-offs we considered. >>>> >>>> We evaluated three candidate signals: >>>> >>>> 1. `by_duration:5secs` >>>> >>>> This covers the metadata blindness window, but has issues the KIP >>>> currently documents >>>> under "Why not use `by_duration`?": >>>> >>>> - Client-side `now() - duration` introduces clock skew across consumers. >>>> - `ListOffsets` retries shift the target forward, potentially missing >>>> records produced between >>>> attempts. >>>> - It applies uniformly to all partitions without committed offsets, >>>> including pre-existing partitions >>>> newly assigned to the group, causing unnecessary replay. >>>> >>>> 2. Group creation time as classifier >>>> >>>> This works cleanly when the consumer is actively running. Our concern >>>> is the idle / late-rejoin case: >>>> >>>> T=0: Group created. >>>> T=1..T=100: Consumer idle (down, disconnected, etc.). >>>> T=50: Partition added during the idle window. >>>> T=100: Consumer resumes. >>>> >>>> Under group creation time, the new partition is classified as new >>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50. >>>> But during `[T=1, T=100]`, base partitions also accumulated data that >>>> the consumer accepts as lost — that is precisely the contract of >>>> `auto.offset.reset=latest`. There is no principled reason to treat >>>> the new partition differently; both contain backlog accumulated during >>>> the same idle window. >>>> >>>> This aligns with the "backlog is backlog” principle you raised in >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on >>>> every other partition during the same idle period; forcing 0-backlog >>>> tolerance only on new partitions would be inconsistent with that >>>> tolerance. >>>> >>>> 3. Partition age vs threshold >>>> >>>> Partition age corresponds to the actual silent data loss window, >>>> the gap between partition creation and the consumer’s metadata >>>> refresh. Within this window, data loss is genuinely silent: the >>>> consumer had no opportunity to know about the partition. Outside this >>>> window, missing data reflects either: >>>> >>>> - (a) the user’s tolerated cost of running with idle consumers, or >>>> - (b) an operational issue to surface via monitoring, not via reset >>> policy. >>>> >>>> We did not choose partition age because it is more elegant than group >>>> creation time — we chose it because its failure mode (requires a >>>> threshold) is >>>> less invasive than the failure mode of group creation time (overrides >>>> user-stated >>>> `latest` intent during idle periods). >>>> >>>> Best Regards, >>>> Jiunn-Yang >>>> >>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： >>>>> >>>>> Hi Jun, >>>>> >>>>> Relying on both creation times will create an inconsistent scenario. A >>>>> consumer that lost all offsets due to a long sleep will seek to the >>>>> beginning for the partitions created later than the group. >>>>> >>>>> That is why we initially proposed KIP-1282 to fix the inconsistency >>>> using a >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327 >>>> goes >>>>> back to using flexible configurations to prevent users from falling >>> into >>>>> that pitfall. >>>>> >>>>> Best, Chia-Ping >>>>> >>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 上午6:49寫道： >>>>> >>>>>> Hi, Jiunn-Yang, >>>>>> >>>>>> Thanks for the reply and sorry for the late reply. >>>>>> >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels weird to >>>> me. >>>>>> It >>>>>> categorizes partitions as new or existing based on the partition >>>> creation >>>>>> time. Intuitively, the categorization should be based on the group >>>> creation >>>>>> time: all partitions existing when the group is created are existing >>> and >>>>>> all partitions created after the group creation are new partitions. >>>>>> >>>>>> Jun >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Manually bumping this thread. If there is no further >>>>>>> discussion, I will close the vote. >>>>>>> >>>>>>> Best Regards, >>>>>>> Jiunn-Yang >>>>>>> >>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： >>>>>>>> >>>>>>>> Hello Jian, >>>>>>>> >>>>>>>> Thanks for your feedback, >>>>>>>> >>>>>>>> Agreed, partition expansion is a common operational task, not an >>> edge >>>>>>>> case. I've updated the Motivation section accordingly. >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Jiunn-Yang >>>>>>>> >>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： >>>>>>>>> >>>>>>>>> Hi Jiunn-Yang: >>>>>>>>> >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that this >>>>>> is a >>>>>>>>> common scenario rather than an edge case, which further >>> demonstrates >>>>>> the >>>>>>>>> need for this optimization. For example: >>>>>>>>> A partition expansion is a common operational task in Kafka: To >>>>>> balance >>>>>>>>> resource utilization and cost, topics are typically created with a >>>>>>> moderate >>>>>>>>> default partition count. However, as traffic grows over time, it is >>>>>>> often >>>>>>>>> necessary to increase the number of partitions to accommodate the >>>>>> higher >>>>>>>>> workload. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Jian >>>>>>>>> >>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： >>>>>>>>> >>>>>>>>>> Hello chia, >>>>>>>>>> >>>>>>>>>> Thanks for the comments, I have updated the KIP! >>>>>>>>>> >>>>>>>>>> Best Regards, >>>>>>>>>> Jiunn-Yang >>>>>>>>>> >>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 寫道： >>>>>>>>>>> >>>>>>>>>>> Hi Jiunn-Yang, >>>>>>>>>>> >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when >>> describing >>>>>>>>>>> partitions in the KIP? I understand you are using them to >>> describe >>>>>> the >>>>>>>>>>> "freshness" or the users' need for the records, but applying >>> these >>>>>>> terms >>>>>>>>>> to >>>>>>>>>>> the partition itself feels a bit unnatural. >>>>>>>>>>> >>>>>>>>>>> After all, in this scenario, users don't really care whether a >>>>>>> partition >>>>>>>>>> is >>>>>>>>>>> newly expanded or not. Their only expectation is that they won't >>>>>>> silently >>>>>>>>>>> lose any live records produced to the topic during their active >>>>>>>>>> consumption. >>>>>>>>>>> >>>>>>>>>>> Best, Chia-Ping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： >>>>>>>>>>> >>>>>>>>>>>> Hello Jun, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation >>>> section. >>>>>>>>>>>> >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>> >>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 凌晨1:12 寫道： >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation for >>>>>> the >>>>>>>>>> KIP. >>>>>>>>>>>>> >>>>>>>>>>>>> The KIP says "The core insight is that not all partitions >>> without >>>>>> a >>>>>>>>>>>>> committed offset are the same. A newly expanded partition (hot) >>>> is >>>>>>>>>>>>> fundamentally different from a partition the consumer has never >>>>>> seen >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot partition >>>>>>>>>>>>> fundamentally different from the cold? >>>>>>>>>>>>> >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also >>>> insufficient >>>>>>>>>>>> because: >>>>>>>>>>>>> >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across >>> nodes >>>>>>> due >>>>>>>>>>>> to >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large >>> duration, >>>>>>>>>>>> causing >>>>>>>>>>>>> unnecessary reprocessing. >>>>>>>>>>>>> - On network errors, the client recalculates the seek time on >>>>>> retry, >>>>>>>>>>>>> shifting the target timestamp forward and risking data loss." >>>>>>>>>>>>> >>>>>>>>>>>>> However, both of these situations are rare. If these issues >>>>>> persist, >>>>>>>>>> more >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't >>>>>> need a >>>>>>>>>>>> common >>>>>>>>>>>>> solution. If users care about those rare situations, they can >>>>>>> implement >>>>>>>>>>>>> customized logic using >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). >>>>>>>>>>>>> >>>>>>>>>>>>> Jun >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <[email protected]> >>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello chia, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the feedback, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> If the creation time exists, the returned value should always >>>> be >>>>>>>>>>>> greater >>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>> field >>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>> >>>>>>>>>>>>>> The earliest point at which we can detect the version mismatch >>>> is >>>>>>>>>> during >>>>>>>>>>>>>> the >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside >>>>>> poll(). >>>>>>>>>>>>>> Therefore, the >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from >>> poll(). >>>>>>> I’ll >>>>>>>>>>>>>> clarify this in the KIP. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 下午4:50 寫道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> hi Jiunn >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this >>> partition >>>>>> in >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as >>>>>>>>>> broker_current_time >>>>>>>>>>>> - >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not >>>>>> support >>>>>>>>>> this >>>>>>>>>>>>>> feature or the partition creation time is unknown. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If the creation time exists, the returned value should always >>>> be >>>>>>>>>>>> greater >>>>>>>>>>>>>> than or equal to zero, right? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any >>>>>>> field >>>>>>>>>>>>>> absent UnsupportedVersionException >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling >>>>>>>>>> `poll()`? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: >>>>>>>>>>>>>>>> Hello Jun, chia, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the >>>>>>> discussion >>>>>>>>>>>>>>>> feedback. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset >>> behavior >>>>>>> from >>>>>>>>>>>>>>>> the base auto.offset.reset policy: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all >>>>>>> auto.offset.reset >>>>>>>>>>>>>> values >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to >>>>>>>>>>>>>> auto.offset.reset.new.partitions >>>>>>>>>>>>>>>> config setting >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base >>> auto.offset.reset >>>>>>>>>> policy >>>>>>>>>>>>>> continues >>>>>>>>>>>>>>>> to apply unchanged. >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a >>>> separate >>>>>>>>>>>>>> internal config >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to >>>>>> earliest). >>>>>>>>>> This >>>>>>>>>>>>>> decoupled design makes >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public >>>>>>> user-facing >>>>>>>>>>>>>> configuration in a future KIP. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 清晨7:46 >>> 寫道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed >>>> below: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default >>> value >>>>>> of >>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and >>>>>> latest, >>>>>>> and >>>>>>>>>>>> it >>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest. >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal >>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest >>>> already >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of other >>>>>>> values >>>>>>>>>>>> in a >>>>>>>>>>>>>> separate KIP. >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all >>> for >>>>>>>>>>>>>> consistency. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>> Hi, Chia-Ping, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the reply. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka >>>>>> topic >>>>>>> is >>>>>>>>>>>>>> expanded >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto >>> offset >>>>>>> reset >>>>>>>>>>>>>> policy >>>>>>>>>>>>>>>>>> will silently miss all records produced to those >>> partitions >>>>>>> before >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss >>>>>> issue >>>>>>>>>> could >>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>> happen, right? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will >>>> take >>>>>>> the >>>>>>>>>>>> same >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it >>>>>>> by_duration if >>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < >>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> hi Jun >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something >>>>>>> important >>>>>>>>>>>> from >>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify >>> with >>>>>> a >>>>>>> few >>>>>>>>>>>>>> questions: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to >>>>>>> other >>>>>>>>>>>> reset >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or >>>>>> by_duration >>>>>>>>>>>> don't >>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a >>>>>>> partition >>>>>>>>>> is >>>>>>>>>>>>>> expanded. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to >>>>>> earliest >>>>>>> or >>>>>>>>>>>>>> latest, >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For example, >>>>>> if a >>>>>>>>>>>>>> consumer is >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created during >>>>>> that >>>>>>>>>>>>>> downtime, >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when >>>>>> resuming, >>>>>>>>>>>> rather >>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is >>>>>>> technically >>>>>>>>>>>>>> "new" to >>>>>>>>>>>>>>>>>>> the group. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age >>>>>>> threshold. >>>>>>>>>>>> It >>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely >>>> "hot/new" >>>>>>> and >>>>>>>>>>>>>> what is >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It >>>>>> only >>>>>>>>>>>>>> applies when >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the >>>>>>>>>> motivation >>>>>>>>>>>>>>>>>>> equally >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values >>> like >>>>>>>>>>>>>> by_duration. >>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to >>>> control >>>>>>>>>> newly >>>>>>>>>>>>>> created >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts. >>>>>>> Have we >>>>>>>>>>>>>>>>>>> considered >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new >>>> .partitions? >>>>>>> If >>>>>>>>>>>> this >>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to >>> the >>>>>>>>>> policy >>>>>>>>>>>>>> used >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to >>>>>>>>>> customize >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> behavior for new partitions. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Jun >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <[email protected]> >>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The >>>>>>> PartitionMetadata >>>>>>>>>>>>>> struct >>>>>>>>>>>>>>>>>>>>> gains a new >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed >>>>>>> server-side >>>>>>>>>>>> by >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> broker as >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when >>>>>>> MembershipManager >>>>>>>>>>>>>> detects >>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>> newly assigned >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for >>>> the >>>>>>>>>>>> affected >>>>>>>>>>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the >>>>>> topic >>>>>>> ID >>>>>>>>>> is >>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>> in the cache. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> MB0: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported >>>>>>>>>>>>>> MetadataResponse >>>>>>>>>>>>>>>>>>>>> version via the >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the >>>>>>> negotiated >>>>>>>>>>>>>>>>>>> version is >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at >>> all >>>>>> and >>>>>>>>>> can >>>>>>>>>>>>>>>>>>> throw an >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to >>> latest >>>>>>> and >>>>>>>>>>>>>> risking >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 >>>> 下午4:04 >>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> hi David >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' >>>>>> resolution >>>>>>>>>> from >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control >>>>>> plane >>>>>>>>>>>> clean. >>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>> main >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing >>>> inter-broker >>>>>>>>>> clock >>>>>>>>>>>>>> skew. >>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of >>>>>> truth >>>>>>>>>> for >>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be >>>>>>> negligible. >>>>>>>>>>>>>> Given >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in >>>> minutes >>>>>>> or >>>>>>>>>>>>>> hours, a >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers >>> won't >>>>>>> impact >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> fallback decision. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> 於 >>>>>> 2026年4月29日 >>>>>>>>>>>> 下午3:29 >>>>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous >>>>>>> conversation >>>>>>>>>>>> but I >>>>>>>>>>>>>>>>>>> took a >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; >>> the >>>>>>> KIP >>>>>>>>>>>>>> states >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean >>> is >>>>>>> set. >>>>>>>>>>>>>>>>>>> Implicitly, >>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new >>> partition, >>>>>> it >>>>>>>>>> will >>>>>>>>>>>>>>>>>>> issue a >>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. >>>> Is >>>>>>> my >>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also >>>>>>> explain >>>>>>>>>>>>>> how it >>>>>>>>>>>>>>>>>>>>> fits >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch >>>>>> offsets, >>>>>>>>>>>> etc.). >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place >>> for >>>>>>> this >>>>>>>>>>>> given >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could >>>>>> simply >>>>>>>>>>>>>> include >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious >>>> about >>>>>>> not >>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated >>>> concepts. >>>>>>> The >>>>>>>>>>>> API >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking >>>> partitions. >>>>>>> The >>>>>>>>>>>> fact >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams >>> API >>>>>>> also >>>>>>>>>>>>>> suggests >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we >>>>>>> want to >>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani >>>> via >>>>>>> dev >>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about >>> the >>>>>>> gap. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for >>>>>>> RequestPartitionAges >>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>> Can a >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on >>>>>>>>>>>>>>>>>>> TopicPartitions) >>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended >>>>>> value >>>>>>> or >>>>>>>>>> a >>>>>>>>>>>>>>>>>>> ConfigDef >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the >>>>>>>>>>>> metadata.max.age.ms >>>>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>> Sizing >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka >>>>>> Streams, >>>>>>>>>>>> would >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config >>>>>> auto.offset.reset.latest.max.age >>>>>>> to >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) >>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is >>> the >>>>>>> most >>>>>>>>>>>>>>>>>>> familiar >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly >>>>>>> configure >>>>>>>>>>>>>> it. Or >>>>>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls >>> back >>>>>> to >>>>>>>>>>>>>> earliest" >>>>>>>>>>>>>>>>>>>>> reads as >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition >>> which >>>>>>> isn't >>>>>>>>>>>>>>>>>>> supported. >>>>>>>>>>>>>>>>>>>>> May >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves >>>> the >>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>> position to >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was >>>>>>> applied >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is >>>>>>> unchanged. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>> Murali >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < >>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 >>> 2026年4月28日 >>>>>>> 晚上8:03 >>>>>>>>>>>> 寫道： >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation >>>> time >>>>>>> via >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> Admin >>>>>>>>>>>>>>>>>>>>> API? >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to >>> diagnose >>>>>> and >>>>>>>>>>>>>>>>>>> troubleshoot >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 >>>>>>> Prevent >>>>>>>>>> Hot >>>>>>>>>>>>>> Data >>>>>>>>>>>>>>>>>>>>> Loss >>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy >>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>> >>>> >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, >>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded >>>>>> (hot) >>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold >>>> automatically >>>>>>> fall >>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full >>>>>>> historical >>>>>>>>>>>>>>>>>>> reprocess. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >>
