Hi Jun, Thanks for the feedback. I agree with your conclusion. It would be better to "make haste" with a new KIP to address this. We can focus the discussion on the following points:
1) Should we change the default reset policy to by_duration for all consumers? 2) What is the optimal duration to prevent data loss while avoiding large backlogs? Note that the classic consumer has a 3-second heartbeat interval, whereas the async consumer uses 5 seconds. 3) What are the potential side effects of replacing latest with by_duration as the default behavior? Best, Chia-Ping On 2026/06/30 20:39:24 Jun Rao via dev wrote: > Hi, Chia-Ping, > > I agree that using by_duration with a value of 5s to 10s addresses most > common issues with expanded partitions, given the default > heartbeat.interval.ms is 3 seconds. We can consider setting that as the new > default. The current proposal may address some other rarer cases, but it > feels a bit like over engineering. > > Thanks, > > Jun > > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <[email protected]> wrote: > > > - Short durations fail during extended downtimes, leading to the exact > > silent data > > loss this KIP aims to prevent. > > > > Losing the records produced to expanded partitions during a downtime is > > actually the expected behavior for the latest policy. This is exactly why, > > in the current KIP proposal, we prefer by_duration over earliest for the > > expanded partitions—we want to avoid forcing users to consume a massive > > accumulated backlog just because a partition is technically "new". > > > > To quote my previous comment from May 15: > > "...if a consumer is offline for a while and a new partition is created > > during that downtime, the user might actually want to skip to latest when > > resuming, rather than reading from earliest just because the partition is > > technically 'new' to the group." > > > > On 2026/06/30 12:12:47 黃竣陽 wrote: > > > Hello all, > > > > > > The metadata blindness and consumer downtime windows are inherently > > > unpredictable, making a fixed by_duration approach unviable. No static > > > duration can accommodate both extremes: > > > > > > - Short durations fail during extended downtimes, leading to the exact > > silent data > > > loss this KIP aims to prevent. > > > - Long durations (calibrated for worst-case scenarios) cause routine > > restarts to > > > trigger unnecessary mass replays, undermining the latest offset reset > > policy. > > > > > > This is why the proposed solution uses creation time. It provides a > > discrete, > > > deterministic signal—verifying whether a partition is actually newer > > than the group, > > > rather than relying on an arbitrary time window that has no correlation > > with the consumer’s > > > actual downtime. > > > > > > Best Regards, > > > Jiunn-Yang > > > > > > > Chia-Ping Tsai <[email protected]> 於 2026年6月30日 下午3:05 寫道： > > > > > > > > Hi all, > > > > > > > > I'd like to revisit Jun previous comment about using by_duration=5s. I > > realize now that having a small backlog is actually an acceptable > > trade-off, especially since the latest policy inherently brings a tiny > > backlog to starting consumers anyway due to the time gap between fetching > > the offset and actually consuming. > > > > > > > > If we bring the simple by_duration=5s solution back to the KIP, I'd > > like to discuss a further step: Should we just change the default policy > > from latest to by_duration=5s specifically for the new consumer and share > > consumer? > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote: > > > >> hi Jun > > > >> > > > >> The most important part of this story is how users should expect the > > data > > > >> they can see when using the latest or by_duration policy with expanded > > > >> partitions. > > > >> > > > >> Yes, the by_duration policy can minimize data loss, but it is > > > >> non-deterministic, which means users will either read too many > > historical > > > >> records from existing partitions or lose some records from expanded > > > >> partitions. > > > >> > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to > > > >> understand, and that is why I preferred having a whole new policy > > based > > > >> entirely on group creation time (KIP-1282) > > > >> > > > >> Best, > > > >> Chia-Ping > > > >> > > > >> Jun Rao via dev <[email protected]> 於 2026年6月16日週二 上午1:08寫道： > > > >> > > > >>> Hi, Chia-Ping and Jiunn-Yang, > > > >>> > > > >>> Thanks for the reply. I am still trying to understand the value of > > the new > > > >>> configs with the KIP. > > > >>> > > > >>> The motivation of the KIP is that a user doesn't want to miss the > > data if > > > >>> the backlog is small. The backlog of the existing partition is easy > > to > > > >>> understand because it relates to retention time. The backlog for the > > new > > > >>> partition is a bit subtle to understand since it depends on the > > metadata > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs > > to > > > >>> understand the metadata refresh delay on the consumer side and use > > it to > > > >>> set the config. > > > >>> > > > >>> Now, let's consider the alternative: setting the same value for the > > > >>> existing by_duration policy. The KIP lists three issues with this > > approach. > > > >>> 1. It computes the seek target client-side as now() - duration, which > > > >>> introduces clock skew across consumers and forces operators to choose > > > >>> overly large durations, causing unnecessary reprocessing. > > > >>> 2. The target timestamp is recomputed on each retry, so failed > > > >>> ListOffsetsRequest retries can shift the target forward and > > potentially > > > >>> miss records produced between attempts. > > > >>> 3. It applies uniformly to all partitions without committed offsets, > > and > > > >>> cannot distinguish newly expanded partitions from long-existing > > partitions > > > >>> newly assigned to the group, leading to unnecessary replay. > > > >>> > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit > > buffer to > > > >>> the metadata refresh delay. We could also consider improving the > > > >>> implementation. For issue 3, the metadata refresh delay is typically > > low > > > >>> (in the order of minutes with the classic consumer and tens of > > seconds with > > > >>> the new consumer). If a user is ok with reading that much backlog > > for new > > > >>> partitions, it seems they will be ok doing the same for existing > > > >>> partitions. > > > >>> > > > >>> So, instead of introducing a new config, could we just reuse the > > existing > > > >>> config with better documentation and/or implementation? > > > >>> > > > >>> Jun > > > >>> > > > >>> > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <[email protected]> wrote: > > > >>> > > > >>>> Hello Jun, > > > >>>> > > > >>>> You're right that group creation time is the more intuitive answer > > at > > > >>>> first glance, > > > >>>> the KIP's own motivation talks about partitions that "predate the > > group" > > > >>>> vs partitions > > > >>>> "created during group runtime," which directly points to a > > > >>> group-lifecycle > > > >>>> classifier. > > > >>>> I'd like to walk through why we landed on partition age, and the > > > >>>> trade-offs we considered. > > > >>>> > > > >>>> We evaluated three candidate signals: > > > >>>> > > > >>>> 1. `by_duration:5secs` > > > >>>> > > > >>>> This covers the metadata blindness window, but has issues the KIP > > > >>>> currently documents > > > >>>> under "Why not use `by_duration`?": > > > >>>> > > > >>>> - Client-side `now() - duration` introduces clock skew across > > consumers. > > > >>>> - `ListOffsets` retries shift the target forward, potentially > > missing > > > >>>> records produced between > > > >>>> attempts. > > > >>>> - It applies uniformly to all partitions without committed offsets, > > > >>>> including pre-existing partitions > > > >>>> newly assigned to the group, causing unnecessary replay. > > > >>>> > > > >>>> 2. Group creation time as classifier > > > >>>> > > > >>>> This works cleanly when the consumer is actively running. Our > > concern > > > >>>> is the idle / late-rejoin case: > > > >>>> > > > >>>> T=0: Group created. > > > >>>> T=1..T=100: Consumer idle (down, disconnected, etc.). > > > >>>> T=50: Partition added during the idle window. > > > >>>> T=100: Consumer resumes. > > > >>>> > > > >>>> Under group creation time, the new partition is classified as new > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50. > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated data > > that > > > >>>> the consumer accepts as lost — that is precisely the contract of > > > >>>> `auto.offset.reset=latest`. There is no principled reason to treat > > > >>>> the new partition differently; both contain backlog accumulated > > during > > > >>>> the same idle window. > > > >>>> > > > >>>> This aligns with the "backlog is backlog” principle you raised in > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on > > > >>>> every other partition during the same idle period; forcing 0-backlog > > > >>>> tolerance only on new partitions would be inconsistent with that > > > >>>> tolerance. > > > >>>> > > > >>>> 3. Partition age vs threshold > > > >>>> > > > >>>> Partition age corresponds to the actual silent data loss window, > > > >>>> the gap between partition creation and the consumer’s metadata > > > >>>> refresh. Within this window, data loss is genuinely silent: the > > > >>>> consumer had no opportunity to know about the partition. Outside > > this > > > >>>> window, missing data reflects either: > > > >>>> > > > >>>> - (a) the user’s tolerated cost of running with idle consumers, or > > > >>>> - (b) an operational issue to surface via monitoring, not via reset > > > >>> policy. > > > >>>> > > > >>>> We did not choose partition age because it is more elegant than > > group > > > >>>> creation time — we chose it because its failure mode (requires a > > > >>>> threshold) is > > > >>>> less invasive than the failure mode of group creation time > > (overrides > > > >>>> user-stated > > > >>>> `latest` intent during idle periods). > > > >>>> > > > >>>> Best Regards, > > > >>>> Jiunn-Yang > > > >>>> > > > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年6月13日 上午11:52 寫道： > > > >>>>> > > > >>>>> Hi Jun, > > > >>>>> > > > >>>>> Relying on both creation times will create an inconsistent > > scenario. A > > > >>>>> consumer that lost all offsets due to a long sleep will seek to the > > > >>>>> beginning for the partitions created later than the group. > > > >>>>> > > > >>>>> That is why we initially proposed KIP-1282 to fix the inconsistency > > > >>>> using a > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, > > KIP-1327 > > > >>>> goes > > > >>>>> back to using flexible configurations to prevent users from falling > > > >>> into > > > >>>>> that pitfall. > > > >>>>> > > > >>>>> Best, Chia-Ping > > > >>>>> > > > >>>>> Jun Rao via dev <[email protected]> 於 2026年6月13日週六 上午6:49寫道： > > > >>>>> > > > >>>>>> Hi, Jiunn-Yang, > > > >>>>>> > > > >>>>>> Thanks for the reply and sorry for the late reply. > > > >>>>>> > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels > > weird to > > > >>>> me. > > > >>>>>> It > > > >>>>>> categorizes partitions as new or existing based on the partition > > > >>>> creation > > > >>>>>> time. Intuitively, the categorization should be based on the group > > > >>>> creation > > > >>>>>> time: all partitions existing when the group is created are > > existing > > > >>> and > > > >>>>>> all partitions created after the group creation are new > > partitions. > > > >>>>>> > > > >>>>>> Jun > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <[email protected]> wrote: > > > >>>>>> > > > >>>>>>> Hi all, > > > >>>>>>> > > > >>>>>>> Manually bumping this thread. If there is no further > > > >>>>>>> discussion, I will close the vote. > > > >>>>>>> > > > >>>>>>> Best Regards, > > > >>>>>>> Jiunn-Yang > > > >>>>>>> > > > >>>>>>>> 黃竣陽 <[email protected]> 於 2026年6月1日 晚上7:16 寫道： > > > >>>>>>>> > > > >>>>>>>> Hello Jian, > > > >>>>>>>> > > > >>>>>>>> Thanks for your feedback, > > > >>>>>>>> > > > >>>>>>>> Agreed, partition expansion is a common operational task, not an > > > >>> edge > > > >>>>>>>> case. I've updated the Motivation section accordingly. > > > >>>>>>>> > > > >>>>>>>> Best Regards, > > > >>>>>>>> Jiunn-Yang > > > >>>>>>>> > > > >>>>>>>>> jian fu <[email protected]> 於 2026年6月1日 下午5:49 寫道： > > > >>>>>>>>> > > > >>>>>>>>> Hi Jiunn-Yang: > > > >>>>>>>>> > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that > > this > > > >>>>>> is a > > > >>>>>>>>> common scenario rather than an edge case, which further > > > >>> demonstrates > > > >>>>>> the > > > >>>>>>>>> need for this optimization. For example: > > > >>>>>>>>> A partition expansion is a common operational task in Kafka: To > > > >>>>>> balance > > > >>>>>>>>> resource utilization and cost, topics are typically created > > with a > > > >>>>>>> moderate > > > >>>>>>>>> default partition count. However, as traffic grows over time, > > it is > > > >>>>>>> often > > > >>>>>>>>> necessary to increase the number of partitions to accommodate > > the > > > >>>>>> higher > > > >>>>>>>>> workload. > > > >>>>>>>>> > > > >>>>>>>>> Regards > > > >>>>>>>>> Jian > > > >>>>>>>>> > > > >>>>>>>>> 黃竣陽 <[email protected]> 于2026年5月30日周六 22:31写道： > > > >>>>>>>>> > > > >>>>>>>>>> Hello chia, > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP! > > > >>>>>>>>>> > > > >>>>>>>>>> Best Regards, > > > >>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>> > > > >>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月30日 晚上8:29 寫道： > > > >>>>>>>>>>> > > > >>>>>>>>>>> Hi Jiunn-Yang, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when > > > >>> describing > > > >>>>>>>>>>> partitions in the KIP? I understand you are using them to > > > >>> describe > > > >>>>>> the > > > >>>>>>>>>>> "freshness" or the users' need for the records, but applying > > > >>> these > > > >>>>>>> terms > > > >>>>>>>>>> to > > > >>>>>>>>>>> the partition itself feels a bit unnatural. > > > >>>>>>>>>>> > > > >>>>>>>>>>> After all, in this scenario, users don't really care whether > > a > > > >>>>>>> partition > > > >>>>>>>>>> is > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that they > > won't > > > >>>>>>> silently > > > >>>>>>>>>>> lose any live records produced to the topic during their > > active > > > >>>>>>>>>> consumption. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best, Chia-Ping > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月30日週六 下午12:30寫道： > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Hello Jun, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation > > > >>>> section. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Jun Rao via dev <[email protected]> 於 2026年5月30日 > > 凌晨1:12 寫道： > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger > > motivation for > > > >>>>>> the > > > >>>>>>>>>> KIP. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all partitions > > > >>> without > > > >>>>>> a > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded partition > > (hot) > > > >>>> is > > > >>>>>>>>>>>>> fundamentally different from a partition the consumer has > > never > > > >>>>>> seen > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot > > partition > > > >>>>>>>>>>>>> fundamentally different from the cold? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also > > > >>>> insufficient > > > >>>>>>>>>>>> because: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across > > > >>> nodes > > > >>>>>>> due > > > >>>>>>>>>>>> to > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large > > > >>> duration, > > > >>>>>>>>>>>> causing > > > >>>>>>>>>>>>> unnecessary reprocessing. > > > >>>>>>>>>>>>> - On network errors, the client recalculates the seek time > > on > > > >>>>>> retry, > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking data > > loss." > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> However, both of these situations are rare. If these issues > > > >>>>>> persist, > > > >>>>>>>>>> more > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations > > don't > > > >>>>>> need a > > > >>>>>>>>>>>> common > > > >>>>>>>>>>>>> solution. If users care about those rare situations, they > > can > > > >>>>>>> implement > > > >>>>>>>>>>>>> customized logic using > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned(). > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <[email protected]> > > > >>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hello chia, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Thanks for the feedback, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value should > > always > > > >>>> be > > > >>>>>>>>>>>> greater > > > >>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any > > > >>>>>>> field > > > >>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> The earliest point at which we can detect the version > > mismatch > > > >>>> is > > > >>>>>>>>>> during > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside > > > >>>>>> poll(). > > > >>>>>>>>>>>>>> Therefore, the > > > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from > > > >>> poll(). > > > >>>>>>> I’ll > > > >>>>>>>>>>>>>> clarify this in the KIP. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月17日 > > 下午4:50 寫道： > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> hi Jiunn > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this > > > >>> partition > > > >>>>>> in > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as > > > >>>>>>>>>> broker_current_time > > > >>>>>>>>>>>> - > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not > > > >>>>>> support > > > >>>>>>>>>> this > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value should > > always > > > >>>> be > > > >>>>>>>>>>>> greater > > > >>>>>>>>>>>>>> than or equal to zero, right? > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> New Old (MetadataResponse v0–13) positive any > > > >>>>>>> field > > > >>>>>>>>>>>>>> absent UnsupportedVersionException > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when > > calling > > > >>>>>>>>>> `poll()`? > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>> Hello Jun, chia, > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the > > > >>>>>>> discussion > > > >>>>>>>>>>>>>>>> feedback. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset > > > >>> behavior > > > >>>>>>> from > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all > > > >>>>>>> auto.offset.reset > > > >>>>>>>>>>>>>> values > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none). > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions > > > >>>>>>>>>>>>>>>> config setting > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base > > > >>> auto.offset.reset > > > >>>>>>>>>> policy > > > >>>>>>>>>>>>>> continues > > > >>>>>>>>>>>>>>>> to apply unchanged. > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a > > > >>>> separate > > > >>>>>>>>>>>>>> internal config > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to > > > >>>>>> earliest). > > > >>>>>>>>>> This > > > >>>>>>>>>>>>>> decoupled design makes > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public > > > >>>>>>> user-facing > > > >>>>>>>>>>>>>> configuration in a future KIP. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年5月16日 > > 清晨7:46 > > > >>> 寫道： > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed > > > >>>> below: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default > > > >>> value > > > >>>>>> of > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and > > > >>>>>> latest, > > > >>>>>>> and > > > >>>>>>>>>>>> it > > > >>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest. > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an > > internal > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest > > > >>>> already > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of > > other > > > >>>>>>> values > > > >>>>>>>>>>>> in a > > > >>>>>>>>>>>>>> separate KIP. > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to > > all > > > >>> for > > > >>>>>>>>>>>>>> consistency. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> WDYT? > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Thanks for the reply. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a > > Kafka > > > >>>>>> topic > > > >>>>>>> is > > > >>>>>>>>>>>>>> expanded > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto > > > >>> offset > > > >>>>>>> reset > > > >>>>>>>>>>>>>> policy > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those > > > >>> partitions > > > >>>>>>> before > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record > > loss > > > >>>>>> issue > > > >>>>>>>>>> could > > > >>>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>>>>> happen, right? > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions > > will > > > >>>> take > > > >>>>>>> the > > > >>>>>>>>>>>> same > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it > > > >>>>>>> by_duration if > > > >>>>>>>>>>>>>> needed. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai < > > > >>>>>>>>>> [email protected] > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> hi Jun > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something > > > >>>>>>> important > > > >>>>>>>>>>>> from > > > >>>>>>>>>>>>>> your > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to > > clarify > > > >>> with > > > >>>>>> a > > > >>>>>>> few > > > >>>>>>>>>>>>>> questions: > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this > > logic to > > > >>>>>>> other > > > >>>>>>>>>>>> reset > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or > > > >>>>>> by_duration > > > >>>>>>>>>>>> don't > > > >>>>>>>>>>>>>> seem > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a > > > >>>>>>> partition > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>> expanded. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to > > > >>>>>> earliest > > > >>>>>>> or > > > >>>>>>>>>>>>>> latest, > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For > > example, > > > >>>>>> if a > > > >>>>>>>>>>>>>> consumer is > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created > > during > > > >>>>>> that > > > >>>>>>>>>>>>>> downtime, > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when > > > >>>>>> resuming, > > > >>>>>>>>>>>> rather > > > >>>>>>>>>>>>>> than > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is > > > >>>>>>> technically > > > >>>>>>>>>>>>>> "new" to > > > >>>>>>>>>>>>>>>>>>> the group. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a > > max.age > > > >>>>>>> threshold. > > > >>>>>>>>>>>> It > > > >>>>>>>>>>>>>> gives > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely > > > >>>> "hot/new" > > > >>>>>>> and > > > >>>>>>>>>>>>>> what is > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote: > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang, > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit > > weird. It > > > >>>>>> only > > > >>>>>>>>>>>>>> applies when > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that > > the > > > >>>>>>>>>> motivation > > > >>>>>>>>>>>>>>>>>>> equally > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other > > values > > > >>> like > > > >>>>>>>>>>>>>> by_duration. > > > >>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to > > > >>>> control > > > >>>>>>>>>> newly > > > >>>>>>>>>>>>>> created > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group > > starts. > > > >>>>>>> Have we > > > >>>>>>>>>>>>>>>>>>> considered > > > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new > > > >>>> .partitions? > > > >>>>>>> If > > > >>>>>>>>>>>> this > > > >>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults > > to > > > >>> the > > > >>>>>>>>>> policy > > > >>>>>>>>>>>>>> used > > > >>>>>>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it > > explicitly to > > > >>>>>>>>>> customize > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions. > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 < > > [email protected]> > > > >>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread. > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <[email protected]> 於 2026年5月1日 晚上10:37 寫道： > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Hello all, > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02: > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The > > > >>>>>>> PartitionMetadata > > > >>>>>>>>>>>>>> struct > > > >>>>>>>>>>>>>>>>>>>>> gains a new > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed > > > >>>>>>> server-side > > > >>>>>>>>>>>> by > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>> broker as > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when > > > >>>>>>> MembershipManager > > > >>>>>>>>>>>>>> detects > > > >>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>> newly assigned > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata > > for > > > >>>> the > > > >>>>>>>>>>>> affected > > > >>>>>>>>>>>>>>>>>>> topic > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if > > the > > > >>>>>> topic > > > >>>>>>> ID > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>> already > > > >>>>>>>>>>>>>>>>>>>>> in the cache. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> MB0: > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported > > > >>>>>>>>>>>>>> MetadataResponse > > > >>>>>>>>>>>>>>>>>>>>> version via the > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the > > > >>>>>>> negotiated > > > >>>>>>>>>>>>>>>>>>> version is > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs > > at > > > >>> all > > > >>>>>> and > > > >>>>>>>>>> can > > > >>>>>>>>>>>>>>>>>>> throw an > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to > > > >>> latest > > > >>>>>>> and > > > >>>>>>>>>>>>>> risking > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3: > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP. > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月29日 > > > >>>> 下午4:04 > > > >>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> hi David > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' > > > >>>>>> resolution > > > >>>>>>>>>> from > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the > > control > > > >>>>>> plane > > > >>>>>>>>>>>> clean. > > > >>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>> main > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing > > > >>>> inter-broker > > > >>>>>>>>>> clock > > > >>>>>>>>>>>>>> skew. > > > >>>>>>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single > > source of > > > >>>>>> truth > > > >>>>>>>>>> for > > > >>>>>>>>>>>>>> time. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be > > > >>>>>>> negligible. > > > >>>>>>>>>>>>>> Given > > > >>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in > > > >>>> minutes > > > >>>>>>> or > > > >>>>>>>>>>>>>> hours, a > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers > > > >>> won't > > > >>>>>>> impact > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>> fallback decision. > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <[email protected]> 於 > > > >>>>>> 2026年4月29日 > > > >>>>>>>>>>>> 下午3:29 > > > >>>>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous > > > >>>>>>> conversation > > > >>>>>>>>>>>> but I > > > >>>>>>>>>>>>>>>>>>> took a > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one. > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with > > the > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new > > boolean; > > > >>> the > > > >>>>>>> KIP > > > >>>>>>>>>>>>>> states > > > >>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this > > boolean > > > >>> is > > > >>>>>>> set. > > > >>>>>>>>>>>>>>>>>>> Implicitly, > > > >>>>>>>>>>>>>>>>>>>>> this > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new > > > >>> partition, > > > >>>>>> it > > > >>>>>>>>>> will > > > >>>>>>>>>>>>>>>>>>> issue a > > > >>>>>>>>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the > > ages. > > > >>>> Is > > > >>>>>>> my > > > >>>>>>>>>>>>>>>>>>>>> understanding > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and > > also > > > >>>>>>> explain > > > >>>>>>>>>>>>>> how it > > > >>>>>>>>>>>>>>>>>>>>> fits > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch > > > >>>>>> offsets, > > > >>>>>>>>>>>> etc.). > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder > > if > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right > > place > > > >>> for > > > >>>>>>> this > > > >>>>>>>>>>>> given > > > >>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>> a new > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it > > could > > > >>>>>> simply > > > >>>>>>>>>>>>>> include > > > >>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather > > cautious > > > >>>> about > > > >>>>>>> not > > > >>>>>>>>>>>>>>>>>>> overloading > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated > > > >>>> concepts. > > > >>>>>>> The > > > >>>>>>>>>>>> API > > > >>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking > > > >>>> partitions. > > > >>>>>>> The > > > >>>>>>>>>>>> fact > > > >>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>> we > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding > > Streams > > > >>> API > > > >>>>>>> also > > > >>>>>>>>>>>>>> suggests > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do > > if we > > > >>>>>>> want to > > > >>>>>>>>>>>>>>>>>>> support > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future? > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>> David > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar > > Basani > > > >>>> via > > > >>>>>>> dev > > > >>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn, > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know > > about > > > >>> the > > > >>>>>>> gap. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for > > > >>>>>>> RequestPartitionAges > > > >>>>>>>>>>>>>> field. > > > >>>>>>>>>>>>>>>>>>> Can a > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, > > PartitionAges on > > > >>>>>>>>>>>>>>>>>>> TopicPartitions) > > > >>>>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump? > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a > > recommended > > > >>>>>> value > > > >>>>>>> or > > > >>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>> ConfigDef > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the > > > >>>>>>>>>>>> metadata.max.age.ms > > > >>>>>>>>>>>>>> ? > > > >>>>>>>>>>>>>>>>>>>>> Sizing > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka > > > >>>>>> Streams, > > > >>>>>>>>>>>> would > > > >>>>>>>>>>>>>> it > > > >>>>>>>>>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>>>>> better > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config > > > >>>>>> auto.offset.reset.latest.max.age > > > >>>>>>> to > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) > > > >>>>>>>>>>>>>>>>>>>>> for a > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This > > is > > > >>> the > > > >>>>>>> most > > > >>>>>>>>>>>>>>>>>>> familiar > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily > > mistakenly > > > >>>>>>> configure > > > >>>>>>>>>>>>>> it. Or > > > >>>>>>>>>>>>>>>>>>>>> may be > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls > > > >>> back > > > >>>>>> to > > > >>>>>>>>>>>>>> earliest" > > > >>>>>>>>>>>>>>>>>>>>> reads as > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition > > > >>> which > > > >>>>>>> isn't > > > >>>>>>>>>>>>>>>>>>> supported. > > > >>>>>>>>>>>>>>>>>>>>> May > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer > > resolves > > > >>>> the > > > >>>>>>>>>>>> initial > > > >>>>>>>>>>>>>>>>>>>>> position to > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if > > earliest was > > > >>>>>>> applied > > > >>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is > > > >>>>>>> unchanged. > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 < > > > >>>>>>> [email protected]> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change. > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <[email protected]> 於 > > > >>> 2026年4月28日 > > > >>>>>>> 晚上8:03 > > > >>>>>>>>>>>> 寫道： > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition > > creation > > > >>>> time > > > >>>>>>> via > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> Admin > > > >>>>>>>>>>>>>>>>>>>>> API? > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to > > > >>> diagnose > > > >>>>>> and > > > >>>>>>>>>>>>>>>>>>> troubleshoot > > > >>>>>>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote: > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on > > KIP-1327 > > > >>>>>>> Prevent > > > >>>>>>>>>> Hot > > > >>>>>>>>>>>>>> Data > > > >>>>>>>>>>>>>>>>>>>>> Loss > > > >>>>>>>>>>>>>>>>>>>>>>>>> on > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$ > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age, > > > >>>>>>>>>>>>>>>>>>>>> a > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly > > expanded > > > >>>>>> (hot) > > > >>>>>>>>>>>>>> partitions > > > >>>>>>>>>>>>>>>>>>>>> from > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold > > > >>>> automatically > > > >>>>>>> fall > > > >>>>>>>>>>>>>> back > > > >>>>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a > > full > > > >>>>>>> historical > > > >>>>>>>>>>>>>>>>>>> reprocess. > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > >
