hi Jun Negative value are allowed for `retention.ms`, so setting a lower bound break the behavior of disabling retention.
If the lower bound is intended to avoid frequent segment rolling, we could consider disabling the action: `retention.ms` would be ignored when there is only one segment. Best, Chia-Ping On 2025/08/06 16:28:00 Jun Rao wrote: > Hi, Divij, > > Another comment regarding the changes to segment.ms. > > Currently, retention.ms has the following doc. > "Additionally, retention.ms configuration operates independently of " > segment.ms" and "segment.bytes" configurations. Moreover, it triggers the > rolling of new segment if the retention.ms condition is satisfied." > > So, if we set a lower bound for segment.ms, should we do the same for > retention.ms too? > > Thanks, > > Jun > > On Thu, Dec 5, 2024 at 1:26 AM Divij Vaidya <divijvaidy...@gmail.com> wrote: > > > Hi Ismael > > > > You are right. In hindsight, I should have started deprecation with one of > > the earlier 3.x versions. > > > > But since that ship has already sailed, we can either wait for 5.0 to make > > the actual change with just a deprecation notice in 4.x OR we can make the > > change starting in 4.0 itself. > > > > I think that the latter (changing in 4.0) is desirable because the changes > > are low risk. I will elaborate why I think so. > > > > The practice of having a deprecation period achieves two purposes: > > 1\ provide users with ample time to prepare for upgrades where code changes > > may be required. > > 2\ allow users to measure the impact of the changes prior to moving to 4.0. > > Testing on existing versions is important since in 4.0, it will be > > difficult to isolate the impact of these changes due to the presence of > > many other changes. > > > > Purpose 2 can still be achieved with the current proposal since all the > > proposed changes could be tested with existing Kafka versions. There are no > > new configurations which have been added in the proposal which cannot be > > tested with older versions. > > > > For point 1, with the current proposal, 99% of users will require no code > > change since we do not anticipate existing workloads to be using the now > > removed lower bounds for the configurations. Here's a breakdown of risk at > > a per-config level. > > > > 1. The changes in configuration to prevent small segments are expected to > > be compatible with 99% of production workloads. This is because Kafka will > > not be functional with extremely low segment size due to OS imposed limits > > on file descriptors and mmaps. Hence, the risk of breaking existing > > workloads by adding these new constraints is very low. As noted in the > > discussion, we may impact some testing scenarios but there are quick ways > > to work around them. We will also provide examples on how to write tests > > where you expect a quick segment roll. > > 2. The config changes which are associated with Tiered Storage (thread pool > > configs) were introduced in 3.9. The customers who use the now removed, > > "-1" value will have to make a change before upgrading to 4.0. I agree that > > this does cause user friction during upgrade. To reduce the chance of > > un-intentional error, we will add a "Pre-Upgrade Validation Tool". > > 3. The change associated with the num recovery thread is an internal change > > to the broker. I do not anticipate any negative side effects of changing > > this default. It also does not require any code change from the customer > > since no new constraint was added. > > 4. `linger.ms` change is potentially the one with the largest impact to > > existing production workloads. I would assume that users who > > intentionally set linger.ms = 0 are expert users (since this quite an > > unusual setting to get to behave correctly) and these users would pay > > attention to upgrade notes (similar to how we made ack=all as default in > > producers starting in 3.0). For users who un-intentionally set it to 0, I > > believe that the new setting will work better for their workloads. > > > > I will update the KIP with the above explanation. > > > > Looking forward to hearing your thoughts. > > > > -- > > Divij Vaidya > > > > > > > > On Thu, Dec 5, 2024 at 4:01 AM Ismael Juma <m...@ismaeljuma.com> wrote: > > > > > Hi Divij, > > > > > > The KIP didn't state this, but the usual practice is to have a > > deprecation > > > period before we make incompatible changes. Why did we reject this > > option? > > > We should mention that explicitly in the KIP. > > > > > > Ismael > > > > > > On Tue, Nov 19, 2024, 2:55 AM Divij Vaidya <divijvaidy...@gmail.com> > > > wrote: > > > > > > > KT1 - That is right. We will throw a ConfigException. That is why this > > > > change is considered backward incompatible. To be honest, given the > > > nature > > > > of suggested changes, I don't see any valid use case on why a user may > > > have > > > > a value which will be invalid after the new constraints. > > > > > > > > > > > > -- > > > > Divij Vaidya > > > > > > > > > > > > > > > > On Tue, Nov 19, 2024 at 2:21 AM Kirk True <k...@kirktrue.pro> wrote: > > > > > > > > > Hi Divij, > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > My only question: > > > > > > > > > > KT1. In the case where we change the constraints so that a user's > > > > > previously valid configuration is now invalid, do we do anything > > other > > > > than > > > > > throw a ConfigException? > > > > > > > > > > Thanks, > > > > > Kirk > > > > > > > > > > On Mon, Nov 18, 2024, at 2:13 AM, Divij Vaidya wrote: > > > > > > Hey folks > > > > > > > > > > > > With 4.0, we have an opportunity to reset the default values and > > add > > > > > > constraints in the configurations based on our learnings since 3.0. > > > > > > > > > > > > Here's a KIP which modifies defaults for some properties and > > modifies > > > > the > > > > > > constraints for a few others. > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations > > > > > > > > > > > > > > > > > > Looking forward for your feedback. > > > > > > > > > > > > (Previous discussion thread on this topic - > > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp ) > > > > > > > > > > > > > > > > > > > > >
