Hi Ismael You are right. In hindsight, I should have started deprecation with one of the earlier 3.x versions.
But since that ship has already sailed, we can either wait for 5.0 to make the actual change with just a deprecation notice in 4.x OR we can make the change starting in 4.0 itself. I think that the latter (changing in 4.0) is desirable because the changes are low risk. I will elaborate why I think so. The practice of having a deprecation period achieves two purposes: 1\ provide users with ample time to prepare for upgrades where code changes may be required. 2\ allow users to measure the impact of the changes prior to moving to 4.0. Testing on existing versions is important since in 4.0, it will be difficult to isolate the impact of these changes due to the presence of many other changes. Purpose 2 can still be achieved with the current proposal since all the proposed changes could be tested with existing Kafka versions. There are no new configurations which have been added in the proposal which cannot be tested with older versions. For point 1, with the current proposal, 99% of users will require no code change since we do not anticipate existing workloads to be using the now removed lower bounds for the configurations. Here's a breakdown of risk at a per-config level. 1. The changes in configuration to prevent small segments are expected to be compatible with 99% of production workloads. This is because Kafka will not be functional with extremely low segment size due to OS imposed limits on file descriptors and mmaps. Hence, the risk of breaking existing workloads by adding these new constraints is very low. As noted in the discussion, we may impact some testing scenarios but there are quick ways to work around them. We will also provide examples on how to write tests where you expect a quick segment roll. 2. The config changes which are associated with Tiered Storage (thread pool configs) were introduced in 3.9. The customers who use the now removed, "-1" value will have to make a change before upgrading to 4.0. I agree that this does cause user friction during upgrade. To reduce the chance of un-intentional error, we will add a "Pre-Upgrade Validation Tool". 3. The change associated with the num recovery thread is an internal change to the broker. I do not anticipate any negative side effects of changing this default. It also does not require any code change from the customer since no new constraint was added. 4. `linger.ms` change is potentially the one with the largest impact to existing production workloads. I would assume that users who intentionally set linger.ms = 0 are expert users (since this quite an unusual setting to get to behave correctly) and these users would pay attention to upgrade notes (similar to how we made ack=all as default in producers starting in 3.0). For users who un-intentionally set it to 0, I believe that the new setting will work better for their workloads. I will update the KIP with the above explanation. Looking forward to hearing your thoughts. -- Divij Vaidya On Thu, Dec 5, 2024 at 4:01 AM Ismael Juma <m...@ismaeljuma.com> wrote: > Hi Divij, > > The KIP didn't state this, but the usual practice is to have a deprecation > period before we make incompatible changes. Why did we reject this option? > We should mention that explicitly in the KIP. > > Ismael > > On Tue, Nov 19, 2024, 2:55 AM Divij Vaidya <divijvaidy...@gmail.com> > wrote: > > > KT1 - That is right. We will throw a ConfigException. That is why this > > change is considered backward incompatible. To be honest, given the > nature > > of suggested changes, I don't see any valid use case on why a user may > have > > a value which will be invalid after the new constraints. > > > > > > -- > > Divij Vaidya > > > > > > > > On Tue, Nov 19, 2024 at 2:21 AM Kirk True <k...@kirktrue.pro> wrote: > > > > > Hi Divij, > > > > > > Thanks for the KIP! > > > > > > My only question: > > > > > > KT1. In the case where we change the constraints so that a user's > > > previously valid configuration is now invalid, do we do anything other > > than > > > throw a ConfigException? > > > > > > Thanks, > > > Kirk > > > > > > On Mon, Nov 18, 2024, at 2:13 AM, Divij Vaidya wrote: > > > > Hey folks > > > > > > > > With 4.0, we have an opportunity to reset the default values and add > > > > constraints in the configurations based on our learnings since 3.0. > > > > > > > > Here's a KIP which modifies defaults for some properties and modifies > > the > > > > constraints for a few others. > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations > > > > > > > > > > > > Looking forward for your feedback. > > > > > > > > (Previous discussion thread on this topic - > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp ) > > > > > > > > > >
