Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Gaurav, You have a valid concern regarding the proposed default value for log.message.timestamp.after.max.ms being only 1 hour. We can keep this as Long.MAX_VALUE for the first phase implementation of this KIP and change the default value to 1 hour in future iterations. In the meantime, us

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Mehari, Thanks for the KIP! The default value of *log.message.timestamp.after.max.ms * is proposed to be 1 hour. Given that this is a client application breaking change, should we consider an opt-in grace period to give a proper heads up to the users?

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Mehari, I was just getting annoyed looking at logs of kafka clusters with timestamp issues. Let me take final look at the KIP. Thanks, Justine On Wed, Jun 21, 2023 at 11:13 AM Beyene, Mehari wrote: > Hi Justine, > > I have initiated the voting process for this KIP here: > https://lists.apac

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Justine, I have initiated the voting process for this KIP here: https://lists.apache.org/thread/y3yfnphsmrgwfdhx3xfhjtwdb7p1dn0v We have already received two binding votes, and I am seeking a third vote for the adoption of the KIP. As you have previously reviewed this KIP, would you be will

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Mehari, Thanks for the update. The KIP LGTM now. Luke On Tue, Jun 13, 2023 at 12:20 AM Beyene, Mehari wrote: > Hi Luke, Divij and All, > > I have updated the KIP to incorporate Luke's comment. We now have two new > configurations for validating the message timestamp difference. The > exist

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Luke, Divij and All, I have updated the KIP to incorporate Luke's comment. We now have two new configurations for validating the message timestamp difference. The existing configuration, "log.message.timestamp.difference.max.ms", will be deprecated. The "before" validation uses the same def

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi all, Please ignore this message. I'm just using this message to bump this thread so that it will show up in Gaurav's inbox. He wanted to send out review comments for this KIP. Cheers, Kowshik On Wed, Jun 7, 2023 at 1:39 PM Beyene, Mehari wrote: > > Although it's more verbose, splitting th

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

> Although it's more verbose, splitting the configuration into explicit ‘past’ > and ‘future’ would provide the appropriate tradeoff between constraint and > flexibility, right? +1

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Luke, thank you for the suggestion of introducing the two configurations. I have discussed this option internally with Divij, and your suggestion does have its merits. I will update the KIP with your suggestions and will revert back in a day or two. Kirk, thank you for participating in the KIP

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Mehari, Thanks for the KIP and keeping it up-to-date with the discussions here! Question: 1. Is it possible to check for invalid timestamps in the client? Suppose we were to develop a means to determine with high confidence that the user had provided a timestamp in nanoseconds vs. milliseco

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Divij/all, > On Jun 6, 2023, at 5:42 AM, Divij Vaidya wrote: > > Hi Luke > > Thank you for your participation in reviewing this KIP. > > #1 Updated the KIP with correct configuration names and hyperlinks. > > #2 Yes, the semantics change from a perspective that the difference is > always i

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Luke Thank you for your participation in reviewing this KIP. #1 Updated the KIP with correct configuration names and hyperlinks. #2 Yes, the semantics change from a perspective that the difference is always in the past (or at most 1 hour into the future). Updated the compatibility section to

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Beyene, Thanks for the KIP. Questions: 1. We don't have "*max.message.time.difference.ms *" config, I think you're referring to "log.message.timestamp.difference.max.ms"? 2. After this KIP, the semantics of "log.message.timestamp.difference.max.ms" will b

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Justine and Divij, Thank you for the recommendations. I've made the updates to the KIP and added a new section called "Future Work: Update Message Format to Include Both Client Timestamp and LogAppend Timestamp." Please take a look when get some time and let me know if there's anything else

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Divij, Yeah, this makes sense. Let's just include this in the KIP as well. Thanks! On Mon, Jun 5, 2023 at 10:16 AM Divij Vaidya wrote: > Hey Justine > > Thank you for bringing this up. We had a discussion earlier in this [1] > thread and concluded that bumping up the message version is a ve

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Justine Thank you for bringing this up. We had a discussion earlier in this [1] thread and concluded that bumping up the message version is a very expensive operation. Hence, we want to bundle together a bunch of impactful changes that we will perform on the message version and change it in v4

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Mehari, Thanks for adding that section. I think one other thing folks have considered is including two timestamps in the message format -- one for the client side timestamp and one for the server side. Of course, this would require a bump to the message format, and that hasn't happened in a whi

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Justine, I added a section under Proposed Changes -> Timestamp Validation Logic to capture how the INVALID_TIMESTAMP is handled in this KIP. Please let me know if there are any additional areas you would like me to address. Thanks, Mehari

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Thank you for the feedback Justine. Yes, I can expand on the KIP what the behavior is when we return INVALID_TIMESTAMP. On 5/30/23, 1:34 PM, "Justine Olshan" mailto:jols...@confluent.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or op

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Mehari, This is an interesting KIP! I've seen my fair share of issues due to future timestamps, so this is definitely an area that could be improved. I noticed in the compatibility section it says: > There are no changes to public interfaces that will impact clients. However, this change is co

[DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Everyone, I would like to start a discussion on KIP-937: Improve Message Timestamp Validation (https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation). This is a small KIP that aims to tighten the current validation logic of a message timestamp. Th