Re: [VOTE] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Divij Vaidya
interested folks may not have had a chance to look into it yet. -- Divij Vaidya On Tue, Jul 25, 2023 at 6:45 PM Satish Duggana wrote: > > +1 for the KIP. > > Thanks, > Satish. > > On Tue, 25 Jul 2023 at 18:31, Kamal Chandraprakash > wrote: > > > > +1 (non-bind

Re: Debugging Jenkins test failures

2023-07-26 Thread Divij Vaidya
rlin [2] https://ge.apache.org/scans/failures?search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe/Berlin -- Divij Vaidya On Tue, Jul 25, 2023 at 8:09 PM Kirk True wrote: > > Hi all! > > I’ve noticed that we’re back in the state where it’s tough to get

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread Divij Vaidya
+1 (binding) -- Divij Vaidya On Wed, Jul 26, 2023 at 2:56 PM ziming deng wrote: > > +1 (binding) from me. > > Thanks for the KIP! > > -- > Ziming > > > On Jul 26, 2023, at 20:18, Luke Chen wrote: > > > > +1 (binding) from me. > > > > T

Re: Apache Kafka 3.6.0 release

2023-07-27 Thread Divij Vaidya
nge in future but the implementation is not recommended to be used in production. Thoughts? -- Divij Vaidya On Wed, Jul 26, 2023 at 6:31 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: > > Yes, still need one more binding vote to pass. I'll send a reminder if the > vote is still p

Re: Apache Kafka 3.6.0 release

2023-07-27 Thread Divij Vaidya
hich case I apologize for bringing this naive point). -- Divij Vaidya On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma wrote: > > Hi Divij, > > Some of these are launch checklist items (not really goals) and some are > compatibility guarantees. More below. > > On Thu, Jul 27, 2023,

Re: Apache Kafka 3.6.0 release

2023-07-27 Thread Divij Vaidya
aning of early > access, we have two options: > > 1. Have a KIP specifically for tiered storage. > 2. Have a KIP to define general guidelines for what early access means. > > Does this make sense? > > Ismael > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya > wrote:

Re: Debugging Jenkins test failures

2023-08-02 Thread Divij Vaidya
No. We started using gradle analysis starting July 12th. Prior to that, the only data that we have is coming from Apache CI which AFAIK doesn't have a per-test history view - https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/ -- Divij Vaidya On Wed, Aug 2, 2023 at 1:04 AM Kirk

Re: [DISCUSS] KIP-942: Add Power(ppc64le) support

2023-08-03 Thread Divij Vaidya
ns on every commit on every PR. From our earlier discussion on this thread, we discussed to only add it for "branch builder". Also, please add option of adding test to "PR builder" in the rejected alternative section. -- Divij Vaidya On Thu, Aug 3, 2023 at 8:40 AM Vaibhav Nazar

Re: [DISCUSS] Cluster-wide disablement of Tiered Storage

2023-08-04 Thread Divij Vaidya
houghts on that thread. -- Divij Vaidya On Fri, Aug 4, 2023 at 6:03 PM Andrew Schofield wrote: > > Hi Christo, > I agree with you. > > Option 4.1 without a KIP seems like an acceptable starting point for something > which will be relatively rare, provided that it’s easy for the

Re: What are the biggest issues with Apache Kafka?

2023-08-11 Thread Divij Vaidya
ss+Release+Notes [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage -- Divij Vaidya Divij Vaidya On Fri, Aug 11, 2023 at 4:55 AM ziming deng wrote: > Hi Liam, > > The Apache Kafka project has several modules, I think you should firstly > sele

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-08-14 Thread Divij Vaidya
+1 (binding) -- Divij Vaidya On Wed, Jul 26, 2023 at 9:04 AM Vaibhav Nazare wrote: > > I'd like to call a vote on KIP-942

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-08-28 Thread Divij Vaidya
ge we get is that, unlike nightly builds which have a chance of being ignored, branch builds are actively monitored by folks in the community. Hence, we will be able to add this new suite without adding a new routine in the maintenance. -- Divij Vaidya On Fri, Aug 25, 2023 at 6:49 PM Colin M

FYI - CI failures due to Apache Infra (Issue with creating launcher for agent)

2023-08-28 Thread Divij Vaidya
while, the workaround is to restart the tests. Cheers! -- Divij Vaidya

Re: FYI - CI failures due to Apache Infra (Issue with creating launcher for agent)

2023-08-31 Thread Divij Vaidya
This should be fixed now. Please comment on https://issues.apache.org/jira/browse/INFRA-24927 if you find a case where you still hit this problem. -- Divij Vaidya On Mon, Aug 28, 2023 at 12:05 PM Luke Chen wrote: > > Thanks for the info, Divij! > > Luke > > On Mon, Aug 28, 202

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-09-11 Thread Divij Vaidya
you please quantify (with an experiment) the expected perf impact of adding the partition dimension? This could be done as part of "test plan" section and would serve as a data point for users to understand the potential impact if they decide to turn it on. -- Divij Vaidya On Sat, Sep 9,

Re: Complete Kafka replication protocol description

2023-09-11 Thread Divij Vaidya
adding the TLA+ spec as part of the community owned project itself. Right now, things are a bit busy on my end, but I am looking forward to exploring what you shared above in the coming weeks (perhaps a first review by end of sept). Thank you again for starting this conversation. -- Divij Vaidya

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-09-13 Thread Divij Vaidya
this test suite 5 times won't be a burden to the CI > > infrastructure. The advantage we get is that, unlike nightly builds > > which have a chance of being ignored, branch builds are actively > > monitored by folks in the community. Hence, we will be able to add > &

Re: [VOTE] 3.6.0 RC0

2023-09-19 Thread Divij Vaidya
e as part of tiered storage 3. verified that public keys for RM are available at https://keys.openpgp.org/ 4. verified that public keys for RM are available at https://people.apache.org/keys/committer/ -- Divij Vaidya On Tue, Sep 19, 2023 at 12:41 PM Sagar wrote: > > Hey Satish, > >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-20 Thread Divij Vaidya
Hey Satish My comments about documentation misses from RC0 vote thread [1] are still not addressed (such as missing metric documentation, formatting problems etc). Could you please mention why we shouldn't consider them as blockers to make RC1 as the final release? [1] https://lists.apache.org/th

Re: Apache Kafka 3.6.0 release

2023-09-21 Thread Divij Vaidya
Hey Satish I filed a PR to fix the website formatting bug in 3.6 documentation - https://github.com/apache/kafka/pull/14419 Please take a look when you get a chance. -- Divij Vaidya On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton wrote: > > Hi Satish, > > I think this qualifies

Re: [ANNOUNCE] New committer: Yash Mayya

2023-09-21 Thread Divij Vaidya
Congratulations Yash! Divij Vaidya On Thu, Sep 21, 2023 at 6:18 PM Sagar wrote: > > Congrats Yash ! > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin wrote: > > > Awesome ! Congratulations Yash !! > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar > > w

Re: Apache Kafka 3.6.0 release

2023-09-22 Thread Divij Vaidya
ifferently. -- Divij Vaidya On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana wrote: > > Thanks Divij for raising a PR for doc formatting issue. > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, wrote: > > > Hey Satish > > > > I filed a PR to fix the website form

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

2023-09-23 Thread Divij Vaidya
Congratulations Justine! On Sat 23. Sep 2023 at 07:06, Chris Egerton wrote: > Congrats Justine! > On Fri, Sep 22, 2023, 20:47 Guozhang Wang > wrote: > > > Congratulations! > > > > On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai > > > wrote: > > > > > > Congratulations Justine! > > > > > >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-25 Thread Divij Vaidya
could post the results of system tests before I vote on this. I am particularly interested in looking at producer/consumer performance results since we have quite a few changes in this release. What is the plan on the system tests? -- Divij Vaidya On Mon, Sep 25, 2023 at 9:10 AM Luke Chen wrote

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-25 Thread Divij Vaidya
Correction: posted the wrong JIRA in my previous email. Instead of https://issues.apache.org/jira/browse/KAFKA-15001, please consider this https://issues.apache.org/jira/browse/KAFKA-15487 -- Divij Vaidya On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya wrote: > > Hi Satish > > 1.

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
ng it here as an FYI. -- Divij Vaidya On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya wrote: > > Found a bug while testing TS feature in 3.6 - > https://issues.apache.org/jira/browse/KAFKA-15481 > > I don't consider it as a blocker for release since it's a concurrency > b

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
Hey team I need help in determining whether https://github.com/apache/kafka/pull/14457 is a release blocker bug or not. If someone is familiar with replication protocol (on the log diverange and reconciliation process), please add your comments on the PR. -- Divij Vaidya On Wed, Sep 27, 2023 at

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
Ismael, Thank you for checking. Multiple other folks have validated after I left the comment here that it doesn't impact log truncation and hence won't lead to data loss. I agree that it's not a blocker. (ref: https://github.com/apache/kafka/pull/14457) -- Divij Vaidya On Wed, Se

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Divij Vaidya
se blocker but we should add it as a fast follow-up. 2. Metric added in https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18 isn't available in documentation (cc: David). I don't consider this as a release blocker but we should add it as a fast follow-up. -- Di

Re: [DISCUSS] 3.5.2 Release

2023-10-12 Thread Divij Vaidya
operational stability-wise, otherwise). For 3.5.2, I think there are sufficient pending (including some CVE fixes) to start a bug fix release. We just need a volunteer for the release manager. -- Divij Vaidya On Thu, Oct 12, 2023 at 9:57 AM Levani Kokhreidze wrote: > Hello, > > KAFKA

Re: [DISCUSS] Apache Kafka 3.5.2 release

2023-10-17 Thread Divij Vaidya
Thank you for volunteering Luke. -- Divij Vaidya On Tue, Oct 17, 2023 at 3:26 PM Bill Bejeck wrote: > Thanks for driving the release, Luke. > > +1 > -Bill > > On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana > wrote: > > > Thanks Luke for volunteering for 3.5.2 r

[DISCUSS] Change website "blog" to "news"

2023-10-24 Thread Divij Vaidya
ilities on PMC). Later, we can work towards adding RSS feed to enable "subscription" to this news and add additional filters such as CVE clarifications etc. What do you think? -- Divij Vaidya

[DISCUSS] How to detect (and prevent) complex bugs in Kafka?

2023-10-24 Thread Divij Vaidya
es.apache.org/jira/browse/KAFKA-15653 [2] https://lists.apache.org/thread/pfrkk0yb394l5qp8h5mv9vwthx15084j -- Divij Vaidya

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

2023-10-27 Thread Divij Vaidya
Congratulations Satish! And thank you for your contributions so far. -- Divij Vaidya On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy wrote: > Congrats! > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar > wrote: > > > > Congrats! > > > > On Fri, Oct

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-10-30 Thread Divij Vaidya
r? Also, I am happy with having a Kafka level dynamic config that we can use to filter our metric/dimensionality since we have a precedence at KIP-544. Hence, my suggestion to push this filtering to metric library can be ignored. -- Divij Vaidya On Sat, Oct 28, 2023 at 11:37 AM Qichao Ch

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-01 Thread Divij Vaidya
ell) 4. what happens when there are contradictory regEx rules, e.g. a topic defined in high as well as low. It is generally solved by defining precedence. In our case, we can choose that high has more precedence than low. What do you think? -- Divij Vaidya On Wed, Nov 1, 2023 at 2:07 PM Qichao Chu

Re: Request for Removal From the Mail Thread

2023-11-03 Thread Divij Vaidya
Hey Dasun Subscribe/unsubscribe to the mailing list is self service. You can find the details at https://kafka.apache.org/contact -- Divij Vaidya On Fri, Nov 3, 2023 at 7:37 PM Dasun Nirmitha wrote: > Hello Apache Kafka Mail Thread, > Thanks for having me in your mail thread for a

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-07 Thread Divij Vaidya
time to think about the implications of current approach towards extendibility to other dimensions in future (such as e.g. client language instead of partition). I will get back to you before the end of this week. -- Divij Vaidya On Tue, Nov 7, 2023 at 4:05 PM Qichao Chu wrote: > Hi Di

Re: tiered storage - remote data to topic binding

2023-11-08 Thread Divij Vaidya
f not, RemoteLogSegmentMetadata also contains a map, CustomMetadata which is opaque to the broker and is populated by RemoteLogMetadataManager. You can choose to feed any attributes you require into this custom metadata and read it in RSM. Does this answer your question? -- Divij Vaidya On Wed, Nov 8, 20

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-13 Thread Divij Vaidya
viewers in allocating their bandwidth effectively. -- Divij Vaidya On Sun, Nov 12, 2023 at 2:58 AM Justine Olshan wrote: > I will say that I have also seen tests that seem to be more flaky > intermittently. It may be ok for some time and suddenly the CI is > overloaded and we see is

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-13 Thread Divij Vaidya
Thank you for updating the KIP Qichao. I don't have any more questions or suggestions. Looks good to move forward from my perspective. -- Divij Vaidya On Fri, Nov 10, 2023 at 2:25 PM Qichao Chu wrote: > Thank you again for the nice suggestions, Jorge! > I will wait for Divij

Apache Kafka 3.6.1 release

2023-11-13 Thread Divij Vaidya
e are targeting for 3.6.1 along with the detailed timeline. If anyone is interested in releasing this sooner, please feel free to take over from me. Thanks! Regards, Divij Vaidya Apache Kafka Committer [1] https://issues.apache.org/jira/browse/KAFKA-15653 [2] https://issues.apache.org/jira/browse/

Re: [DISCUSS] KIP-1002: Fetch remote segment indexes at once

2023-11-13 Thread Divij Vaidya
As a future step in Tiered Storage, I would actually prefer to move towards a direction where we are lazily fetching indexes on-demand instead of fetching them together as proposed in the KIP. -- Divij Vaidya On Fri, Nov 10, 2023 at 4:00 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.co

Re: Apache Kafka 3.6.1 release

2023-11-13 Thread Divij Vaidya
o, my offer to volunteer for Jan release could be considered as a backup. -- Divij Vaidya On Mon, Nov 13, 2023 at 3:40 PM Ismael Juma wrote: > Hi Divij, > > I think we should be releasing 3.6.1 this year rather than next. There are > some critical bugs in 3.6.0 and I don't thin

Re: Apache Kafka 3.6.1 release

2023-11-13 Thread Divij Vaidya
ffort to merge some of the critical ones by next week. I will nudge the contributors where things are pending for a while. [1] https://issues.apache.org/jira/browse/KAFKA-15420 -- Divij Vaidya On Mon, Nov 13, 2023 at 4:10 PM Mickael Maison wrote: > Hi Divij, > > You beat me to i

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-13 Thread Divij Vaidya
s://kafka.apache.org/contributing -> Finding a project to work on Divij Vaidya On Mon, Nov 13, 2023 at 4:24 PM Николай Ижиков wrote: > > > To kickstart this effort, we can publish a list of such tickets in the > community and assign one or more committers the role of a «shepherd&quo

Re: [DISCUSS] KIP-1002: Fetch remote segment indexes at once

2023-11-14 Thread Divij Vaidya
1] https://github.com/apache/kafka/blob/832627fc78484fdc7c8d6da8a2d20e7691dbf882/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1310 [2] https://github.com/apache/kafka/blob/832627fc78484fdc7c8d6da8a2d20e7691dbf882/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1358 -- Di

Important Security Notice for Apache Kafka Users

2023-11-14 Thread Divij Vaidya
ny further questions regarding this, please don't hesitate to reach out to us at secur...@kafka.apache.org or post a comment at https://issues.apache.org/jira/browse/KAFKA-15658 Best Regards, Divij Vaidya On behalf of Apache Kafka PMC [1] https://zookeeper.apache.org/security.html#CVE-2023-

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

2023-11-15 Thread Divij Vaidya
+1 (binding) I was involved in the discussion thread for this KIP and support it in its current form. -- Divij Vaidya On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu wrote: > Hi all, > > I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics. > > Ple

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

2023-11-20 Thread Divij Vaidya
+ 1 (binding) This Kip will greatly improve Tiered Storage troubleshooting. Thank you Christo. On Mon 20. Nov 2023 at 17:21, Christo Lolov wrote: > Hello all! > > Now that the discussion for KIP-963 has winded down, I would like to open > it for a vote targeting 3.7.0 as the release. You can fi

Re: [DISCUSS] KIP-1052: Align the naming convention for config and default variables in *Config classes

2024-06-06 Thread Divij Vaidya
, just to get a better aligned conventional does not seem worthwhile to me. Is there an obvious benefit that I am missing here which would make this proposal a good trade off with the cost? — Divij Vaidya On Thu 6. Jun 2024 at 21:13, Eric Lu wrote: > Hi, > > I wanted to follow-

Re: [DISCUSS] KIP-1046: Expose producer.id.expiration.check.interval.ms as dynamic broker configuration

2024-06-06 Thread Divij Vaidya
approach (and other alternatives) to solve this issue? — Divij Vaidya On Thu 16. May 2024 at 22:15, Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Thanks Justine. I have updated the KIP with the configuration details. > > On Thu, 16 May 2024 at 21:14, Justine Ol

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

2024-06-16 Thread Divij Vaidya
ibility will be delegated to the decoder class which the operator can define using the flag "--value-decoder-class". Is there a reason that you are unable to use the rejected solution (which requires no changes) for debugging purposes? -- Divij Vaidya On Sat, Jun 15, 2024 at 4:43 PM Federico

Re: [DISCUSS] KIP-1052: Enable warmup in producer performance test

2024-06-16 Thread Divij Vaidya
cer buffer) but we have started recording end-to-end latency for the steady-state messages. I imagine this should be ok because it achieves the purpose of removing bootstrap times, but I haven't been able to reason about it in my head. What are your thoughts on this? -- Divij Vaidya On Fr

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

2024-06-17 Thread Divij Vaidya
oposed API take "segmentId" as a parameter instead of "topicIdPartition"? Suggesting because isTxnIdEmpty is not a property of a partition, instead it's a property of a specific segment. Looking forward to hearing your thoughts about the alternatives. Let's get this f

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

2024-06-25 Thread Divij Vaidya
ve to iterate through the metadata for the dominant case where the partition had no transactions at all. -- Divij Vaidya On Tue, Jun 25, 2024 at 11:42 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Bump. Please review this proposal. > > > On Mon, Jun 17

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

2024-06-25 Thread Divij Vaidya
ecify a particular schema version? -- Divij Vaidya On Thu, Jun 20, 2024 at 5:57 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Federico, > > Thanks for the KIP! +1 from me. > > On Wed, Jun 19, 2024, 17:36 Luke Chen wrote: > > > Hi Federico, &

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

2024-06-26 Thread Divij Vaidya
Thank you Federico for answering the questions. No more questions/concerns from me. The KIP looks good. -- Divij Vaidya On Wed, Jun 26, 2024 at 11:02 AM Federico Valeri wrote: > Hi Divij, thanks for you questions and suggestions, much appreciated. > > On Tue, Jun 25, 2024 at 1:12

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

2024-06-27 Thread Divij Vaidya
+1 (binding) I have participated in the discussion and agree with the proposal. -- Divij Vaidya On Thu, Jun 27, 2024 at 12:56 PM Satish Duggana wrote: > Thanks Federico for the KIP. > > +1 > > ~Satish. > > On Thu, 27 Jun 2024 at 13:44, Federico Valeri > wrote: >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-07-02 Thread Divij Vaidya
OffsetMovedToTieredException 3. Follower builds aux state from [0-10] and sets the fetch offset to 11 4. But leader has already uploaded more data and now the new last-tiered-offset is 15 5. Go back to 2 This could cause a cycle where the replica will be stuck trying to reconcile with the leader. -- Divij Vaidya On Fri

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-07-02 Thread Divij Vaidya
te to respond to the follower. The amount of this data can be controlled via the configured value Z which can be set based on how aggressive the upload/archival process is. -- Divij Vaidya On Tue, Jul 2, 2024 at 12:25 PM Divij Vaidya wrote: > Hi folks. > > I am late to the party bu

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-07-23 Thread Divij Vaidya
gt; available anymore, the follower will eventually be able to catch up with > the leader by resetting its fetch offset until the offset > is available on the leader's local log. Once it catches up, replication on > the follower can proceed. > > Regards, > Abhijeet. > > >

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-07-24 Thread Divij Vaidya
world. I am ok with adding a note in the KIP but the note should say that it has an elevated risk for this scenario due to increased probability of having an aggressive local cleanup with Tiered Storage. -- Divij Vaidya On Wed, Jul 24, 2024 at 1:22 PM Abhijeet Kumar wrote: > Hi Divij, >

Re: [DISCUSS] Road to Kafka 4.0

2023-12-21 Thread Divij Vaidya
tely new functionality. [1] https://issues.apache.org/jira/browse/KAFKA-15495 [2] https://issues.apache.org/jira/browse/KAFKA-15489 [3] https://issues.apache.org/jira/browse/KAFKA-14874 -- Divij Vaidya On Wed, Dec 20, 2023 at 4:59 PM Josep Prat wrote: > Hi Justine, Luke, and others, >

Re: [DISCUSS] Road to Kafka 4.0

2023-12-21 Thread Divij Vaidya
ous other things (I came across another one https://issues.apache.org/jira/browse/KAFKA-16041) that we can start doing now for 4.x. What do you think? -- Divij Vaidya On Thu, Dec 21, 2023 at 4:30 PM David Jacot wrote: > Hi Divij, > > > Release 4.0 as an "experimental" release

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2023-12-22 Thread Divij Vaidya
r upgrade on a Kafka version. From 4.0, we will of course, default to using incrementalAlterConfig. -- Divij Vaidya On Fri, Dec 22, 2023 at 6:54 AM ziming deng wrote: > +1 for adding them to rejected alternatives, These kafka-ui tools should > also evolve with the iterations of Kafk

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

2023-12-22 Thread Divij Vaidya
on and compression that occurs during message version transformation, i.e. when client send message with V1 and broker expects in V2, we convert the message and recompress it. -- Divij Vaidya On Mon, Dec 18, 2023 at 7:22 PM Diop, Assane wrote: > I would like to bring some attention to

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

2023-12-22 Thread Divij Vaidya
error means for operations that can be performed on a partition. [1] https://github.com/apache/kafka/blob/82808873cbf6a95611243c2e7984c4aa6ff2cfff/core/src/main/scala/kafka/log/UnifiedLog.scala#L1336 -- Divij Vaidya On Tue, Dec 12, 2023 at 9:58 AM Kamal Chandraprakash < kamal.chandraprak..

Re: Kafka trunk test & build stability

2023-12-22 Thread Divij Vaidya
shared here. -- Divij Vaidya On Fri, Dec 22, 2023 at 2:05 PM Luke Chen wrote: > Hi Sophie and Philip and all, > > I share the same pain as you. > I've been waiting for a CI build result in a PR for days. Unfortunately, I > can only get 1 result each day because it takes 8 hou

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2023-12-25 Thread Divij Vaidya
, this incompatibility will be part of one amongst many changes that users will perform to upgrade to 4.x. This is unlike a major version change from 3.7 to 3.8 where users expect a simple upgrade without any code changes. Let's wait and hear what others think about this. -- Divij Vaidya O

Re: [DISCUSS] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2023-12-26 Thread Divij Vaidya
anecdote, a recent performance regression was found in 17, https://bugs.openjdk.org/browse/JDK-8317960, which was already fixed in 21) -- Divij Vaidya On Tue, Dec 26, 2023 at 9:58 PM Ismael Juma wrote: > Hi Colin, > > A couple of comments: > > 1. It is true that full support for

Re: Kafka trunk test & build stability

2023-12-27 Thread Divij Vaidya
I have started to perform an analysis of the OOM at https://issues.apache.org/jira/browse/KAFKA-16052. Please feel free to contribute to the investigation. -- Divij Vaidya On Wed, Dec 27, 2023 at 1:23 AM Justine Olshan wrote: > I am still seeing quite a few OOM errors in the builds and I

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

2023-12-28 Thread Divij Vaidya
Thank you everyone for your warm wishes 🙏 -- Divij Vaidya On Thu, Dec 28, 2023 at 2:37 PM Yash Mayya wrote: > Congratulations Divij! > > On Wed, Dec 27, 2023 at 5:15 PM Luke Chen wrote: > > > Hi, Everyone, > > > > Divij has been a Kafka committer since Ju

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

2024-01-02 Thread Divij Vaidya
interfaces in the KIP. -- Divij Vaidya On Tue, Nov 21, 2023 at 2:19 PM Christo Lolov wrote: > Heya! > > Thanks a lot for this. I have updated the KIP to include exposing the > tiered-offset as well. Let me know whether the Public Interfaces section > needs more explanation

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2024-01-03 Thread Divij Vaidya
+1 (binding) -- Divij Vaidya On Wed, Jan 3, 2024 at 11:06 AM Viktor Somogyi-Vass wrote: > Hi Ismael, > > I think it's important to make this change, the youtube video you posted on > the discussion thread makes very good arguments and so does the KIP. Java 8 > is almost

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

2024-01-05 Thread Divij Vaidya
+1 (binding) -- Divij Vaidya On Thu, Dec 21, 2023 at 10:30 AM Luke Chen wrote: > Hi Kamal, > > Thanks for the KIP. > +1 (binding) from me. > > Luke > > On Thu, Dec 21, 2023 at 4:51 PM Christo Lolov > wrote: > > > Heya Kamal, > > > > The propos

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

2024-01-05 Thread Divij Vaidya
Thank you for addressing my concerns Kamal. Though, instead of the KIP, I actually was suggesting to add it in JavaDoc so that someone looking at the exception is able to understand what it means. We can discuss that during the PR review though. The KIP looks good to me. -- Divij Vaidya On

Re: Kafka trunk test & build stability

2024-01-10 Thread Divij Vaidya
ith AWS & Apache Infra in the next 1-2 months. -- Divij Vaidya On Tue, Jan 9, 2024 at 9:21 PM Colin McCabe wrote: > Sorry, but to put it bluntly, the current build setup isn't good enough at > partial rebuilds that build caching would make sense. All Kafka devs have > had t

Re: [PROPOSAL] Add commercial support page on website

2024-01-10 Thread Divij Vaidya
her to add them or not? If we keep a blanket criteria of accepting all PRs, then we may end up in a situation where the llink redirects to a phishing page or nefarious website. Hence, we might have to at least perform some basic due diligence which adds overhead to the resources of the community.

Re: Kafka 3.0 support Java 8

2024-01-10 Thread Divij Vaidya
/pages/viewpage.action?pageId=284789510 Does this answer your question? -- Divij Vaidya On Wed, Jan 10, 2024 at 9:37 PM Devinder Saggu wrote: > Hi, > > I wonder how long Kafka 3.0 can support Java 8. > > Thanks & Regards, > > *Devinder Singh* > P *Please consider the environment before printing this email* >

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

2024-01-11 Thread Divij Vaidya
Thank you for making the change Christo. It looks good to me. -- Divij Vaidya On Thu, Jan 11, 2024 at 11:19 AM Christo Lolov wrote: > Thank you Divij! > > I have updated the KIP to explicitly state that the broker will have a > different behaviour when a timestamp of -5 is reque

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

2024-01-11 Thread Divij Vaidya
+1 (binding) Divij Vaidya On Tue, Dec 26, 2023 at 7:05 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > +1 (non-binding). Thanks for the KIP! > > -- > Kamal > > On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov > wrote: > > > Heya all

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-23 Thread Divij Vaidya
+1 (binding) I have participated in the discussion for this and looked at the most recent version of this KIP. It looks good to me. -- Divij Vaidya On Tue, Jan 23, 2024 at 8:17 AM David Jacot wrote: > Hi Chris, Ziming, > > Thanks for the clarification. I am glad that it does not i

Re: Apache Kafka 3.7.0 Release

2024-02-02 Thread Divij Vaidya
ed+Storage+Early+Access+Release+Notes -- Divij Vaidya On Thu, Jan 11, 2024 at 1:03 PM Luke Chen wrote: > Hi all, > > There is a bug KAFKA-16101 > <https://issues.apache.org/jira/browse/KAFKA-16101> reporting that "Kafka > cluster will be unavailable during KRaft migrat

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

2024-02-07 Thread Divij Vaidya
compression controls in Kafka? It might be beneficial to have a joint proposal of these two KIPs which may satisfy both use cases. -- Divij Vaidya On Wed, Feb 7, 2024 at 11:14 AM Mickael Maison wrote: > Hi, > > I'm resurrecting this old thread as this KIP would be a nice > improve

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

2024-02-08 Thread Divij Vaidya
Sounds good. I am onboard to start with first steps and eventually move towards a place where compression codec settings are more generic / pluggable. -- Divij Vaidya On Wed, Feb 7, 2024 at 3:40 PM Mickael Maison wrote: > Hi Divij, > > Thanks for bringing that point. After reading K

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Divij Vaidya
test results Test: Spot check the results of system tests. Result: I have verified that the system tests are passing across different runs. -- Divij Vaidya On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski wrote: > The latest system test build completed successfully - > > https

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Divij Vaidya
kafka binaries. I am +1 (binding) assuming Mickael's concerns about update notes to 3.7 are addressed before release. -- Divij Vaidya On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison wrote: > Hi, > > I agree with Josep, I don't think it's worth making a new RC j

Re: [VOTE] 3.7.0 RC4

2024-02-20 Thread Divij Vaidya
e your PRs on kafka-site repo get merged. 4. Send a release announcement by email. -- Divij Vaidya On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski wrote: > Thanks for testing the release! And thanks for the review on the > documentation. Good catch on the license too. > > I have

Re: Request to assign an issue (KAFKA-4094)

2024-02-25 Thread Divij Vaidya
Hey Vaibhav I have provided you with contributor permission to the JIRA. You should be able to assign the JIRA to yourself now. -- Divij Vaidya On Sun, Feb 25, 2024 at 12:23 AM Vaibhav Kushwaha wrote: > Hi team! > > I was going through the list of starter bugs and found one I coul

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-02-27 Thread Divij Vaidya
Thank you for volunteering Josep. +1 from me. -- Divij Vaidya On Tue, Feb 27, 2024 at 9:35 AM Bruno Cadonna wrote: > Thanks Josep! > > +1 > > Best, > Bruno > > On 2/26/24 9:53 PM, Chris Egerton wrote: > > Thanks Josep, I'm +1 as well. > > > >

Re: [VOTE] 3.7.0 RC4

2024-02-27 Thread Divij Vaidya
ntral (mvnrepository.com or maven.org)." in the release process wiki. -- Divij Vaidya On Tue, Feb 27, 2024 at 4:43 PM Stanislav Kozlovski wrote: > Hey all, > > Everything site-related is merged. > > I have been following the final steps of the release process. > - Docker

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 3.7.0

2024-02-28 Thread Divij Vaidya
ful changes in this release but the one closest to my heart is https://issues.apache.org/jira/browse/KAFKA-15046. I am very glad this is fixed. The P999 latency spikes were driving me crazy for a long time now. -- Divij Vaidya On Wed, Feb 28, 2024 at 10:06 AM Satish Duggana wrote: > Thanks Sta

Re: KIP process

2024-03-07 Thread Divij Vaidya
. You can either create a KIP offline and share it in the discussion email or I can create the KIP page on your behalf and copy/paste the content that you send to me. -- Divij Vaidya On Thu, Mar 7, 2024 at 5:15 PM Arpit Goyal wrote: > + @Kamal Chandraprakash @Vaidya, Divij > Can you h

[DISCUSS] Minimum constraint for segment.ms

2024-03-11 Thread Divij Vaidya
t want to work with a segment.ms of less than ~10s (number chosen arbitrary that looks sane) What was the historical context of setting the minimum threshold to 1ms for this setting? [1] https://kafka.apache.org/documentation.html#topicconfigs_segment.ms -- Divij Vaidya

Re: [DISCUSS] Minimum constraint for segment.ms

2024-03-13 Thread Divij Vaidya
once it is ready to be published, we can start a discussion thread on it. I am also going to start a thread to solicit feedback from users@ mailing list as well. -- Divij Vaidya On Wed, Mar 13, 2024 at 12:55 PM Christopher Shannon < christopher.l.shan...@gmail.com> wrote: > I thi

Re: [DISCUSS] Minimum constraint for segment.ms

2024-03-13 Thread Divij Vaidya
adjusted. Please reply in this thread directly. -- Divij Vaidya Apache Kafka PMC On Wed, Mar 13, 2024 at 12:56 PM Divij Vaidya wrote: > Thanks for the discussion folks. I have started a KIP > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+valu

Re: [DISCUSS] Apache Kafka 3.6.2 release

2024-03-13 Thread Divij Vaidya
+1 Thank you for volunteering. -- Divij Vaidya On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan wrote: > Thanks Manikumar! > +1 from me > > Justine > > On Wed, Mar 13, 2024 at 8:52 AM Manikumar > wrote: > > > Hi, > > > > I'd like to volunteer

Re: [DISCUSS] Apache Kafka 3.6.2 release

2024-03-14 Thread Divij Vaidya
.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC -- Divij Vaidya On Thu, Mar 14, 2024 at 7:55 AM Manikumar wrote: > Hi all, > > Here is the release plan for 3.6.2: > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2 > > Currently there is one open non-

Re: [VOTE] 3.6.2 RC2

2024-04-04 Thread Divij Vaidya
zstd) Things look good to me. We don't need another RC for fixing docs. +1 (binding) from me. -- Divij Vaidya On Thu, Apr 4, 2024 at 10:04 AM Manikumar wrote: > Hi Justine, > > Thanks for catching this. looks like we have missed updating > `docs/documentation.html` in kafka re

  1   2   3   4   5   >