Re: Manage dynamic members through java admin client

[Chia-Ping Tsai]

hi Armando

Thanks for your question and contribution. Few questions are listed below.
PTAL

> I'm working in a java service that allows managing consumer group's
members, specially to pause/resume them on demand.

Out of curiosity, could you elaborate on how the
removeMembersFromConsumerGroup method will manage dynamic members? Even
after removing a dynamic member from the group, the consumer has the
potential to rejoin the group at any time.

> MemberToRemove class is using a
static value "JoinGroupRequest.UNKNOWN_MEMBER_ID" for the memberId field
that is being used in the member deletion logic, so it is expected that
kafka service is responding such thing.

I understand that MemberToRemove is designed to remove either a specific
static member or "all" members from the group. Since dynamic members have
randomly assigned IDs, I'm curious how you plan to pause a specific dynamic
consumer. More specifically, how does the administrator obtain the (random)
member ID? I assume this is achieved by comparing the clientId and host
information returned by `describeConsumerGroups`?

Best,
Chia-Ping



Armando Soriano  於 2025年2月4日 週二 下午8:30寫道：

> Hi there,
>
> I'm working in a java service that allows managing consumer group's
> members, specially to pause/resume them on demand. After checking here and
> there it seems that the actual AdminClient makes this feasible through
> removeMembersFromConsumerGroup method [
>
> https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html#removeMembersFromConsumerGroup(java.lang.String,org.apache.kafka.clients.admin.RemoveMembersFromConsumerGroupOptions)
> ].
> The point is that trying to implement the solution, found that somehow the
> client was returning an error saying something like "unknown member is not
> found". Jumped to the actual sources to find out the reasons and found that
> actually the error was accurate, cause the MemberToRemove class is using a
> static value "JoinGroupRequest.UNKNOWN_MEMBER_ID" for the memberId field
> that is being used in the member deletion logic, so it is expected that
> kafka service is responding such thing.
>
> As a PoC, I just updated the code to be able to set a given memberId value
> (extracted from the describeConsumerGroups method) and it just worked as
> expected. I created a PR https://github.com/apache/kafka/pull/18738 as a
> quick explanation to apply to ASF Jira. In that PR I was gently asked to
> start the conversation through this mailing list, so here we are.
>
> I figured out that to make sure that members are not connecting again to
> the consumer group, we'd require the listener (we're using spring-kafka) to
> stop trying. Guess it could be done by pausing it from the service code
> itself, by watching the reason the disconnection happened (this thing we
> pass on removeMembersFromConsumerGroup). But everything lies on being able
> to drop the members by setting the proper value as memberId from the admin
> client.
>
> Do you think it makes sense to be able to handle the members like that? Are
> there any security or kafka-wise service malfunction implications in
> handling them that way?
>
> Thanks in advance,
>

Re: [DISCUSS] KIP-1126: Serialize changes to Kafka with a build queue

[David Arthur]

Ismael, thanks for taking a look!

We would need to alter the default commit message setting for the
repository. The default (which we currently use) is PR title + PR body if
there are multiple commits, or the commit subject and body if it's a single
commit PR. We can change this to always be the PR title and PR body.

Additionally, we can add a status check that validates that the PR body has
the expected "Reviewers" field before merging. It might also be possible to
automatically set that field.

I have a feeling it might be possible to alter the commit message in the
merge queue job, but that will take some experimentation.

-David A

On Tue, Feb 4, 2025 at 5:09 AM Ismael Juma  wrote:

> This looks good to me.
>
> My only comment is that merge queues have a weird limitation: you cannot
> edit the commit message (unlike the traditional flow). I worry that this
> will result in lower quality commit messages. Is there a way to configure
> where the commit message comes from (PR title/description versus
> aggregating all the commit messages in the PR)?
>
> Ismael
>
> On Mon, Feb 3, 2025, 8:02 AM David Arthur  wrote:
>
> > Happy Monday, all!
> >
> > I'd like to close out this discussion at some point this week, so I
> wanted
> > to bump this up once before moving to a vote.
> >
> > Did anyone have other feedback or concerns about moving towards the
> GitHub
> > Merge Queue?
> >
> > Thanks!
> > David A
> >
> > On Thu, Jan 30, 2025 at 11:11 AM David Arthur  wrote:
> >
> > > Uchechi,
> > >
> > > While the build queue can run the builds concurrently, each build is
> > > cumulative.
> > >
> > > For example, A, B, and C are added to the queue and A causes a problem
> > for
> > > C.
> > >
> > > Merge Queue: A, B, C
> > >
> > > Suppose we have concurrent builds and batching turned on. We might end
> up
> > > with two builds for these three PRs:
> > >
> > > Build 1: trunk + A
> > > Build 2: trunk + A + B + C
> > >
> > > Each build is going to be cumulative. The point of the concurrency is
> so
> > > we don't have to wait for an in-progress build to finish before we
> start
> > > validating new PRs from the queue. In this case, build 2 would fail and
> > PRs
> > > B and C would get kicked back to the authors.
> > >
> > > As you can see, the batching and concurrency can complicate matters a
> bit
> > > which is why I suggest we start with no batching or concurrency.
> > >
> > > Hope this helps!
> > > David A
> > >
> > > On Wed, Jan 29, 2025 at 2:34 AM Ukpa Uchechi 
> > > wrote:
> > >
> > >> Hi David,
> > >>
> > >> Thanks for this and for your work improving the Kafka CI.
> > >>
> > >> I have a question:
> > >>
> > >> A problem was mentioned about two PRs acting against each other when
> > >> there’s no delay and being merged concurrently. Wouldn’t this still
> > happen
> > >> in the queue with concurrent merging as well?
> > >> Best Regards,
> > >> Uchechi.
> > >>
> > >> On Fri, 24 Jan 2025 at 03:35, David Arthur  wrote:
> > >>
> > >> > Greetings, Kafka community!
> > >> >
> > >> > At long last, the GitHub Merge Queue is upon us. This is a feature
> > that
> > >> > many of us have wanted for quite a while. After many months of
> > >> discussion,
> > >> > the excellent ASF Infra team has delivered!
> > >> >
> > >> > Since this is quite a significant change, I have written up the
> > details
> > >> as
> > >> > a KIP.
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1126%3A+Serialize+changes+to+Kafka+with+build+queue
> > >> >
> > >> > Please let me know what you think.
> > >> >
> > >> > --
> > >> > David Arthur
> > >> >
> > >>
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> >
> > --
> > David Arthur
> >
>


-- 
David Arthur

[jira] [Created] (KAFKA-18725) Fix test_bounce.py sys test

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-18725:
--

 Summary: Fix test_bounce.py sys test 
 Key: KAFKA-18725
 URL: https://issues.apache.org/jira/browse/KAFKA-18725
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


we should just remove the zk test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18726) Support for root cause exception as a nested exception to TimeoutException for Producer framework

[Vineet Kumar (Jira)]

Vineet Kumar created KAFKA-18726:


 Summary: Support for root cause exception as a nested exception to 
TimeoutException for Producer framework
 Key: KAFKA-18726
 URL: https://issues.apache.org/jira/browse/KAFKA-18726
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.7.2
Reporter: Vineet Kumar
Assignee: Vineet Kumar


* On invoking getCause() method on throwable object returned by KafkaProducer 
class it returns null value



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1126: Serialize changes to Kafka with a build queue

[Ismael Juma]

Hi David,

Setting the repo config to always be PR title & body would be good as it's
a lot more predictable (most PRs have more than one commit by the end of
the PR review, but there are exceptions and it's harder for a committer to
update the commit message in such cases).

The reviewers verification makes sense. It would also be good to do
something about the "Committer Checklist" bit - it would be annoying to
have that in every commit message. We can either remove it from the
template (if we think it doesn't add value), automatically remove it during
the merge queue job or have a status check for it (along with the reviewers
bit).

Ismael

On Tue, Feb 4, 2025 at 6:55 AM David Arthur  wrote:

> Ismael, thanks for taking a look!
>
> We would need to alter the default commit message setting for the
> repository. The default (which we currently use) is PR title + PR body if
> there are multiple commits, or the commit subject and body if it's a single
> commit PR. We can change this to always be the PR title and PR body.
>
> Additionally, we can add a status check that validates that the PR body has
> the expected "Reviewers" field before merging. It might also be possible to
> automatically set that field.
>
> I have a feeling it might be possible to alter the commit message in the
> merge queue job, but that will take some experimentation.
>
> -David A
>
> On Tue, Feb 4, 2025 at 5:09 AM Ismael Juma  wrote:
>
> > This looks good to me.
> >
> > My only comment is that merge queues have a weird limitation: you cannot
> > edit the commit message (unlike the traditional flow). I worry that this
> > will result in lower quality commit messages. Is there a way to configure
> > where the commit message comes from (PR title/description versus
> > aggregating all the commit messages in the PR)?
> >
> > Ismael
> >
> > On Mon, Feb 3, 2025, 8:02 AM David Arthur  wrote:
> >
> > > Happy Monday, all!
> > >
> > > I'd like to close out this discussion at some point this week, so I
> > wanted
> > > to bump this up once before moving to a vote.
> > >
> > > Did anyone have other feedback or concerns about moving towards the
> > GitHub
> > > Merge Queue?
> > >
> > > Thanks!
> > > David A
> > >
> > > On Thu, Jan 30, 2025 at 11:11 AM David Arthur 
> wrote:
> > >
> > > > Uchechi,
> > > >
> > > > While the build queue can run the builds concurrently, each build is
> > > > cumulative.
> > > >
> > > > For example, A, B, and C are added to the queue and A causes a
> problem
> > > for
> > > > C.
> > > >
> > > > Merge Queue: A, B, C
> > > >
> > > > Suppose we have concurrent builds and batching turned on. We might
> end
> > up
> > > > with two builds for these three PRs:
> > > >
> > > > Build 1: trunk + A
> > > > Build 2: trunk + A + B + C
> > > >
> > > > Each build is going to be cumulative. The point of the concurrency is
> > so
> > > > we don't have to wait for an in-progress build to finish before we
> > start
> > > > validating new PRs from the queue. In this case, build 2 would fail
> and
> > > PRs
> > > > B and C would get kicked back to the authors.
> > > >
> > > > As you can see, the batching and concurrency can complicate matters a
> > bit
> > > > which is why I suggest we start with no batching or concurrency.
> > > >
> > > > Hope this helps!
> > > > David A
> > > >
> > > > On Wed, Jan 29, 2025 at 2:34 AM Ukpa Uchechi  >
> > > > wrote:
> > > >
> > > >> Hi David,
> > > >>
> > > >> Thanks for this and for your work improving the Kafka CI.
> > > >>
> > > >> I have a question:
> > > >>
> > > >> A problem was mentioned about two PRs acting against each other when
> > > >> there’s no delay and being merged concurrently. Wouldn’t this still
> > > happen
> > > >> in the queue with concurrent merging as well?
> > > >> Best Regards,
> > > >> Uchechi.
> > > >>
> > > >> On Fri, 24 Jan 2025 at 03:35, David Arthur 
> wrote:
> > > >>
> > > >> > Greetings, Kafka community!
> > > >> >
> > > >> > At long last, the GitHub Merge Queue is upon us. This is a feature
> > > that
> > > >> > many of us have wanted for quite a while. After many months of
> > > >> discussion,
> > > >> > the excellent ASF Infra team has delivered!
> > > >> >
> > > >> > Since this is quite a significant change, I have written up the
> > > details
> > > >> as
> > > >> > a KIP.
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1126%3A+Serialize+changes+to+Kafka+with+build+queue
> > > >> >
> > > >> > Please let me know what you think.
> > > >> >
> > > >> > --
> > > >> > David Arthur
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
>
>
> --
> David Arthur
>

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #200

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

[Bill Bejeck]

Hi All,

Just a quick update on the KIP.  After doing some work with IQv2 we will
need to explicitly return the standby partitions in the
`PartitionsByUserEndpoint` struct.
This update involves adding a field `StandbyPartitions` of type
`[]TopicPartition` in the `IQ-related` section of the response schema.

Thanks,
Bill

On Thu, Jan 2, 2025 at 4:48 AM Lucas Brutschy
 wrote:

> Hi PoAn,
>
> thanks for looking into the KIP!
>
> PY1: You are right, zkBroker was added following KIP-848 during
> development, and is indeed not required anymore. We don't want to
> allow these RPCs on the controller, so we have to remove `zkBroker`,
> not replace it with `controller`. I updated the KIP, I see you already
> fixed it in trunk.
>
> Cheers,
> Lucas
>
> On Thu, Dec 26, 2024 at 11:28 AM PoAn Yang  wrote:
> >
> > Hi Lucas,
> >
> > Thanks for the KIP.
> >
> > PY1: In StreamsGroupHeartbeatRequest, it uses broker and zkBroker in
> listeners field.
> > Is this intended? IIUC, the KIP will be implemented in 4.1. The
> zookeeper will be
> > removed in 4.0. Probably, we should change it as broker and controller.
> > We may need a similar change for StreamsGroupDescribeRequest.
> >
> > Best,
> > PoAn
> >
> > On 2024/09/02 12:23:22 Bruno Cadonna wrote:
> > > Hi all,
> > >
> > > For your info, I updated the StreamsGroupInitialize request with the
> > > following changes:
> > >
> > > 1. I added the topology ID to the request so that the group coordinator
> > > knows for which topology it got the initialization.
> > >
> > > 2. I renamed field "Subtopology" to "SubtopologyId" since the field is
> > > the ID of the subtopology but that was not clear from the name.
> > >
> > > Best,
> > > Bruno
> > >
> > >
> > > On 8/28/24 2:06 PM, Lucas Brutschy wrote:
> > > > Hi Sophie,
> > > >
> > > > Thanks for your detailed comments - much appreciated! I think you
> read
> > > > a version of the KIP that did not yet include the admin command-line
> > > > tool and the Admin API extensions, so some of the comments are
> already
> > > > addressed in the KIP.
> > > >
> > > > S1. StreamsGroupHeartbeat and StreamsGroupInitialize are called in
> the
> > > > consumer background thread. Note that in the new consumer threading
> > > > model, all RPCs are run by the background thread. Check out this:
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+refactor+design
> > > > for more information. Both our RPCs are just part of the group
> > > > membership management and do not need to be invoked explicitly by the
> > > > application thread. You could imagine calling the initialize RPC
> > > > explicitly (to implement explicit initialization), but this would
> > > > still mean sending an event to the background thread, and the
> > > > background thread in turn invokes the RPC. However, explicit
> > > > initialization would require some additional public interfaces that
> we
> > > > are not including in this KIP. StreamsGroupDescribe is called by the
> > > > AdminClient, and used by the command-line tool
> > > > kafka-streams-groups.sh.
> > > >
> > > > S2. I think the max.warmup.replicas=100 suggested by Nick was
> intended
> > > > as the upper limit for setting the group configuration on the broker.
> > > > Just to make sure that this was not a misunderstanding. By default,
> > > > values above 100 should be rejected when setting a specific value for
> > > > group. Are you suggesting 20 or 30 for the default value for groups,
> > > > or the default upper limit for the group configuration?
> > > >
> > > > S3. Yes, it's supposed to be used like SHUTDOWN_APPLICATION. The
> > > > MemberEpoch=-1 is a left-over from an earlier discussion. It means
> > > > that the member is leaving the group, so the intention was that the
> > > > member must leave the group when it asks the other members to shut
> > > > down. We later reconsidered this and decided that all applications
> > > > should just react to the shutdown application signal that is returned
> > > > by the broker, so the client first sets the ShutdownApplication and
> > > > later leaves the group. Thanks for spotting this, I removed it.
> > > >
> > > > S4. Not sure if this refers to the latest version of the KIP. We
> added
> > > > an extension of the admin API and a kafka-streams-groups.sh
> > > > command-line tool to the KIP already.
> > > >
> > > > S5. All RPCs for dealing with offsets will keep working with streams
> > > > groups. The extension of the admin API is rather cosmetic, since the
> > > > method names use "consumer group". The RPCs, however, are generic and
> > > > do not need to be changed.
> > > >
> > > > S6. Yes, you can use the DeleteGroup RPC with any group ID, whether
> > > > streams group or not.
> > > >
> > > > S7. See the admin API section.
> > > >
> > > > S8. I guess for both A and B, I am not sure what you are suggesting.
> > > > Do you want to make the broker-side topology immutable and not
> include
> > > > any information about the topology, like the topolo

Re: [VOTE] KIP-1127: Flexible Windows for Late Arriving Data

[Sophie Blee-Goldman]

+1 (binding)

Thanks for the KIP! Neat and practical idea

On Tue, Feb 4, 2025 at 10:52 AM Almog Gavra  wrote:

> Hello All,
>
> I'd like to start a vote on KIP-1127. Please take a look at the KIP and
> Discussion thread and let me know what you think.
>
> Note that the discussion thread was incorrectly prefixed with KIP-1124
> instead of KIP-1127 (oops!).
>
> Link to KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1127+Flexible+Windows+for+Late+Arriving+Data
>
> Thanks,
> Almog
>

Re: [DISCUSS] KIP-1124: Flexible Windows for Late Arriving Data

[Sophie Blee-Goldman]

One minor suggestion: use BatchWindows instead of BatchedWindows. The
version without the "ed" matches up with the established naming pattern and
grammar used by other Windows classes: eg TimeWindows, SessionWindows,
SlidingWindows

Not a big deal though, won't redact my +1 on the voting thread if you
prefer to keep it as BatchedWindows

On Tue, Feb 4, 2025 at 10:51 AM Almog Gavra  wrote:

> Thanks for the discussion everyone! I've updated the Wiki with the
> following changes:
>
> - Renamed to BatchedWindows
> - Add a note in rejected alternatives about more general purpose
> (micro-)batching functionality since the scope of that is much wider.
>
> Since it looks like we've stabilized the discussion I'm going to go ahead
> and open up the vote! Definitely feel free to take another look and leave
> any additional thoughts.
>
>
> On Thu, Jan 30, 2025 at 12:28 PM Matthias J. Sax  wrote:
>
> > > batch window with max N records,
> > >> and then also specifying a BufferConfig.maxRecords()
> >
> > That's actually two different and independent dimensions. "N records"
> > would be the number of records in the window, but `maxRecords` is the
> > number of unique keys/row in the buffer before it's flushed.
> >
> >
> >
> > > the current proposal
> > > that only includes stream time based batches is no worse than existing
> > > windows which also have that potential indeterminism in this scenario
> > > except for that it's more likely since grace period is always 0
> >
> > Guess this becomes a philosophical question. For TimeWindows, even if
> > grace=0, it's still deterministic into which window each record goes --
> > of course, the "cut off point" when we start to drop late records is
> > subject to "noise" due to repartitioning, as stream-time advances "non
> > deterministically".
> >
> > So if we drop different records on a "re-run" on the same input data, we
> > might still get different windows (as different records would have been
> > dropped). But that is exactly why we have a grace-period to begin with
> > (to avoid it, and to make a re-run deterministic -- I think of a "too
> > short" grace period as a misconfiguration -- or an informed decision to
> > sacrifice determinism for something else)...
> >
> > And as the new window type, by definition, does not need/want a
> > grace-period, IMHO, the new window type would be "worse" (for the lack
> > of a better word...); I don't think its really worse, it just inherently
> > non-deterministic, and that's fine.
> >
> > Guess we are overall on the same page and share common understanding of
> > the trade-off. So I think we are good :)
> >
> >
> > -Matthias
> >
> >
> > On 1/30/25 9:07 AM, Almog Gavra wrote:
> > > I'm not opposed to "BatchedWindows" - I think I like that the most so
> > far.
> > > I'll let that sit on the discussion thread for a while, and change the
> > KIP
> > > to match if no concerns.
> > >
> > >> What I don't understand is, why the relationship to
> > > suppress()/emitStrategy() is relevant? Can you elaborate a little bit?
> > >
> > > The existing proposal has no impact on suppression/emitStrategy, but
> I'm
> > > uncertain it's unrelated if you start to introduce window constraints
> > that
> > > aren't just stream time. Imagine having a batch window with max N
> > records,
> > > and then also specifying a BufferConfig.maxRecords() suppression...
> what
> > > happens in that case? With the current scope of the KIP, we don't need
> to
> > > worry about any of that so it's pretty well contained.
> > >
> > >> if data is auto-repartitioned before a `BatchWindows` step,
> > > repartitioning introduces non-deterministic order in the repartition
> > topic
> > >
> > > This is a good point, I did not think about it (my original comment was
> > > specifically about wall clock time). I guess, though, the current
> > proposal
> > > that only includes stream time based batches is no worse than existing
> > > windows which also have that potential indeterminism in this scenario
> > > except for that it's more likely since grace period is always 0.
> > >
> > >> I don't see any relationship to EOS or ALOS. Can you explain what you
> > > mean?
> > >
> > > Hmm, I'm not totally sure what I was thinking... we can ignore that
> part.
> > >
> > > - Almog
> > >
> > >
> > > On Tue, Jan 28, 2025 at 3:40 PM Matthias J. Sax 
> > wrote:
> > >
> > >> Interesting thoughts. So maybe we could go with `BatchWindows` as a
> > >> name? Again, only spit-balling...
> > >>
> > >> If we really put "(micro-)batching" in the center of this idea, I
> think
> > >> both count-based and time-based (and time could actually be either
> > >> stream-time or wall-clock-time), or any combination of these
> dimensions
> > >> could all make sense.
> > >>
> > >> Having said this: I don't think we need to support all dimensions
> > >> initially, but if we add something like `BatchWindows` we can extent
> the
> > >> supported specification incrementally.
> > >>
> > >>
> > >>
> > >> What I don't und

[jira] [Created] (KAFKA-18727) Move RequestChannel to server module

[Jira]

黃竣陽 created KAFKA-18727:
---

 Summary: Move RequestChannel to server module
 Key: KAFKA-18727
 URL: https://issues.apache.org/jira/browse/KAFKA-18727
 Project: Kafka
  Issue Type: Sub-task
Reporter: 黃竣陽
Assignee: 黃竣陽






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18728) Move ListOffsetsPartitionStatus to server module

[Jira]

黃竣陽 created KAFKA-18728:
---

 Summary: Move ListOffsetsPartitionStatus to server module
 Key: KAFKA-18728
 URL: https://issues.apache.org/jira/browse/KAFKA-18728
 Project: Kafka
  Issue Type: Sub-task
Reporter: 黃竣陽
Assignee: 黃竣陽






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18729) Improve consumer check for assigned topics from regex

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-18729:
--

 Summary: Improve consumer check for assigned topics from regex
 Key: KAFKA-18729
 URL: https://issues.apache.org/jira/browse/KAFKA-18729
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Lianet Magrans
 Fix For: 4.1.0


When using subscriptions to broker-side regex RE2J, the consumer checks if a 
topic received in metadata is assigned in order to keep its info. At the 
moment, the consumer has no other way but to check the topic against the list 
of assigned topicPartitions (which is all that's available). 

We should consider improving the check to do it at the topic level, not 
topicPartition level . We would need to keep the set of assigned topics from 
RE2J regex in the subscription state, updated as assignments as reconciled and 
partitions are add/removed. Then we could rely on this topic set for the check 
(not all the assigned topicPartitions set)  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu resolved KAFKA-18635.

Resolution: Fixed

> Re-enable the unclean shutdown detection when in ELR mode
> -
>
> Key: KAFKA-18635
> URL: https://issues.apache.org/jira/browse/KAFKA-18635
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Colin McCabe
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>
> We need to re-enable the unclean shutdown detection when in ELR mode, which 
> was inadvertently removed during the development process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-18538) Add Streams membership manager

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-18538.
---
Resolution: Fixed

> Add Streams membership manager
> --
>
> Key: KAFKA-18538
> URL: https://issues.apache.org/jira/browse/KAFKA-18538
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Bruno Cadonna
>Assignee: Bruno Cadonna
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu reopened KAFKA-18649:


> Complete ClearElrRecord handling in topic delta
> ---
>
> Key: KAFKA-18649
> URL: https://issues.apache.org/jira/browse/KAFKA-18649
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-16540) Update partitions when the min isr config is updated.

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu reopened KAFKA-16540:


> Update partitions when the min isr config is updated.
> -
>
> Key: KAFKA-16540
> URL: https://issues.apache.org/jira/browse/KAFKA-16540
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>
> If the min isr config is changed, we need to update the partitions 
> accordingly. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu reopened KAFKA-18635:


> Re-enable the unclean shutdown detection when in ELR mode
> -
>
> Key: KAFKA-18635
> URL: https://issues.apache.org/jira/browse/KAFKA-18635
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Colin McCabe
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>
> We need to re-enable the unclean shutdown detection when in ELR mode, which 
> was inadvertently removed during the development process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Tagged fields constraint to start from tagId: 0

[Kamal Chandraprakash]

Got it. Thanks for clearing that up!

--
Kamal

On Tue, Feb 4, 2025 at 4:40 PM David Jacot 
wrote:

> Hi Kamal,
>
> It is fine to relax the constraint in StructSpec for your internal tags.
> For instance, you could allow your internal tags to start from id 1. It
> is very unlikely that we will ever have tags as part of the official
> protocol reaching that.
>
> Best,
> David
>
> On Mon, Feb 3, 2025 at 6:36 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Bumping up on this thread.
> >
> > In a forked Kafka repo, if we want to introduce a new tagged field. Which
> > approach should be taken?
> > Is it fine to relax the constraint in StructSpec
> > <
> >
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> > >
> > to
> > start with a very large tag-id like 1 for custom
> > tagged fields to not clash with future tagged fields being introduced
> > upstream? TIA.
> >
> > Thanks,
> > Kamal
> >
> > On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > In KIP-482
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions
> > >,
> > > we have introduced support for optional tagged fields when the
> > > API-message supports "flexibleVersions" feature. One of the motivations
> > of
> > > the KIP is to support attaching extra information to a message without
> > > requiring a version bump for everyone who has to read that message.
> > >
> > > In StructSpec
> > > <
> >
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> > >,
> > > there is a constraint to start the tagId of the tagged field from 0 and
> > > should be contiguous. Once the user introduces a custom taggedField
> that
> > > starts from 0, there is a risk of protocol divergence between the
> server
> > > and client when the upstream/Apache introduces a new taggedField in
> > future
> > > that starts from 0.
> > >
> > > Once a tag-id gets introduced, it cannot be used for anything else. We
> > > want to propagate a custom trace-id in the API-message between
> > > server/client and want to use a large tagId like 10001 (2 bytes varint)
> > to
> > > avoid conflict with the future API message specification.
> > >
> > > We would like to understand the reason behind the tagId constraint to
> > > start from 0 and contiguous. One use-case of having this constraint
> > ensures
> > > that the tag-id starts from 0 and won't be random. If it is random,
> then
> > > the user won't be sure in defining the unique tagged-ids for their
> > > user-defined fields.
> > >
> > > I’d appreciate it if you could take a look and share your thoughts or
> > > feedback.
> > >
> > > Thanks,
> > > Kamal
> > >
> > >
> > >
> > >
> >
>

Manage dynamic members through java admin client

[Armando Soriano]

Hi there,

I'm working in a java service that allows managing consumer group's
members, specially to pause/resume them on demand. After checking here and
there it seems that the actual AdminClient makes this feasible through
removeMembersFromConsumerGroup method [
https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html#removeMembersFromConsumerGroup(java.lang.String,org.apache.kafka.clients.admin.RemoveMembersFromConsumerGroupOptions)].
The point is that trying to implement the solution, found that somehow the
client was returning an error saying something like "unknown member is not
found". Jumped to the actual sources to find out the reasons and found that
actually the error was accurate, cause the MemberToRemove class is using a
static value "JoinGroupRequest.UNKNOWN_MEMBER_ID" for the memberId field
that is being used in the member deletion logic, so it is expected that
kafka service is responding such thing.

As a PoC, I just updated the code to be able to set a given memberId value
(extracted from the describeConsumerGroups method) and it just worked as
expected. I created a PR https://github.com/apache/kafka/pull/18738 as a
quick explanation to apply to ASF Jira. In that PR I was gently asked to
start the conversation through this mailing list, so here we are.

I figured out that to make sure that members are not connecting again to
the consumer group, we'd require the listener (we're using spring-kafka) to
stop trying. Guess it could be done by pausing it from the service code
itself, by watching the reason the disconnection happened (this thing we
pass on removeMembersFromConsumerGroup). But everything lies on being able
to drop the members by setting the proper value as memberId from the admin
client.

Do you think it makes sense to be able to handle the members like that? Are
there any security or kafka-wise service malfunction implications in
handling them that way?

Thanks in advance,

[jira] [Resolved] (KAFKA-18691) Flaky test testFencingOnTransactionExpiration

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-18691.

Fix Version/s: 4.0.0
   Resolution: Fixed

> Flaky test testFencingOnTransactionExpiration
> -
>
> Key: KAFKA-18691
> URL: https://issues.apache.org/jira/browse/KAFKA-18691
> Project: Kafka
>  Issue Type: Bug
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 4.0.0
>
>
> Introduced by 
> [https://github.com/apache/kafka/commit/ccab9eb8b4781eb4b8edbd03b505ae31e1abdff7|https://github.com/apache/kafka/commit/ccab9eb8b4781eb4b8edbd03b505ae31e1abdff7.]
>  I will fix this :) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18730) Add replaying streams group state from offset topic

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-18730:
--

 Summary: Add replaying streams group state from offset topic
 Key: KAFKA-18730
 URL: https://issues.apache.org/jira/browse/KAFKA-18730
 Project: Kafka
  Issue Type: Sub-task
Reporter: Lucas Brutschy
Assignee: Lucas Brutschy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1124: Flexible Windows for Late Arriving Data

[Almog Gavra]

Thanks for the discussion everyone! I've updated the Wiki with the
following changes:

- Renamed to BatchedWindows
- Add a note in rejected alternatives about more general purpose
(micro-)batching functionality since the scope of that is much wider.

Since it looks like we've stabilized the discussion I'm going to go ahead
and open up the vote! Definitely feel free to take another look and leave
any additional thoughts.


On Thu, Jan 30, 2025 at 12:28 PM Matthias J. Sax  wrote:

> > batch window with max N records,
> >> and then also specifying a BufferConfig.maxRecords()
>
> That's actually two different and independent dimensions. "N records"
> would be the number of records in the window, but `maxRecords` is the
> number of unique keys/row in the buffer before it's flushed.
>
>
>
> > the current proposal
> > that only includes stream time based batches is no worse than existing
> > windows which also have that potential indeterminism in this scenario
> > except for that it's more likely since grace period is always 0
>
> Guess this becomes a philosophical question. For TimeWindows, even if
> grace=0, it's still deterministic into which window each record goes --
> of course, the "cut off point" when we start to drop late records is
> subject to "noise" due to repartitioning, as stream-time advances "non
> deterministically".
>
> So if we drop different records on a "re-run" on the same input data, we
> might still get different windows (as different records would have been
> dropped). But that is exactly why we have a grace-period to begin with
> (to avoid it, and to make a re-run deterministic -- I think of a "too
> short" grace period as a misconfiguration -- or an informed decision to
> sacrifice determinism for something else)...
>
> And as the new window type, by definition, does not need/want a
> grace-period, IMHO, the new window type would be "worse" (for the lack
> of a better word...); I don't think its really worse, it just inherently
> non-deterministic, and that's fine.
>
> Guess we are overall on the same page and share common understanding of
> the trade-off. So I think we are good :)
>
>
> -Matthias
>
>
> On 1/30/25 9:07 AM, Almog Gavra wrote:
> > I'm not opposed to "BatchedWindows" - I think I like that the most so
> far.
> > I'll let that sit on the discussion thread for a while, and change the
> KIP
> > to match if no concerns.
> >
> >> What I don't understand is, why the relationship to
> > suppress()/emitStrategy() is relevant? Can you elaborate a little bit?
> >
> > The existing proposal has no impact on suppression/emitStrategy, but I'm
> > uncertain it's unrelated if you start to introduce window constraints
> that
> > aren't just stream time. Imagine having a batch window with max N
> records,
> > and then also specifying a BufferConfig.maxRecords() suppression... what
> > happens in that case? With the current scope of the KIP, we don't need to
> > worry about any of that so it's pretty well contained.
> >
> >> if data is auto-repartitioned before a `BatchWindows` step,
> > repartitioning introduces non-deterministic order in the repartition
> topic
> >
> > This is a good point, I did not think about it (my original comment was
> > specifically about wall clock time). I guess, though, the current
> proposal
> > that only includes stream time based batches is no worse than existing
> > windows which also have that potential indeterminism in this scenario
> > except for that it's more likely since grace period is always 0.
> >
> >> I don't see any relationship to EOS or ALOS. Can you explain what you
> > mean?
> >
> > Hmm, I'm not totally sure what I was thinking... we can ignore that part.
> >
> > - Almog
> >
> >
> > On Tue, Jan 28, 2025 at 3:40 PM Matthias J. Sax 
> wrote:
> >
> >> Interesting thoughts. So maybe we could go with `BatchWindows` as a
> >> name? Again, only spit-balling...
> >>
> >> If we really put "(micro-)batching" in the center of this idea, I think
> >> both count-based and time-based (and time could actually be either
> >> stream-time or wall-clock-time), or any combination of these dimensions
> >> could all make sense.
> >>
> >> Having said this: I don't think we need to support all dimensions
> >> initially, but if we add something like `BatchWindows` we can extent the
> >> supported specification incrementally.
> >>
> >>
> >>
> >> What I don't understand is, why the relationship to
> >> suppress()/emitStrategy() is relevant? Can you elaborate a little bit?
> >>
> >>
> >>
> >>> how to make it deterministic with regards to time
> >>
> >> I think, in any case, we are leaving determinism-land (at least to some
> >> extent). I guess, as long as the `BatchWindows` are applied directly to
> >> an input topic, we get determinism for both stream-time size, as well as
> >> count-size windows (not for wall-clock0time, of course).
> >>
> >> However, if data is auto-repartitioned before a `BatchWindows` step,
> >> repartitioning introduces non-deterministic order in t

[VOTE] KIP-1127: Flexible Windows for Late Arriving Data

[Almog Gavra]

Hello All,

I'd like to start a vote on KIP-1127. Please take a look at the KIP and
Discussion thread and let me know what you think.

Note that the discussion thread was incorrectly prefixed with KIP-1124
instead of KIP-1127 (oops!).

Link to KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1127+Flexible+Windows+for+Late+Arriving+Data

Thanks,
Almog

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #159

[Apache Jenkins Server]

See 

[PR] Add Reddit as powered by Kafka [kafka-site]

[via GitHub]

nickgarvey opened a new pull request, #662:
URL: https://github.com/apache/kafka-site/pull/662

   This adds Reddit to the powered-by.html page as a business which is powered 
by Kafka. We make heavy use of Kafka heavily internally, thank you!
   
   https://kafka.apache.org/powered-by


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu resolved KAFKA-18635.

Resolution: Fixed

> Re-enable the unclean shutdown detection when in ELR mode
> -
>
> Key: KAFKA-18635
> URL: https://issues.apache.org/jira/browse/KAFKA-18635
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Colin McCabe
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>
> We need to re-enable the unclean shutdown detection when in ELR mode, which 
> was inadvertently removed during the development process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16540) Update partitions when the min isr config is updated.

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu resolved KAFKA-16540.

Resolution: Fixed

> Update partitions when the min isr config is updated.
> -
>
> Key: KAFKA-16540
> URL: https://issues.apache.org/jira/browse/KAFKA-16540
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>
> If the min isr config is changed, we need to update the partitions 
> accordingly. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu resolved KAFKA-18649.

Resolution: Fixed

> Complete ClearElrRecord handling in topic delta
> ---
>
> Key: KAFKA-18649
> URL: https://issues.apache.org/jira/browse/KAFKA-18649
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Calvin Liu
>Assignee: Calvin Liu
>Priority: Blocker
> Fix For: 4.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18731) KStream-GlobalKTabel ValueJoinerWithKey passed incorrect key

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-18731:
---

 Summary: KStream-GlobalKTabel ValueJoinerWithKey passed incorrect 
key
 Key: KAFKA-18731
 URL: https://issues.apache.org/jira/browse/KAFKA-18731
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax


We introduce "key access" (rich-)functions in the Kafka Streams DSL via 
KIP-149: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner]
 

In particular, we added `ValueJoinerWithKey` interface. While the KIP is not 
very specific about it, in the motivation section it says:
{quote}it seems like extending the interface to pass the join key along as well 
would be helpful
{quote}
The underlying implementation has `KStreamKTableJoinProcessor` which is used 
for both stream-table and stream-globalTable join. For stream-table join, the 
stream-key y (which is the join key) is passed into `ValueJoinerWithKey`.

For stream-globalTable join, we thus also pass the stream-key, however, for 
this case, the stream-key is not the join-key, but the join-key is computed on 
the fly using the provided `keySelector` (a `KeyValueMapper`).

This seems to be a bug and/or bad design (well, I am sure it was just a small 
detail which was missed).

Overall there is three options:
 # keep the code as-is (does not seem to be "correct", in the spirit of the KIP)
 # declare it as a bug, and just change it
 # do a follow up KIP to change stream-globalTable join to pass in both, the 
stream-key and the join-key (whit would require a KIP)

For other joins, ie, stream-stream or table-table there is no such issue, as 
the input-key is the join-key (similar to stream-table join). The only other 
exception would be fk-table-table join which was added much later; however, it 
only takes a `ValueJoiner` and thus also does not see this issue. If we go with 
option (3), and do KIP, we should consider to support `ValueJoinerWithKey` for 
FK-table-table joins, too. If we go with option (2), we should file a follow up 
ticket for this idea to extend FK-table-table join in this way.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 4.0.0 release

[José Armando García Sancio]

Hi David,

Thanks again for being the release manager for 4.0.0.

I have this bug that I would like to include in 4.0.0:
https://issues.apache.org/jira/browse/KAFKA-18723. I explained in a
comment that it is not a regression but it would be good to include
the fix if it makes the RC.

What do you think?

Thanks!
-- 
-José

Re: Tagged fields constraint to start from tagId: 0

[David Jacot]

Hi Kamal,

It is fine to relax the constraint in StructSpec for your internal tags.
For instance, you could allow your internal tags to start from id 1. It
is very unlikely that we will ever have tags as part of the official
protocol reaching that.

Best,
David

On Mon, Feb 3, 2025 at 6:36 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Bumping up on this thread.
>
> In a forked Kafka repo, if we want to introduce a new tagged field. Which
> approach should be taken?
> Is it fine to relax the constraint in StructSpec
> <
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> >
> to
> start with a very large tag-id like 1 for custom
> tagged fields to not clash with future tagged fields being introduced
> upstream? TIA.
>
> Thanks,
> Kamal
>
> On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > In KIP-482
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions
> >,
> > we have introduced support for optional tagged fields when the
> > API-message supports "flexibleVersions" feature. One of the motivations
> of
> > the KIP is to support attaching extra information to a message without
> > requiring a version bump for everyone who has to read that message.
> >
> > In StructSpec
> > <
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> >,
> > there is a constraint to start the tagId of the tagged field from 0 and
> > should be contiguous. Once the user introduces a custom taggedField that
> > starts from 0, there is a risk of protocol divergence between the server
> > and client when the upstream/Apache introduces a new taggedField in
> future
> > that starts from 0.
> >
> > Once a tag-id gets introduced, it cannot be used for anything else. We
> > want to propagate a custom trace-id in the API-message between
> > server/client and want to use a large tagId like 10001 (2 bytes varint)
> to
> > avoid conflict with the future API message specification.
> >
> > We would like to understand the reason behind the tagId constraint to
> > start from 0 and contiguous. One use-case of having this constraint
> ensures
> > that the tag-id starts from 0 and won't be random. If it is random, then
> > the user won't be sure in defining the unique tagged-ids for their
> > user-defined fields.
> >
> > I’d appreciate it if you could take a look and share your thoughts or
> > feedback.
> >
> > Thanks,
> > Kamal
> >
> >
> >
> >
>

Re: [DISCUSS] KIP-1126: Serialize changes to Kafka with a build queue

[Ismael Juma]

This looks good to me.

My only comment is that merge queues have a weird limitation: you cannot
edit the commit message (unlike the traditional flow). I worry that this
will result in lower quality commit messages. Is there a way to configure
where the commit message comes from (PR title/description versus
aggregating all the commit messages in the PR)?

Ismael

On Mon, Feb 3, 2025, 8:02 AM David Arthur  wrote:

> Happy Monday, all!
>
> I'd like to close out this discussion at some point this week, so I wanted
> to bump this up once before moving to a vote.
>
> Did anyone have other feedback or concerns about moving towards the GitHub
> Merge Queue?
>
> Thanks!
> David A
>
> On Thu, Jan 30, 2025 at 11:11 AM David Arthur  wrote:
>
> > Uchechi,
> >
> > While the build queue can run the builds concurrently, each build is
> > cumulative.
> >
> > For example, A, B, and C are added to the queue and A causes a problem
> for
> > C.
> >
> > Merge Queue: A, B, C
> >
> > Suppose we have concurrent builds and batching turned on. We might end up
> > with two builds for these three PRs:
> >
> > Build 1: trunk + A
> > Build 2: trunk + A + B + C
> >
> > Each build is going to be cumulative. The point of the concurrency is so
> > we don't have to wait for an in-progress build to finish before we start
> > validating new PRs from the queue. In this case, build 2 would fail and
> PRs
> > B and C would get kicked back to the authors.
> >
> > As you can see, the batching and concurrency can complicate matters a bit
> > which is why I suggest we start with no batching or concurrency.
> >
> > Hope this helps!
> > David A
> >
> > On Wed, Jan 29, 2025 at 2:34 AM Ukpa Uchechi 
> > wrote:
> >
> >> Hi David,
> >>
> >> Thanks for this and for your work improving the Kafka CI.
> >>
> >> I have a question:
> >>
> >> A problem was mentioned about two PRs acting against each other when
> >> there’s no delay and being merged concurrently. Wouldn’t this still
> happen
> >> in the queue with concurrent merging as well?
> >> Best Regards,
> >> Uchechi.
> >>
> >> On Fri, 24 Jan 2025 at 03:35, David Arthur  wrote:
> >>
> >> > Greetings, Kafka community!
> >> >
> >> > At long last, the GitHub Merge Queue is upon us. This is a feature
> that
> >> > many of us have wanted for quite a while. After many months of
> >> discussion,
> >> > the excellent ASF Infra team has delivered!
> >> >
> >> > Since this is quite a significant change, I have written up the
> details
> >> as
> >> > a KIP.
> >> >
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1126%3A+Serialize+changes+to+Kafka+with+build+queue
> >> >
> >> > Please let me know what you think.
> >> >
> >> > --
> >> > David Arthur
> >> >
> >>
> >
> >
> > --
> > David Arthur
> >
>
>
> --
> David Arthur
>

[jira] [Resolved] (KAFKA-18725) Fix test_bounce.py sys test

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-18725.

Resolution: Won't Fix

duplicate to https://github.com/apache/kafka/pull/18798

> Fix test_bounce.py sys test 
> 
>
> Key: KAFKA-18725
> URL: https://issues.apache.org/jira/browse/KAFKA-18725
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Major
>
> we should just remove the zk test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)