Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Divij Vaidya]

> Is the new method enough for doing size-based retention?

Yes. You are right in assuming that this API only provides the Remote
storage size (for current epoch chain). We would use this API for size
based retention along with a value of localOnlyLogSegmentSize which is
computed as Log.sizeInBytes(logSegments.filter(_.baseOffset >
highestOffsetWithRemoteIndex)). Hence, (total_log_size =
remoteLogSizeBytes + log.localOnlyLogSegmentSize). I have updated the KIP
with this information. You can also check an example implementation at
https://github.com/satishd/kafka/blob/2.8.x-tiered-storage/core/src/main/scala/kafka/log/Log.scala#L2077


> Do you imagine all accesses to remote metadata will be across the network
or will there be some local in-memory cache?

I would expect a disk-less implementation to maintain a finite in-memory
cache for segment metadata to optimize the number of network calls made to
fetch the data. In future, we can think about bringing this finite size
cache into RLM itself but that's probably a conversation for a different
KIP. There are many other things we would like to do to optimize the Tiered
storage interface such as introducing a circular buffer / streaming
interface from RSM (so that we don't have to wait to fetch the entire
segment before starting to send records to the consumer), caching the
segments fetched from RSM locally (I would assume all RSM plugin
implementations to do this, might as well add it to RLM) etc.

--
Divij Vaidya



On Mon, Dec 12, 2022 at 7:35 PM Jun Rao  wrote:

> Hi, Divij,
>
> Thanks for the reply.
>
> Is the new method enough for doing size-based retention? It gives the total
> size of the remote segments, but it seems that we still don't know the
> exact total size for a log since there could be overlapping segments
> between the remote and the local segments.
>
> You mentioned a disk-less implementation. Do you imagine all accesses to
> remote metadata will be across the network or will there be some local
> in-memory cache?
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Dec 7, 2022 at 3:10 AM Divij Vaidya 
> wrote:
>
> > The method is needed for RLMM implementations which fetch the information
> > over the network and not for the disk based implementations (such as the
> > default topic based RLMM).
> >
> > I would argue that adding this API makes the interface more generic than
> > what it is today. This is because, with the current APIs an implementor
> is
> > restricted to use disk based RLMM solutions only (i.e. the default
> > solution) whereas if we add this new API, we unblock usage of network
> based
> > RLMM implementations such as databases.
> >
> >
> >
> > On Wed 30. Nov 2022 at 20:40, Jun Rao  wrote:
> >
> > > Hi, Divij,
> > >
> > > Thanks for the reply.
> > >
> > > Point#2. My high level question is that is the new method needed for
> > every
> > > implementation of remote storage or just for a specific implementation.
> > The
> > > issues that you pointed out exist for the default implementation of
> RLMM
> > as
> > > well and so far, the default implementation hasn't found a need for a
> > > similar new method. For public interface, ideally we want to make it
> more
> > > general.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Nov 21, 2022 at 7:11 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Thank you Jun and Alex for your comments.
> > > >
> > > > Point#1: You are right Jun. As Alex mentioned, the "derived metadata"
> > can
> > > > increase the size of cached metadata by a factor of 10 but it should
> be
> > > ok
> > > > to cache just the actual metadata. My point about size being a
> > limitation
> > > > for using cache is not valid anymore.
> > > >
> > > > Point#2: For a new replica, it would still have to fetch the metadata
> > > over
> > > > the network to initiate the warm up of the cache and hence, increase
> > the
> > > > start time of the archival process. Please also note the
> repercussions
> > of
> > > > the warm up scan that Alex mentioned in this thread as part of
> #102.2.
> > > >
> > > > 100#: Agreed Alex. Thanks for clarifying that. My point about size
> > being
> > > a
> > > > limitation for using cache is not valid anymore.
> > > >
> > > > 101#: Alex, if I understand correctly, you are suggesting to cache
> the
> > > > total size at the leader and update it on archival. This wouldn't
> work
> > > for
> > > > cases when the leader restarts where we would have to make a full
> scan
> > > > to update the total size entry on startup. We expect users to store
> > data
> > > > over longer duration in remote storage which increases the likelihood
> > of
> > > > leader restarts / failovers.
> > > >
> > > > 102#.1: I don't think that the current design accommodates the fact
> > that
> > > > data corruption could happen at the RLMM plugin (we don't have
> checksum
> > > as
> > > > a field in metadata as part of KIP405). If data corruption occurs, w/
> > or
> > > > w/o the cache, it would be a different problem to solve. I would l

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.3 #136

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #9

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14469) Fix inconsistency in config name between code and KIP-405

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-14469:


 Summary: Fix inconsistency in config name between code and KIP-405 
 Key: KAFKA-14469
 URL: https://issues.apache.org/jira/browse/KAFKA-14469
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Divij Vaidya
Assignee: Divij Vaidya
 Fix For: 3.5.0


We have 3 tiered storage configuration keys that are not adhering to KIP-405 
specification. KIP-405 has ".log" in the keys.

public static final String REMOTE_LOG_STORAGE_ENABLE_CONFIG = 
"remote.storage.enable";
public static final String LOCAL_LOG_RETENTION_MS_CONFIG = "local.retention.ms";
public static final String LOCAL_LOG_RETENTION_BYTES_CONFIG = 
"local.retention.bytes";

Reference:
KIP-405: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Configs.1]
Kafka 3.3: 
[https://github.com/apache/kafka/blob/3.3/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L78-L90]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1433

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1434

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao]

Hi, Justine,

Thanks for the KIP. A couple of comments.

70. Currently, the producer epoch is an int. I am not sure if it's enough
to accommodate all transactions in the lifetime of a producer. Should we
change that to a long or add a new long field like txnId?

71. "it will write the prepare commit message with a bumped epoch and send
WriteTxnMarkerRequests with the bumped epoch." Hmm, the epoch is associated
with the current txn right? So, it seems weird to write a commit message
with a bumped epoch. Should we only bump up the epoch in EndTxnResponse and
rename the field to sth like nextProducerEpoch?

Thanks,

Jun



On Mon, Dec 12, 2022 at 8:54 PM Matthias J. Sax  wrote:

> Thanks for the background.
>
> 20/30: SGTM. My proposal was only focusing to avoid dangling
> transactions if records are added without registered partition. -- Maybe
> you can add a few more details to the KIP about this scenario for better
> documentation purpose?
>
> 40: I think you hit a fair point about race conditions or client bugs
> (incorrectly not bumping the epoch). The complexity/confusion for using
> the bumped epoch I see, is mainly for internal debugging, ie, inspecting
> log segment dumps -- it seems harder to reason about the system for us
> humans. But if we get better guarantees, it would be worth to use the
> bumped epoch.
>
> 60: as I mentioned already, I don't know the broker internals to provide
> more input. So if nobody else chimes in, we should just move forward
> with your proposal.
>
>
> -Matthias
>
>
> On 12/6/22 4:22 PM, Justine Olshan wrote:
> > Hi all,
> > After Artem's questions about error behavior, I've re-evaluated the
> > unknown producer ID exception and had some discussions offline.
> >
> > I think generally it makes sense to simplify error handling in cases like
> > this and the UNKNOWN_PRODUCER_ID error has a pretty long and complicated
> > history. Because of this, I propose adding a new error code
> ABORTABLE_ERROR
> > that when encountered by new clients (gated by the produce request
> version)
> > will simply abort the transaction. This allows the server to have some
> say
> > in whether the client aborts and makes handling much simpler. In the
> > future, we can also use this error in other situations where we want to
> > abort the transactions. We can even use on other apis.
> >
> > I've added this to the KIP. Let me know if there are any questions or
> > issues.
> >
> > Justine
> >
> > On Fri, Dec 2, 2022 at 10:22 AM Justine Olshan 
> wrote:
> >
> >> Hey Matthias,
> >>
> >>
> >> 20/30 — Maybe I also didn't express myself clearly. For older clients we
> >> don't have a way to distinguish between a previous and the current
> >> transaction since we don't have the epoch bump. This means that a late
> >> message from the previous transaction may be added to the new one. With
> >> older clients — we can't guarantee this won't happen if we already sent
> the
> >> addPartitionsToTxn call (why we make changes for the newer client) but
> we
> >> can at least gate some by ensuring that the partition has been added to
> the
> >> transaction. The rationale here is that there are likely LESS late
> arrivals
> >> as time goes on, so hopefully most late arrivals will come in BEFORE the
> >> addPartitionsToTxn call. Those that arrive before will be properly gated
> >> with the describeTransactions approach.
> >>
> >> If we take the approach you suggested, ANY late arrival from a previous
> >> transaction will be added. And we don't want that. I also don't see any
> >> benefit in sending addPartitionsToTxn over the describeTxns call. They
> will
> >> both be one extra RPC to the Txn coordinator.
> >>
> >>
> >> To be clear — newer clients will use addPartitionsToTxn instead of the
> >> DescribeTxns.
> >>
> >>
> >> 40)
> >> My concern is that if we have some delay in the client to bump the
> epoch,
> >> it could continue to send epoch 73 and those records would not be
> fenced.
> >> Perhaps this is not an issue if we don't allow the next produce to go
> >> through before the EndTxn request returns. I'm also thinking about
> cases of
> >> failure. I will need to think on this a bit.
> >>
> >> I wasn't sure if it was that confusing. But if we think it is, we can
> >> investigate other ways.
> >>
> >>
> >> 60)
> >>
> >> I'm not sure these are the same purgatories since one is a produce
> >> purgatory (I was planning on using a callback rather than purgatory) and
> >> the other is simply a request to append to the log. Not sure we have any
> >> structure here for ordering, but my understanding is that the broker
> could
> >> handle the write request before it hears back from the Txn Coordinator.
> >>
> >> Let me know if I misunderstood something or something was unclear.
> >>
> >> Justine
> >>
> >> On Thu, Dec 1, 2022 at 12:15 PM Matthias J. Sax 
> wrote:
> >>
> >>> Thanks for the details Justine!
> >>>
>  20)
> 
>  The client side change for 2 is removing the addPartitions to
> >>> transaction
>

Re: [DISCUSS] Apache Kafka 3.3.2

[José Armando García Sancio]

Hey all,

Here are the system test results for the 3.3 branch:
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1670984851--apache--3.3--22af3f29ce/2022-12-13--001./2022-12-13--001./report.html

The commit for that run is 22af3f29ced9e969c254004f592c06c8a45b84cc.
It is also encoded in the URL above.

Thanks
-- 
-José

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan]

Matthias — thanks again for taking time to look a this. You said:

> My proposal was only focusing to avoid dangling

transactions if records are added without registered partition. -- Maybe

you can add a few more details to the KIP about this scenario for better

documentation purpose?


I'm not sure I understand what you mean here. The motivation section
describes two scenarios about how the record can be added without a
registered partition:


> This can happen when a message gets stuck or delayed due to networking
issues or a network partition, the transaction aborts, and then the delayed
message finally comes in.


> Another way hanging transactions can occur is that a client is buggy and
may somehow try to write to a partition before it adds the partition to the
transaction.



For the first example of this would it be helpful to say that this message
comes in after the abort, but before the partition is added to the next
transaction so it becomes "hanging." Perhaps the next sentence describing
the message becoming part of the next transaction (a different case) was
not properly differentiated.



Jun — thanks for reading the KIP.

70. The int typing was a concern. Currently we have a mechanism in place to
fence the final epoch when the epoch is about to overflow and assign a new
producer ID with epoch 0. Of course, this is a bit tricky when it comes to
the response back to the client.
Making this a long could be another option, but I wonder are there any
implications on changing this field if the epoch is persisted to disk? I'd
need to check the usages.

71.This was something Matthias asked about as well. I was considering a
possible edge case where a produce request from a new transaction somehow
gets sent right after the marker is written, but before the producer is
alerted of the newly bumped epoch. In this case, we may include this record
when we don't want to. I suppose we could try to do something client side
to bump the epoch after sending an endTxn as well in this scenario — but I
wonder how it would work when the server is aborting based on a server-side
error. I could also be missing something and this scenario is actually not
possible.

Thanks again to everyone reading and commenting. Let me know about any
further questions or comments.

Justine

On Wed, Dec 14, 2022 at 9:41 AM Jun Rao  wrote:

> Hi, Justine,
>
> Thanks for the KIP. A couple of comments.
>
> 70. Currently, the producer epoch is an int. I am not sure if it's enough
> to accommodate all transactions in the lifetime of a producer. Should we
> change that to a long or add a new long field like txnId?
>
> 71. "it will write the prepare commit message with a bumped epoch and send
> WriteTxnMarkerRequests with the bumped epoch." Hmm, the epoch is associated
> with the current txn right? So, it seems weird to write a commit message
> with a bumped epoch. Should we only bump up the epoch in EndTxnResponse and
> rename the field to sth like nextProducerEpoch?
>
> Thanks,
>
> Jun
>
>
>
> On Mon, Dec 12, 2022 at 8:54 PM Matthias J. Sax  wrote:
>
> > Thanks for the background.
> >
> > 20/30: SGTM. My proposal was only focusing to avoid dangling
> > transactions if records are added without registered partition. -- Maybe
> > you can add a few more details to the KIP about this scenario for better
> > documentation purpose?
> >
> > 40: I think you hit a fair point about race conditions or client bugs
> > (incorrectly not bumping the epoch). The complexity/confusion for using
> > the bumped epoch I see, is mainly for internal debugging, ie, inspecting
> > log segment dumps -- it seems harder to reason about the system for us
> > humans. But if we get better guarantees, it would be worth to use the
> > bumped epoch.
> >
> > 60: as I mentioned already, I don't know the broker internals to provide
> > more input. So if nobody else chimes in, we should just move forward
> > with your proposal.
> >
> >
> > -Matthias
> >
> >
> > On 12/6/22 4:22 PM, Justine Olshan wrote:
> > > Hi all,
> > > After Artem's questions about error behavior, I've re-evaluated the
> > > unknown producer ID exception and had some discussions offline.
> > >
> > > I think generally it makes sense to simplify error handling in cases
> like
> > > this and the UNKNOWN_PRODUCER_ID error has a pretty long and
> complicated
> > > history. Because of this, I propose adding a new error code
> > ABORTABLE_ERROR
> > > that when encountered by new clients (gated by the produce request
> > version)
> > > will simply abort the transaction. This allows the server to have some
> > say
> > > in whether the client aborts and makes handling much simpler. In the
> > > future, we can also use this error in other situations where we want to
> > > abort the transactions. We can even use on other apis.
> > >
> > > I've added this to the KIP. Let me know if there are any questions or
> > > issues.
> > >
> > > Justine
> > >
> > > On Fri, Dec 2, 2022 at 10:22 AM Justine Olshan 
> > wrote:
> > >

[GitHub] [kafka-site] mjsax opened a new pull request, #465: MINOR: update Streams upgrade guide for 3.1 release

[GitBox]

mjsax opened a new pull request, #465:
URL: https://github.com/apache/kafka-site/pull/465

   Follow up PR to https://github.com/apache/kafka/pull/12926


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mjsax merged pull request #465: MINOR: update Streams upgrade guide for 3.1 release

[GitBox]

mjsax merged PR #465:
URL: https://github.com/apache/kafka-site/pull/465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mjsax opened a new pull request, #466: MINOR: add docs table entries for new metrics

[GitBox]

mjsax opened a new pull request, #466:
URL: https://github.com/apache/kafka-site/pull/466

   Follow up PR to https://github.com/apache/kafka/pull/12934


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mjsax merged pull request #466: MINOR: add docs table entries for new metrics

[GitBox]

mjsax merged PR #466:
URL: https://github.com/apache/kafka-site/pull/466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-890 Server Side Defense

[Matthias J. Sax]

What I mean is the following:

For both scenarios, late message or missing addPartitionToTxnRequest, a 
record r is written to partition X, but X is not registered at the 
TX-coordinator. Now there are two cases:


(1) A follow up transaction writes more data to the same partition X, 
and r becomes part of the follow up transaction. This is an error 
obviously, but we don't get a hanging transaction.


(2) X is not part of any follow up transaction and thus X starts to 
block consumer reading data.


If we let the partition leader send the addPartitionToTxnRequest to the 
TX-coordinator, scenario (2) always turns into scenario (1) -- at least, 
if there is one more transaction for this producer (what I think we can 
assume). Even if the follow up transaction doesn't write data to X, X 
still becomes part of the TX and X won't hang and won't block consumers 
any longer.


We still end up with not fixing (1) though... Your proposal seems to 
address case (1) in addition to case (2), at least for most cases. There 
is still the race condition (that we cannot fix without the epoch bump) 
that r comes in _very_ late, and the follow up transaction would have 
written more data to X already, and thus X is indeed already registered 
and r would just be added successfully. Of course, the race condition 
window is much smaller, so your proposal is much better than what I had 
in mind.



-Matthias

On 12/14/22 10:43 AM, Justine Olshan wrote:

Matthias — thanks again for taking time to look a this. You said:


My proposal was only focusing to avoid dangling


transactions if records are added without registered partition. -- Maybe

you can add a few more details to the KIP about this scenario for better

documentation purpose?


I'm not sure I understand what you mean here. The motivation section
describes two scenarios about how the record can be added without a
registered partition:



This can happen when a message gets stuck or delayed due to networking

issues or a network partition, the transaction aborts, and then the delayed
message finally comes in.



Another way hanging transactions can occur is that a client is buggy and

may somehow try to write to a partition before it adds the partition to the
transaction.



For the first example of this would it be helpful to say that this message
comes in after the abort, but before the partition is added to the next
transaction so it becomes "hanging." Perhaps the next sentence describing
the message becoming part of the next transaction (a different case) was
not properly differentiated.



Jun — thanks for reading the KIP.

70. The int typing was a concern. Currently we have a mechanism in place to
fence the final epoch when the epoch is about to overflow and assign a new
producer ID with epoch 0. Of course, this is a bit tricky when it comes to
the response back to the client.
Making this a long could be another option, but I wonder are there any
implications on changing this field if the epoch is persisted to disk? I'd
need to check the usages.

71.This was something Matthias asked about as well. I was considering a
possible edge case where a produce request from a new transaction somehow
gets sent right after the marker is written, but before the producer is
alerted of the newly bumped epoch. In this case, we may include this record
when we don't want to. I suppose we could try to do something client side
to bump the epoch after sending an endTxn as well in this scenario — but I
wonder how it would work when the server is aborting based on a server-side
error. I could also be missing something and this scenario is actually not
possible.

Thanks again to everyone reading and commenting. Let me know about any
further questions or comments.

Justine

On Wed, Dec 14, 2022 at 9:41 AM Jun Rao  wrote:


Hi, Justine,

Thanks for the KIP. A couple of comments.

70. Currently, the producer epoch is an int. I am not sure if it's enough
to accommodate all transactions in the lifetime of a producer. Should we
change that to a long or add a new long field like txnId?

71. "it will write the prepare commit message with a bumped epoch and send
WriteTxnMarkerRequests with the bumped epoch." Hmm, the epoch is associated
with the current txn right? So, it seems weird to write a commit message
with a bumped epoch. Should we only bump up the epoch in EndTxnResponse and
rename the field to sth like nextProducerEpoch?

Thanks,

Jun



On Mon, Dec 12, 2022 at 8:54 PM Matthias J. Sax  wrote:


Thanks for the background.

20/30: SGTM. My proposal was only focusing to avoid dangling
transactions if records are added without registered partition. -- Maybe
you can add a few more details to the KIP about this scenario for better
documentation purpose?

40: I think you hit a fair point about race conditions or client bugs
(incorrectly not bumping the epoch). The complexity/confusion for using
the bumped epoch I see, is mainly for internal debugging, ie, inspecting
log segment dumps -- it s

Re: [DISCUSS] KIP-889 Versioned State Stores

[Matthias J. Sax]

Thanks for clarifying about the null-question. SGTM.

On 12/13/22 3:06 PM, Victoria Xia wrote:

Hi Matthias,

Thanks for chiming in! Barring objections from anyone on this thread, I
will start the vote for this KIP on Thursday. That should be enough time to
incorporate any lingering minor changes.


I slightly prefer to add `VersionedRecord` interface (also

like the name). I agree that it's low overhead and providing a clean
path forward for future changes seems worth it to me.

OK, that makes two of us. I updated the KIP just now to formally include
VersionedRecord as the new return type from the various
VersionedKeyValueStore methods.


if we introduce `VersionedRecord`, I think we can keep the not-null

requirement for `ValueAndTimestamp`

Not quite. VersionedRecord is only used as a return type from read methods,
which is why VersionedRecord is able to enforce that its value is never
null. If the value being returned would have been null, then we return a
null VersionedRecord instead, rather than non-null VersionedRecord with
null value. So, there's no use case for a VersionedRecord with null value.

In contrast, even though ValueAndTimestamp is not anywhere in the public
VersionedKeyValueStore interface, ValueAndTimestamp still needs to be used
internally when representing a versioned key-value store as a
TimestampedKeyValueStore, since TimestampedKeyValueStore is used everywhere
throughout the internals of the codebase. In order to represent a versioned
key-value store as a TimestampedKeyValueStore, we have to support `put(K
key, ValueAndTimestamp value)`, which means ValueAndTimestamp needs to
support null value (with timestamp). Otherwise we cannot put a tombstone
into a versioned key-value store when using the internal
TimestampedKeyValueStore representation.

It's very much an implementation detail that ValueAndTimestamp needs to be
relaxed to allow null values. I think this is a minor enough change that is
still preferable to the alternatives (refactoring the processors to not
require TimestampedKeyValueStore, or introducing a separate workaround
`put()` method on the TimestampedKeyValueStore representation of versioned
key-value stores), so I have left it in as part of the KIP.

Best,
Victoria

On Mon, Dec 12, 2022 at 8:42 PM Matthias J. Sax  wrote:


Thanks Victoria.

I did not re-read the KIP in full on the wiki but only your email.

Points (1)-(8) SGTM.

About (9): I slightly prefer to add `VersionedRecord` interface (also
like the name). I agree that it's low overhead and providing a clean
path forward for future changes seems worth it to me. Btw: if we
introduce `VersionedRecord`, I think we can keep the not-null
requirement for `ValueAndTimestamp` what seems a small side benefit.
(Btw: your code snippet in the KIP shows what `VersionedRecord` would
have a non-null requirement for the value, but I think it would need to
allow null as value?)


-Matthias

On 12/7/22 5:23 PM, Victoria Xia wrote:

Thanks for the discussion, Bruno, Sagar, and Matthias!

It seems we've reached consensus on almost all of the discussion points.
I've updated the KIP with the following:
1) renamed "timestampTo" in `get(key, timestampTo)` to "asOfTimestamp" to
clarify that this timestamp bound is inclusive, per the SQL guideline

that

"AS OF " queries are inclusive. In the future, if we want to
introduce a timestamp range query, we can use `get(key, timestampFrom,
timestampTo)` and specify that timestampTo is exclusive in this method,
while avoiding confusing with the inclusive asOfTimestamp parameter in

the

other method, given that the names are different.
2) added a description of "history retention" semantics into the
VersionedKeyValueStore interface Javadoc, and updated the Javadoc for
`get(key, asOfTimestamp)` to mention explicitly that a null result is
returned if the provided timestamp bound is not within history retention.
3) added a `delete(key, timestamp)` method (with return type
`ValueAndTimestamp`) to the VersionedKeyValueStore interface.
4) updated the Javadoc for `segmentInterval` to clarify that the only
reason a user might be interested in this parameter is performance.

Other points we discussed which did not result in updates include:
5) whether to automatically update the `min.compaction.lag.ms` config on
changelog topics when history retention is changed -- there's support for
this but let's not bundle it with this KIP. We can have a separate KIP to
change this behavior for the existing windowed changelog topics, in
addition to versioned changelog topics.
6) should we expose segmentInterval in this KIP -- let's go ahead and
expose it now since we'll almost certainly expose it (in this same

manner)

in a follow-on KIP anyway, and so that poor performance for user

workloads

is less likely to be a barrier for users getting started with this

feature.

I updated the Javadoc for this parameter to clarify why the Javadoc
mentions performance despite Javadocs typically not doing so.
7) `get(tim

[VOTE] KIP-887 - Add ConfigProvider to make use of environment variables

[Roman Schmitz]

Hi all,

Thank you for the feedback so far.
The KIP is rather straightforward and I'd like to start a vote on it.
Please have a look at the KIP: https://cwiki.apache.org/confluence/x/15jGDQ

Thanks,
Roman

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1435

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 435669 lines...]
[2022-12-14T20:55:06.024Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2022-12-14T20:55:06.024Z] > Task :metadata:compileTestJava UP-TO-DATE
[2022-12-14T20:55:06.024Z] > Task :metadata:testClasses UP-TO-DATE
[2022-12-14T20:55:06.024Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2022-12-14T20:55:06.024Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2022-12-14T20:55:06.024Z] 
[2022-12-14T20:55:06.024Z] > Task :streams:processMessages
[2022-12-14T20:55:06.024Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2022-12-14T20:55:06.024Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.6/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-12-14T20:55:06.024Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2022-12-14T20:55:06.024Z] 
[2022-12-14T20:55:06.024Z] > Task :streams:compileJava UP-TO-DATE
[2022-12-14T20:55:06.024Z] > Task :streams:classes UP-TO-DATE
[2022-12-14T20:55:06.024Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-12-14T20:55:07.231Z] > Task :streams:copyDependantLibs
[2022-12-14T20:55:07.231Z] > Task :streams:jar UP-TO-DATE
[2022-12-14T20:55:07.231Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-12-14T20:55:09.584Z] 
[2022-12-14T20:55:09.584Z] > Task :clients:javadoc
[2022-12-14T20:55:09.584Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginCallbackHandler.java:151:
 warning - Tag @link: reference not found: 
[2022-12-14T20:55:09.584Z] 
[2022-12-14T20:55:09.584Z] > Task :connect:api:javadoc
[2022-12-14T20:55:09.584Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task :connect:api:jar UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-12-14T20:55:09.584Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task :connect:json:jar UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-12-14T20:55:09.584Z] > Task :connect:api:javadocJar
[2022-12-14T20:55:09.584Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-12-14T20:55:09.584Z] > Task :connect:json:publishToMavenLocal
[2022-12-14T20:55:09.584Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-12-14T20:55:09.584Z] > Task :connect:api:testJar
[2022-12-14T20:55:10.385Z] > Task :connect:api:testSrcJar
[2022-12-14T20:55:10.385Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-12-14T20:55:10.385Z] > Task :connect:api:publishToMavenLocal
[2022-12-14T20:55:10.385Z] 
[2022-12-14T20:55:10.385Z] > Task :streams:javadoc
[2022-12-14T20:55:10.385Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/processor/StreamPartitioner.java:54:
 warning - Tag @link: reference not found: 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-12-14T20:55:11.401Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/src/main/java/org/apac

[jira] [Created] (KAFKA-14470) Move log layer to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14470:
---

 Summary: Move log layer to storage module
 Key: KAFKA-14470
 URL: https://issues.apache.org/jira/browse/KAFKA-14470
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma


We introduced the `storage` module as part of KIP-405, but the existing log 
layer remains in the `core` module. Moving the log layer to the `storage` 
module would be another step towards improved modularity and build times 
(similar to `metadata`, `raft` and `group-coordinator`).

We should do this in an incremental manner to make the code review process 
easier. I will create separate tasks, each one mapping to one pull request.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14472) Move TransactionIndex and related to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14472:
---

 Summary: Move TransactionIndex and related to storage module
 Key: KAFKA-14472
 URL: https://issues.apache.org/jira/browse/KAFKA-14472
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14471) Move IndexEntry and related to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14471:
---

 Summary: Move IndexEntry and related to storage module
 Key: KAFKA-14471
 URL: https://issues.apache.org/jira/browse/KAFKA-14471
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14473) Move AbstractIndex to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14473:
---

 Summary: Move AbstractIndex to storage module
 Key: KAFKA-14473
 URL: https://issues.apache.org/jira/browse/KAFKA-14473
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14474) Move OffsetIndex to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14474:
---

 Summary: Move OffsetIndex to storage module
 Key: KAFKA-14474
 URL: https://issues.apache.org/jira/browse/KAFKA-14474
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14475) Move TimestampIndex/LazyIndex to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14475:
---

 Summary: Move TimestampIndex/LazyIndex to storage module
 Key: KAFKA-14475
 URL: https://issues.apache.org/jira/browse/KAFKA-14475
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14476) Move OffsetMap to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14476:
---

 Summary: Move OffsetMap to storage module
 Key: KAFKA-14476
 URL: https://issues.apache.org/jira/browse/KAFKA-14476
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.3.2

[Chris Egerton]

Hi José,

Thanks for your help! I was able to locally rerun two of the three failed
tests and get passing results; the reports for them can be found at:

https://home.apache.org/~cegerton/system_tests/2022-12-14--015/report.html
(Connect REST test)
https://home.apache.org/~cegerton/system_tests/2022-12-14--016/report.html
(Consume bench test)

I still haven't been able to get a green run of the upgrade test, although
the cause of failure varied across runs. Is this test known to be
particularly flaky?

Cheers,

Chris

On Wed, Dec 14, 2022 at 1:36 PM José Armando García Sancio
 wrote:

> Hey all,
>
> Here are the system test results for the 3.3 branch:
>
> https://eu01.z.antigena.com/l/Sx1FolWhf8d4T5FvVLnTK~tC4vADTaIz7lWjWml4UO3e6C-ymjdh8HOo-t~kEiLXcZLMkLT029Qe93L0YJU0AGLTWA-GpIMGZnlNZl~0ddmKP9Zwp-WtF4T0XBTGjc4FsrU4LV3x8fkIOgl9gH2qHb9jjh9XeSnCZ0S9GXdYwPwzoAIBNzC~5ZtASMc4T9DQBZvmrYF52BkZ9yC0ff1xl7YIzCZuYLoPt5KbWG4Ak8YQnS-eArBO9UgzanUOYJ585J7X-q7-DmREnRp9Rh
>
> The commit for that run is 22af3f29ced9e969c254004f592c06c8a45b84cc.
> It is also encoded in the URL above.
>
> Thanks
> --
> -José
>
>

[jira] [Created] (KAFKA-14477) Move LogValidator to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14477:
---

 Summary: Move LogValidator to storage module
 Key: KAFKA-14477
 URL: https://issues.apache.org/jira/browse/KAFKA-14477
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14479) Move CleanerConfig to storage

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14479:
---

 Summary: Move CleanerConfig to storage
 Key: KAFKA-14479
 URL: https://issues.apache.org/jira/browse/KAFKA-14479
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14478) Move LogConfig to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14478:
---

 Summary: Move LogConfig to storage module
 Key: KAFKA-14478
 URL: https://issues.apache.org/jira/browse/KAFKA-14478
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14480) Move ProducerStateManager to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14480:
---

 Summary: Move ProducerStateManager to storage module
 Key: KAFKA-14480
 URL: https://issues.apache.org/jira/browse/KAFKA-14480
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14481) Move LogSegment/LogSegments to storage

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14481:
---

 Summary: Move LogSegment/LogSegments to storage
 Key: KAFKA-14481
 URL: https://issues.apache.org/jira/browse/KAFKA-14481
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14482) Move LogLoader to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14482:
---

 Summary: Move LogLoader to storage module
 Key: KAFKA-14482
 URL: https://issues.apache.org/jira/browse/KAFKA-14482
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14483) Move LocalLog to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14483:
---

 Summary: Move LocalLog to storage module
 Key: KAFKA-14483
 URL: https://issues.apache.org/jira/browse/KAFKA-14483
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14484) Move UnifiedLog to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14484:
---

 Summary: Move UnifiedLog to storage module
 Key: KAFKA-14484
 URL: https://issues.apache.org/jira/browse/KAFKA-14484
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14485) Move LogCleaner to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14485:
---

 Summary: Move LogCleaner to storage module
 Key: KAFKA-14485
 URL: https://issues.apache.org/jira/browse/KAFKA-14485
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14486) Move LogCleanerManager to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14486:
---

 Summary: Move LogCleanerManager to storage module
 Key: KAFKA-14486
 URL: https://issues.apache.org/jira/browse/KAFKA-14486
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14488) Move log layer tests to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14488:
---

 Summary: Move log layer tests to storage module
 Key: KAFKA-14488
 URL: https://issues.apache.org/jira/browse/KAFKA-14488
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma


This should be split into multiple tasks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14487) Move LogManager to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14487:
---

 Summary: Move LogManager to storage module
 Key: KAFKA-14487
 URL: https://issues.apache.org/jira/browse/KAFKA-14487
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14489) Adjust visibility of classes moved to storage module

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14489:
---

 Summary: Adjust visibility of classes moved to storage module
 Key: KAFKA-14489
 URL: https://issues.apache.org/jira/browse/KAFKA-14489
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma


Once the log layer has been completely migrated to the storage module, we 
should adjust the visibility of classes that are only used within the log layer 
to be package private.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan]

Thanks Matthias, I think we are on the same page.
The concern I had about your solution with the old clients is that we can't
distinguish between a late message and a message intended for the new
transaction on old clients -- basically any late message can turn into case
1.
I chose to rely on adding partitions to transaction as the determining
factor of whether the record belonged to the transaction to rule out some
of these late message cases. Of course, in some cases we will still run
into case 1 unfortunately if the message is really late (also as you
mention in the last paragraph), but I believed that is ok if we avoid it in
most cases (best effort).

Let me know if you think some part of this is not clear on the original
KIP, and I can better address it there as well.

Thanks again for taking time to think through this with me,
Justine

On Wed, Dec 14, 2022 at 12:24 PM Matthias J. Sax  wrote:

> What I mean is the following:
>
> For both scenarios, late message or missing addPartitionToTxnRequest, a
> record r is written to partition X, but X is not registered at the
> TX-coordinator. Now there are two cases:
>
> (1) A follow up transaction writes more data to the same partition X,
> and r becomes part of the follow up transaction. This is an error
> obviously, but we don't get a hanging transaction.
>
> (2) X is not part of any follow up transaction and thus X starts to
> block consumer reading data.
>
> If we let the partition leader send the addPartitionToTxnRequest to the
> TX-coordinator, scenario (2) always turns into scenario (1) -- at least,
> if there is one more transaction for this producer (what I think we can
> assume). Even if the follow up transaction doesn't write data to X, X
> still becomes part of the TX and X won't hang and won't block consumers
> any longer.
>
> We still end up with not fixing (1) though... Your proposal seems to
> address case (1) in addition to case (2), at least for most cases. There
> is still the race condition (that we cannot fix without the epoch bump)
> that r comes in _very_ late, and the follow up transaction would have
> written more data to X already, and thus X is indeed already registered
> and r would just be added successfully. Of course, the race condition
> window is much smaller, so your proposal is much better than what I had
> in mind.
>
>
> -Matthias
>
> On 12/14/22 10:43 AM, Justine Olshan wrote:
> > Matthias — thanks again for taking time to look a this. You said:
> >
> >> My proposal was only focusing to avoid dangling
> >
> > transactions if records are added without registered partition. -- Maybe
> >
> > you can add a few more details to the KIP about this scenario for better
> >
> > documentation purpose?
> >
> >
> > I'm not sure I understand what you mean here. The motivation section
> > describes two scenarios about how the record can be added without a
> > registered partition:
> >
> >
> >> This can happen when a message gets stuck or delayed due to networking
> > issues or a network partition, the transaction aborts, and then the
> delayed
> > message finally comes in.
> >
> >
> >> Another way hanging transactions can occur is that a client is buggy and
> > may somehow try to write to a partition before it adds the partition to
> the
> > transaction.
> >
> >
> >
> > For the first example of this would it be helpful to say that this
> message
> > comes in after the abort, but before the partition is added to the next
> > transaction so it becomes "hanging." Perhaps the next sentence describing
> > the message becoming part of the next transaction (a different case) was
> > not properly differentiated.
> >
> >
> >
> > Jun — thanks for reading the KIP.
> >
> > 70. The int typing was a concern. Currently we have a mechanism in place
> to
> > fence the final epoch when the epoch is about to overflow and assign a
> new
> > producer ID with epoch 0. Of course, this is a bit tricky when it comes
> to
> > the response back to the client.
> > Making this a long could be another option, but I wonder are there any
> > implications on changing this field if the epoch is persisted to disk?
> I'd
> > need to check the usages.
> >
> > 71.This was something Matthias asked about as well. I was considering a
> > possible edge case where a produce request from a new transaction somehow
> > gets sent right after the marker is written, but before the producer is
> > alerted of the newly bumped epoch. In this case, we may include this
> record
> > when we don't want to. I suppose we could try to do something client side
> > to bump the epoch after sending an endTxn as well in this scenario — but
> I
> > wonder how it would work when the server is aborting based on a
> server-side
> > error. I could also be missing something and this scenario is actually
> not
> > possible.
> >
> > Thanks again to everyone reading and commenting. Let me know about any
> > further questions or comments.
> >
> > Justine
> >
> > On Wed, Dec 14, 2022 at 9:41 AM Jun Rao 
> wrot

[jira] [Created] (KAFKA-14490) Consider using UncheckdIOException instead of IOException in the log layer

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-14490:
---

 Summary: Consider using UncheckdIOException instead of IOException 
in the log layer
 Key: KAFKA-14490
 URL: https://issues.apache.org/jira/browse/KAFKA-14490
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma


IOException is a checked exception, which makes it difficult to use with 
lambdas. We should consider using UncheckdIOException instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #139

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 367453 lines...]
[2022-12-14T21:48:43.662Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2022-12-14T21:48:43.662Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-12-14T21:48:43.662Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2022-12-14T21:48:43.662Z] 
[2022-12-14T21:48:43.662Z] > Task :streams:compileJava UP-TO-DATE
[2022-12-14T21:48:43.662Z] > Task :streams:classes UP-TO-DATE
[2022-12-14T21:48:43.662Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-12-14T21:48:43.662Z] > Task :streams:copyDependantLibs
[2022-12-14T21:48:43.662Z] > Task :streams:jar UP-TO-DATE
[2022-12-14T21:48:44.687Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-12-14T21:48:47.763Z] > Task :connect:api:javadoc
[2022-12-14T21:48:47.763Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task :connect:api:jar UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-12-14T21:48:47.763Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task :connect:json:jar UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-12-14T21:48:47.763Z] > Task :connect:api:javadocJar
[2022-12-14T21:48:47.763Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-12-14T21:48:47.763Z] > Task :connect:api:testJar
[2022-12-14T21:48:47.763Z] > Task :connect:api:testSrcJar
[2022-12-14T21:48:47.763Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-12-14T21:48:47.763Z] > Task :connect:json:publishToMavenLocal
[2022-12-14T21:48:47.763Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-12-14T21:48:47.763Z] > Task :connect:api:publishToMavenLocal
[2022-12-14T21:48:52.451Z] 
[2022-12-14T21:48:52.451Z] > Task :clients:javadoc
[2022-12-14T21:48:52.452Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-12-14T21:48:52.452Z] 
[2022-12-14T21:48:52.452Z] > Task :streams:javadoc
[2022-12-14T21:48:52.452Z] > Task :streams:javadocJar
[2022-12-14T21:48:53.624Z] 
[2022-12-14T21:48:53.624Z] > Task :clients:javadoc
[2022-12-14T21:48:53.624Z] 1 warning
[2022-12-14T21:48:53.624Z] 
[2022-12-14T21:48:53.624Z] > Task :clients:javadocJar
[2022-12-14T21:48:54.666Z] 
[2022-12-14T21:48:54.666Z] > Task :clients:srcJar
[2022-12-14T21:48:54.666Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-12-14T21:48:54.666Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-12-14T21:48:54.666Z] 
[2022-12-14T21:48:54.666Z] > Task :clients:testJar
[2022-12-14T21:48:55.709Z] > Task :clients:testSrcJar
[2022-12-14T21:48:55.709Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-12-14T21:48:55.709Z] > Task :clients:publishToMavenLocal
[2022-12-14T21:49:14.900Z] > Task :core:compileScala
[2022-12-14T21:50:27.516Z] > Task :core:classes
[2022-12-14T21:50:27.516Z] > Task :core:compileTestJava NO-SOURCE
[2022-12-14T21:50:57.150Z] > Task :core:compileTestScala
[2022-12-14T21:51:58.488Z] > Task :core:testClasses
[2022-12-14T21:52:09.825Z] > Task :streams:compileTestJava
[2022-12-14T21:52:09.825Z] > Task :streams:testClasses
[2022-12-14T21:52:10.861Z] > Task :streams:testJar
[2022-12-14T21:52:10.861Z] > Task :streams:testSrcJar
[2022-12-14T21:52:10.861Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-12-14T21:52:10.861Z] > Task :streams:publishToMavenLocal
[2022-12-14T21:52:10.861Z] 
[2022-12-14T21:52:10.

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #95

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 225023 lines...]
[2022-12-14T22:14:42.229Z] > Task :connect:api:jar UP-TO-DATE
[2022-12-14T22:14:42.229Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-12-14T22:14:42.229Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-12-14T22:14:42.229Z] > Task :connect:json:jar UP-TO-DATE
[2022-12-14T22:14:42.229Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-12-14T22:14:42.229Z] > Task :connect:api:javadocJar
[2022-12-14T22:14:42.229Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-12-14T22:14:42.229Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-12-14T22:14:42.229Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-12-14T22:14:42.229Z] > Task :connect:json:publishToMavenLocal
[2022-12-14T22:14:42.229Z] > Task :connect:api:testJar
[2022-12-14T22:14:42.229Z] > Task :connect:api:testSrcJar
[2022-12-14T22:14:42.229Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-12-14T22:14:42.229Z] > Task :connect:api:publishToMavenLocal
[2022-12-14T22:14:45.327Z] 
[2022-12-14T22:14:45.327Z] > Task :streams:javadoc
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-12-14T22:14:45.327Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-12-14T22:14:45.327Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2022-12-14T22:14:46.392Z] 12 warnings
[2022-12-14T22:14:46.392Z] 
[2022-12-14T22:14:46.392Z] > Task :streams:javadocJar
[2022-12-14T22:14:46.392Z] > Task :streams:processTestResources UP-TO-DATE
[2022-12-14T22:14:46.392Z] 
[2022-12-14T22:14:46.392Z] > Task :clients:javadoc
[2022-12-14T22:14:46.392Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-12-14T22:14:47.457Z] 1 warning
[2022-12-14T22:14:47.457Z] 
[2022-12-14T22:14:47.457Z] > Task :clients:javadocJar
[2022-12-14T22:14:48.522Z] 
[2022-12-14T22:14:48.522Z] > Task :clients:srcJar
[2022-12-14T

[jira] [Created] (KAFKA-14491) Introduce Versioned Key-Value Stores to Kafka Streams

[Victoria Xia (Jira)]

Victoria Xia created KAFKA-14491:


 Summary: Introduce Versioned Key-Value Stores to Kafka Streams
 Key: KAFKA-14491
 URL: https://issues.apache.org/jira/browse/KAFKA-14491
 Project: Kafka
  Issue Type: Improvement
Reporter: Victoria Xia
Assignee: Victoria Xia


The key-value state stores used by Kafka Streams today maintain only the latest 
value associated with each key. In order to support applications which require 
access to older record versions, Kafka Streams should have versioned state 
stores. Versioned state stores are similar to key-value stores except they can 
store multiple record versions for a single key. An example use case for 
versioned key-value stores is in providing proper temporal join semantics for 
stream-tables joins with regards to out-of-order data.

See KIP for more: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #10

[Apache Jenkins Server]

See 

[ANNOUNCE] New committer: Viktor Somogyi-Vass

[Jun Rao]

Hi, Everyone,

The PMC of Apache Kafka is pleased to announce a new Kafka committer Viktor
Somogyi-Vass.

Viktor has been contributing to Kafka since 2018. He contributed 35 PRs and
reviewed 49 PRs.

Congratulations, Viktor!

Thanks,

Jun (on behalf of the Apache Kafka PMC)

[ANNOUNCE] New committer: Ron Dagostino

[Jun Rao]

Hi, Everyone,

The PMC of Apache Kafka is pleased to announce a new Kafka committer Ron
Dagostino.

Ron has been contributing to Kafka since 2018. He has made significant
contributions including OAuth and Sasl re-authentication.

Congratulations, Ron!

Thanks,

Jun (on behalf of the Apache Kafka PMC)

Re: [ANNOUNCE] New committer: Ron Dagostino

[Chris Egerton]

Congrats, Ron!

On Wed, Dec 14, 2022, 18:11 Jun Rao  wrote:

> Hi, Everyone,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer Ron
> Dagostino.
>
> Ron has been contributing to Kafka since 2018. He has made significant
> contributions including OAuth and Sasl re-authentication.
>
> Congratulations, Ron!
>
> Thanks,
>
> Jun (on behalf of the Apache Kafka PMC)
>

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Chris Egerton]

Congrats, Viktor!

On Wed, Dec 14, 2022, 18:10 Jun Rao  wrote:

> Hi, Everyone,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer Viktor
> Somogyi-Vass.
>
> Viktor has been contributing to Kafka since 2018. He contributed 35 PRs and
> reviewed 49 PRs.
>
> Congratulations, Viktor!
>
> Thanks,
>
> Jun (on behalf of the Apache Kafka PMC)
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1436

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 521622 lines...]
[2022-12-14T23:41:04.670Z] 
[2022-12-14T23:41:04.670Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testMigrateEmptyZk() PASSED
[2022-12-14T23:41:04.670Z] 
[2022-12-14T23:41:04.670Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteNewTopicConfigs() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteNewTopicConfigs() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testClaimAndReleaseExistingController() 
STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testClaimAndReleaseExistingController() 
PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteNewClientQuotas() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteNewClientQuotas() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testClaimAbsentController() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testClaimAbsentController() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteExistingTopicConfigs() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteExistingTopicConfigs() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testCreateNewPartitions() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testCreateNewPartitions() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteExistingClientQuotas() STARTED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > ZkMigrationClientTest > testWriteExistingClientQuotas() PASSED
[2022-12-14T23:41:05.718Z] 
[2022-12-14T23:41:05.718Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToController()[1]
 STARTED
[2022-12-14T23:41:07.813Z] 
[2022-12-14T23:41:07.813Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToController()[1]
 PASSED
[2022-12-14T23:41:07.813Z] 
[2022-12-14T23:41:07.813Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToNonController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToNonController()[1]
 STARTED
[2022-12-14T23:41:16.712Z] 
[2022-12-14T23:41:16.712Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 166 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToNonController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToNonController()[1]
 PASSED
[2022-12-14T23:41:23.536Z] 
[2022-12-14T23:41:23.536Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > PlaintextEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe(String) > 
kafka.api.PlaintextEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe(String)[1]
 STARTED
[2022-12-14T23:41:30.871Z] 
[2022-12-14T23:41:30.871Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > PlaintextEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe(String) > 
kafka.api.PlaintextEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe(String)[1]
 PASSED
[2022-12-14T23:41:30.871Z] 
[2022-12-14T23:41:30.871Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 169 > PlaintextEndToEndAuthorizationTest > 
testNoConsumeWithoutD

Re: [DISCUSS] Apache Kafka 3.3.2

[José Armando García Sancio]

Hey Chris,

Here are the results of just running the upgrade system tests on the
latest 3.3 branch:
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1671061000--apache--3.3--69fbaf2457/2022-12-14--001./2022-12-14--001./report.html

Looks like the commit is now 69fbaf2457...

Thanks
-- 
-José

[jira] [Resolved] (KAFKA-14395) Add config to configure client supplier for KafkaStreams

[Hao Li (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li resolved KAFKA-14395.

Resolution: Done

> Add config to configure client supplier for KafkaStreams
> 
>
> Key: KAFKA-14395
> URL: https://issues.apache.org/jira/browse/KAFKA-14395
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
>  Labels: kip
>
> For KIP: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #137

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 336428 lines...]
[2022-12-14T22:42:20.395Z] Exception: java.lang.OutOfMemoryError thrown from 
the UncaughtExceptionHandler in thread "delete-temp-file-shutdown-hook"
[2022-12-14T22:42:55.913Z] 
[2022-12-14T22:42:55.913Z] > Task :streams:integrationTest
[2022-12-14T22:42:55.913Z] 
[2022-12-14T22:42:55.913Z] 
org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade PASSED
[2022-12-14T22:42:55.913Z] 
[2022-12-14T22:42:55.913Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes STARTED
[2022-12-14T22:42:57.705Z] 
[2022-12-14T22:42:57.705Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED
[2022-12-14T22:42:57.705Z] 
[2022-12-14T22:42:57.705Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED
[2022-12-14T22:42:59.549Z] 
[2022-12-14T22:42:59.549Z] > Task :core:integrationTest
[2022-12-14T22:42:59.549Z] 
[2022-12-14T22:42:59.549Z] Exception: java.lang.OutOfMemoryError thrown from 
the UncaughtExceptionHandler in thread "delete-temp-file-shutdown-hook"
[2022-12-14T22:43:18.742Z] 
[2022-12-14T22:43:18.742Z] Exception: java.lang.OutOfMemoryError thrown from 
the UncaughtExceptionHandler in thread "group-metadata-manager-0"
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] > Task :streams:integrationTest
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] STARTED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] PASSED
[2022-12-14T22:43:28.906Z] 
[2022-12-14T22:43:28.906Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTes

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Satish Duggana]

Thanks Divij for the KIP.

I wanted to clarify on a couple of points mentioned in your earlier reply.

1. RSM fetch interface already exposes a streaming interface and it also
does not mandate reading the complete segment before serving the fetch
request. Let me know if we are missing anything here.

2. Having a segment cache was discussed earlier during KIP-405 discussions
but it was decided to keep that out of the  scope for KIP-405. HDFS
implementation that was shared earlier uses inmemory cache, the plan is to
pull into RLM layer with a pluggable cache and provide inmemory and file
based options to avoid implementing in RSM implementations. This cache is
already used in RSM plugins with the shared 2.8.x tiered storage
implementation. Users can implement their custom plugin based on their
needs. We are working on a KIP proposal for the same.

Thanks,
Satish.

On Wed, 14 Dec 2022 at 16:57, Divij Vaidya  wrote:

> > Is the new method enough for doing size-based retention?
>
> Yes. You are right in assuming that this API only provides the Remote
> storage size (for current epoch chain). We would use this API for size
> based retention along with a value of localOnlyLogSegmentSize which is
> computed as Log.sizeInBytes(logSegments.filter(_.baseOffset >
> highestOffsetWithRemoteIndex)). Hence, (total_log_size =
> remoteLogSizeBytes + log.localOnlyLogSegmentSize). I have updated the KIP
> with this information. You can also check an example implementation at
>
> https://github.com/satishd/kafka/blob/2.8.x-tiered-storage/core/src/main/scala/kafka/log/Log.scala#L2077
>
>
> > Do you imagine all accesses to remote metadata will be across the network
> or will there be some local in-memory cache?
>
> I would expect a disk-less implementation to maintain a finite in-memory
> cache for segment metadata to optimize the number of network calls made to
> fetch the data. In future, we can think about bringing this finite size
> cache into RLM itself but that's probably a conversation for a different
> KIP. There are many other things we would like to do to optimize the Tiered
> storage interface such as introducing a circular buffer / streaming
> interface from RSM (so that we don't have to wait to fetch the entire
> segment before starting to send records to the consumer), caching the
> segments fetched from RSM locally (I would assume all RSM plugin
> implementations to do this, might as well add it to RLM) etc.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Dec 12, 2022 at 7:35 PM Jun Rao  wrote:
>
> > Hi, Divij,
> >
> > Thanks for the reply.
> >
> > Is the new method enough for doing size-based retention? It gives the
> total
> > size of the remote segments, but it seems that we still don't know the
> > exact total size for a log since there could be overlapping segments
> > between the remote and the local segments.
> >
> > You mentioned a disk-less implementation. Do you imagine all accesses to
> > remote metadata will be across the network or will there be some local
> > in-memory cache?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Dec 7, 2022 at 3:10 AM Divij Vaidya 
> > wrote:
> >
> > > The method is needed for RLMM implementations which fetch the
> information
> > > over the network and not for the disk based implementations (such as
> the
> > > default topic based RLMM).
> > >
> > > I would argue that adding this API makes the interface more generic
> than
> > > what it is today. This is because, with the current APIs an implementor
> > is
> > > restricted to use disk based RLMM solutions only (i.e. the default
> > > solution) whereas if we add this new API, we unblock usage of network
> > based
> > > RLMM implementations such as databases.
> > >
> > >
> > >
> > > On Wed 30. Nov 2022 at 20:40, Jun Rao 
> wrote:
> > >
> > > > Hi, Divij,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > Point#2. My high level question is that is the new method needed for
> > > every
> > > > implementation of remote storage or just for a specific
> implementation.
> > > The
> > > > issues that you pointed out exist for the default implementation of
> > RLMM
> > > as
> > > > well and so far, the default implementation hasn't found a need for a
> > > > similar new method. For public interface, ideally we want to make it
> > more
> > > > general.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Nov 21, 2022 at 7:11 AM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you Jun and Alex for your comments.
> > > > >
> > > > > Point#1: You are right Jun. As Alex mentioned, the "derived
> metadata"
> > > can
> > > > > increase the size of cached metadata by a factor of 10 but it
> should
> > be
> > > > ok
> > > > > to cache just the actual metadata. My point about size being a
> > > limitation
> > > > > for using cache is not valid anymore.
> > > > >
> > > > > Point#2: For a new replica, it would still have to fetch the
> metadata
> > > > over
> > > > > the network to initiate the warm 

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Sophie Blee-Goldman]

Hey Ron, sure, go ahead and merge this to 3.4

Thanks

On Mon, Dec 12, 2022 at 12:25 PM Ron Dagostino  wrote:

> Hi Sophie. https://github.com/apache/kafka/pull/12856 "KAFKA-14392:
> KRaft broker heartbeat timeout should not exceed
> broker.session.timeout.ms" just merged to trunk and seems like a
> potential candidate for 3.4.0 as it is a very small and targeted fix
> with very low risk.  Could it be included even though it is not a
> regression?
>
> Ron
>
> On Sat, Dec 10, 2022 at 1:00 AM Sophie Blee-Goldman
>  wrote:
> >
> > Hey everyone,
> >
> > Rejoice (or not) for we are now past code freeze for the 3.4 release!
> This
> > means you
> > should not be merging any new code to the 3.4 branch unless/until it's
> been
> > approved
> > as a blocker. For now feel free to also merge things like flaky test
> fixes
> > and docs PRs
> > that don't touch any non-testing code -- consider those automatically
> > approved (though
> > it never hurts to ping me for a heads up).
> >
> > I'll be going through the list of unresolved tickets that are still
> marked
> > for 3.4 and will
> > bump anything that's not already approved or a flaky test ticket out to
> > 3.5. So please
> > take a moment to go through any open 3.4 tickets that belong to you. This
> > page 
> > provides a useful view for what's still out there.
> >
> > As always let me know if you have any questions and make sure to raise
> any
> > potential
> > blockers here for approval.
> >
> > Devij: thanks for the offer to help reenable the ARM build, but it was
> only
> > disabled for
> > tooling reasons (often failed due to lack of available nodes). I guess
> when
> > we can
> > finally drop support for Java 8 then we can use those nodes for the ARM
> > build.
> > Until then, rest assured I will still validate the 3.4 release on ARM
> >
> > Sophie
> >
> > On Fri, Dec 9, 2022 at 9:12 AM Divij Vaidya 
> wrote:
> >
> > > Hey Sophie
> > >
> > > I was wondering whether we should try to get the ARM build working
> again so
> > > that we can perform release qualification of 3.4.x on ARM as well
> (which
> > > was disabled in https://github.com/apache/kafka/pull/12380)? I would
> be
> > > happy to pick up the work if someone can point me to the history on
> this
> > > subject.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Thu, Dec 8, 2022 at 3:48 AM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > Thanks everyone for the updates, that all sounds good.
> > > >
> > > > Divij: I pinged some of the relevant reviewers to give your PRs a
> final
> > > > pass, but will
> > > > leave it to their judgement from here as I'm not familiar with
> either.
> > > >
> > > > Everyone else: reminder that today is the official code freeze
> deadline,
> > > so
> > > > please try
> > > > and get your PRs in by EOD! If you have something on the edge and
> worry
> > > > it's not
> > > > ready, feel free to reach out for an extension: I'd rather give you
> an
> > > > extra day or two
> > > > than risk finding a blocker weeks from now because a PR was rushed :)
> > > >
> > > > On Wed, Dec 7, 2022 at 2:04 PM Chris Egerton  >
> > > > wrote:
> > > >
> > > > > Hi Sophie,
> > > > >
> > > > > Thanks for taking a look at the MM2 issue. We've merged a
> > > > > (minimally-scoped) fix and backported it to the 3.4 branch; the
> issue
> > > > > should be resolved now.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, Dec 7, 2022 at 3:12 PM Rajini Sivaram <
> rajinisiva...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Sophie,
> > > > > >
> > > > > > I have merged PR #12954 for KIP-881 to 3.4 branch. Please let me
> know
> > > > if
> > > > > > that is ok.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 7, 2022 at 11:43 AM Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Sophie,
> > > > > > >
> > > > > > > The first PR for KIP-881 which contains protocol changes has
> been
> > > > > merged
> > > > > > > to trunk (https://github.com/apache/kafka/pull/12954). It is a
> > > > > > relatively
> > > > > > > small PR, can we merge to 3.4.0?
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 7, 2022 at 11:16 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hey Sophie
> > > > > > >>
> > > > > > >> I have a couple of pending PRs which have been waiting for
> review
> > > > > since
> > > > > > >> preparation of the 3.3 release. They are not blockers for 3.4
> but
> > > > are
> > > > > > >> being
> > > > > > >> tracked as improvements that we would like to add to 3.4
> release.
> > > > > > >>
> > > > > > >> Please consider taking a look when you get a chance:
> > > > > > >>
> > > > > > >> 1. https://issues.apache.org/jira/browse/KAFKA-7109
> > > > > > >

[jira] [Created] (KAFKA-14492) Extract a method to create LogManager, in order to be overrided by subclass of KafkaServer

[Phantom Ma (Jira)]

Phantom Ma created KAFKA-14492:
--

 Summary: Extract a method to create LogManager, in order to be 
overrided by subclass of KafkaServer
 Key: KAFKA-14492
 URL: https://issues.apache.org/jira/browse/KAFKA-14492
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Phantom Ma


In our scene, we want to implement a subclass of LogManager, add do some 
interception like encrypting/auditing.
Even more, we want enhance the write log performance in the future. 

Split alone a create method can provide an extension point to allow user define 
the action from outside kafka core.
Base kafka, entend it, rather than modification.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Ron Dagostino

[ziming deng]

Congratulations, Ron!
Well deserved!

--
Ziming

> On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> 
> Congratulations, Ron!
> Well deserved!
> 
> Luke

[jira] [Created] (KAFKA-14493) Zk to KRaft migration state machine in KRaft controller

[Akhilesh Chaganti (Jira)]

Akhilesh Chaganti created KAFKA-14493:
-

 Summary: Zk to KRaft migration state machine in KRaft controller
 Key: KAFKA-14493
 URL: https://issues.apache.org/jira/browse/KAFKA-14493
 Project: Kafka
  Issue Type: Sub-task
Reporter: Akhilesh Chaganti
Assignee: Akhilesh Chaganti






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14291) KRaft: ApiVersionsResponse doesn't have finalizedFeatures and finalizedFeatureEpoch in KRaft mode

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-14291.

Resolution: Duplicate

Marking this as a duplicate of KAFKA-13990 as described above

> KRaft: ApiVersionsResponse doesn't have finalizedFeatures and 
> finalizedFeatureEpoch in KRaft mode
> -
>
> Key: KAFKA-14291
> URL: https://issues.apache.org/jira/browse/KAFKA-14291
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Reporter: Akhilesh Chaganti
>Priority: Critical
>
> https://github.com/apache/kafka/blob/d834947ae7abc8a9421d741e742200bb36f51fb3/core/src/main/scala/kafka/server/ApiVersionManager.scala#L53
> ```
> class SimpleApiVersionManager(
>   val listenerType: ListenerType,
>   val enabledApis: collection.Set[ApiKeys],
>   brokerFeatures: Features[SupportedVersionRange]
> ) extends ApiVersionManager {
>   def this(listenerType: ListenerType) = {
> this(listenerType, ApiKeys.apisForListener(listenerType).asScala, 
> BrokerFeatures.defaultSupportedFeatures())
>   }
>   private val apiVersions = 
> ApiVersionsResponse.collectApis(enabledApis.asJava)
>   override def apiVersionResponse(requestThrottleMs: Int): 
> ApiVersionsResponse = {
> ApiVersionsResponse.createApiVersionsResponse(requestThrottleMs, 
> apiVersions, brokerFeatures)
>   }
> }
> ```
> ApiVersionManager for KRaft doesn't add the finalizedFeatures and 
> finalizedFeatureEpoch to the ApiVersionsResponse.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4852) ByteBufferSerializer not compatible with offsets

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-4852.
---
Resolution: Fixed

> ByteBufferSerializer not compatible with offsets
> 
>
> Key: KAFKA-4852
> URL: https://issues.apache.org/jira/browse/KAFKA-4852
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.1
> Environment: all
>Reporter: Werner Daehn
>Assignee: LinShunkang
>Priority: Minor
> Fix For: 3.4.0
>
>
> Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the 
> ByteBuffer was created with an offset? new ByteBuffer(data, 3, 10)? The 
> ByteBufferSerializer will send from pos=0 and not from pos=3 onwards.
> Solution: No rewind() but flip() for reading a ByteBuffer. That's what the 
> flip is meant for.
> Story:
> Imagine the incoming data comes from a byte[], e.g. a network stream 
> containing topicname, partition, key, value, ... and you want to create a new 
> ProducerRecord for that. As the constructor of ProducerRecord requires 
> (topic, partition, key, value) you have to copy from above byte[] the key and 
> value. That means there is a memcopy taking place. Since the payload can be 
> potentially large, that introduces a lot of overhead. Twice the memory.
> A nice solution to this problem is to simply wrap the network byte[] into new 
> ByteBuffers:
> ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength);
> ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength);
> and then use the ByteBufferSerializer instead of the ByteArraySerializer.
> But that does not work as the ByteBufferSerializer does a rewind(), hence 
> both, key and value, will start at position=0 of the data[].
> public class ByteBufferSerializer implements Serializer {
> public byte[] serialize(String topic, ByteBuffer data) {
>  data.rewind();



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13736) Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-13736.

Resolution: Fixed

> Flaky kafka.network.SocketServerTest.closingChannelWithBufferedReceives
> ---
>
> Key: KAFKA-13736
> URL: https://issues.apache.org/jira/browse/KAFKA-13736
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Priority: Blocker
>  Labels: flakey, flaky-test
>
> Examples:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-11895/1/tests
> {code}
> java.lang.AssertionError: receiveRequest timed out
>   at 
> kafka.network.SocketServerTest.receiveRequest(SocketServerTest.scala:140)
>   at 
> kafka.network.SocketServerTest.$anonfun$verifyRemoteCloseWithBufferedReceives$6(SocketServerTest.scala:1521)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
>   at 
> kafka.network.SocketServerTest.$anonfun$verifyRemoteCloseWithBufferedReceives$1(SocketServerTest.scala:1520)
>   at 
> kafka.network.SocketServerTest.verifyRemoteCloseWithBufferedReceives(SocketServerTest.scala:1483)
>   at 
> kafka.network.SocketServerTest.closingChannelWithBufferedReceives(SocketServerTest.scala:1431)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-13602) Allow to broadcast a result record

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman reopened KAFKA-13602:


Reverting in 3.4 due to logging-related perf/security issue, and some open 
semantic questions. Retargeting for 3.5

> Allow to broadcast a result record
> --
>
> Key: KAFKA-13602
> URL: https://issues.apache.org/jira/browse/KAFKA-13602
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Sagar Rao
>Priority: Major
>  Labels: needs-kip, newbie++
> Fix For: 3.4.0
>
>
> From time to time, users ask how they can send a record to more than one 
> partition in a sink topic. Currently, this is only possible by replicate the 
> message N times before the sink and use a custom partitioner to write the N 
> messages into the N different partitions.
> It might be worth to make this easier and add a new feature for it. There are 
> multiple options:
>  * extend `to()` / `addSink()` with a "broadcast" option/config
>  * add `toAllPartitions()` / `addBroadcastSink()` methods
>  * allow StreamPartitioner to return `-1` for "all partitions"
>  * extend `StreamPartitioner` to allow returning more than one partition (ie 
> a list/collection of integers instead of a single int)
> The first three options imply that a "full broadcast" is supported only, so 
> it's less flexible. On the other hand, it's easier to use (especially the 
> first two options are easy as they do not require to implement a custom 
> partitioner).
> The last option would be most flexible and also allow for a "partial 
> broadcast" (aka multi-cast) pattern. It might also be possible to combine two 
> options, or maye even a totally different idea.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Ron Dagostino

[Satish Duggana]

Congratulations, Ron!!

On Thu, 15 Dec 2022 at 07:48, ziming deng  wrote:

> Congratulations, Ron!
> Well deserved!
>
> --
> Ziming
>
> > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> >
> > Congratulations, Ron!
> > Well deserved!
> >
> > Luke
>
>

[jira] [Created] (KAFKA-14494) Kafka Java client can't send data when behind SOCKS proxy - while native client can

[Oleg Zhovtanyuk (Jira)]

Oleg Zhovtanyuk created KAFKA-14494:
---

 Summary: Kafka Java client can't send data when behind SOCKS proxy 
- while native client can
 Key: KAFKA-14494
 URL: https://issues.apache.org/jira/browse/KAFKA-14494
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.3.1
Reporter: Oleg Zhovtanyuk


When Kafka Java client sits behind the SOCK5 proxy, it can connect to the 
cluster, get the list of brokers, but enters the infinite loop trying to detect 
the least loaded broker.

To the contrary, NodeJS client (a wrapper for librdkafka) perform the same 
steps, but proceeds further to the binary data exchange.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)