[jira] [Created] (KAFKA-16811) Punctuate Ratio metric almost impossible to track

[Sebastien Viale (Jira)]

Sebastien Viale created KAFKA-16811:
---

 Summary: Punctuate Ratio metric almost impossible to track
 Key: KAFKA-16811
 URL: https://issues.apache.org/jira/browse/KAFKA-16811
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.7.0
Reporter: Sebastien Viale


The Punctuate ratio metric is returned after the last record of the poll loop. 
It is recomputed in every poll loop.

After a puntuate, the value is close to 1, but there is little chance that 
metric is sampled at this time. 
So its value is almost always 0.   

A solution could be to apply a kind of "sliding window" to it and report the 
value for the last x seconds.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

[Matthias J. Sax]

I was not aware of `InternalFixedKeyRecordFactory`. As the name 
indicates, it's considered an internal class, so not sure if we should 
recommend to use it in test...


I understand why this class is required, and why it was put into a 
public package; the way Java works, enforces this. Not sure if we could 
find a better solution.


Might be good to hear from others.


-Matthias

On 5/21/24 3:57 PM, Shashwat Pandey wrote:

Looking at the ticket and the sample code, I think it would be possible to
continue using `InternalFixedKeyRecordFactory` as the avenue to create
`FixedKeyRecord`s in tests. As long as there was a
MockFixedKeyProcessorContext, I think we would be able to test
FixedKeyProcessors without a Topology.

I created a sample repo with the `MockFixedKeyProcessorContext` here is
what I think the tests would look like:
https://github.com/s7pandey/kafka-processor-tests/blob/main/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java



On Mon, May 20, 2024 at 9:05 PM Matthias J. Sax  wrote:


Had a discussion on https://issues.apache.org/jira/browse/KAFKA-15242
and it was pointed out, that we also need to do something about
`FixedKeyRecord`. It does not have a public constructor (what is
correct; it should not have one). However, this makes testing
`FixedKeyProcessor` impossible w/o extending `FixedKeyRecord` manually
what does not seem to be right (too clumsy).

It seems, we either need some helper builder method (but not clear to me
where to add it in an elegant way) which would provide us with a
`FixedKeyRecord`, or add some sub-class to the test-utils module which
would extend `FixedKeyRecord`? -- Or maybe an even better solution? I
could not think of something else so far.


Thoughts?


On 5/3/24 9:46 AM, Matthias J. Sax wrote:

Please also update the KIP.

To get a wiki account created, please request it via a commet on this
ticket: https://issues.apache.org/jira/browse/INFRA-25451

After you have the account, please share your wiki id, and we can give
you write permission on the wiki.



-Matthias

On 5/3/24 6:30 AM, Shashwat Pandey wrote:

Hi Matthias,

Sorry this fell out of my radar for a bit.

Revisiting the topic, I think you’re right and we accept the duplicated
nesting as an appropriate solution to not affect the larger public API.

I can update my PR with the change.

Regards,
Shashwat Pandey


On Wed, May 1, 2024 at 11:00 PM Matthias J. Sax 

wrote:



Any updates on this KIP?

On 3/28/24 4:11 AM, Matthias J. Sax wrote:

It seems that `MockRecordMetadata` is a private class, and thus not
part
of the public API. If there are any changes required, we don't need to
discuss on the KIP.


For `CapturedPunctuator` and `CapturedForward` it's a little bit more
tricky. My gut feeling is, that the classes might not need to be
changed, but if we use them within `MockProcessorContext` and
`MockFixedKeyProcessorContext` it might be weird to keep the current
nesting... The problem I see is, that it's not straightforward how to
move the classes w/o breaking compatibility, nor if we duplicate
them as
standalone classes w/o a larger "splash radius". (We would need to add
new overloads for MockProcessorContext#scheduledPunctuators() and
MockProcessorContext#forwarded()).

Might be good to hear from others if we think it's worth this larger
changes to get rid of the nesting, or just accept the somewhat not
ideal
nesting as it technically is not a real issue?


-Matthias


On 3/15/24 1:47 AM, Shashwat Pandey wrote:

Thanks for the feedback Matthias!

The reason I proposed the extension of MockProcessorContext was more
to do
with the internals of the class (MockRecordMetadata,
CapturedPunctuator and
CapturedForward).

However, I do see your point, I would then think to split
MockProcessorContext and MockFixedKeyProcessorContext, some of the
internal
classes should also be extracted i.e. MockRecordMetadata,
CapturedPunctuator and probably a new CapturedFixedKeyForward.

Let me know what you think!


Regards,
Shashwat Pandey


On Mon, Mar 11, 2024 at 10:09 PM Matthias J. Sax 
wrote:


Thanks for the KIP Shashwat. Closing this testing gap is great! It
did
come up a few time already...

One question: why do you propose to `extend MockProcessorContext`?

Given how the actual runtime context classes are setup, it seems

that

the regular context and fixed-key-context are distinct, and thus I
believe both mock-context classes should be distinct, too?

What I mean is that FixedKeyProcessorContext does not extend
ProcessorContext. Both classes have a common parent

ProcessINGContext

(note the very similar but different names), but they are "siblings"
only, so why make the mock processor a parent-child relationship?

It seems better to do

public class MockFixedKeyProcessorContext
  implements FixedKeyProcessorContext,
 RecordCollector.Supplier


Of course, if there is code we can share between both mock-context

we

should so this, but it should not leak into the public API?


-Matthia

Permission to contribute to Apache Kafka

[Frédérik Rouleau]

Hi,
As I now have my wiki Id: frouleau and my Jira Id: fred-ro, can I have the
permission to contribute to KIP ?

Regards,

[jira] [Created] (KAFKA-16812) The tools-related tests are slow

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16812:
--

 Summary: The tools-related tests are slow
 Key: KAFKA-16812
 URL: https://issues.apache.org/jira/browse/KAFKA-16812
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


see 
https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2923/testReport/org.apache.kafka.tools/

Maybe we run too many cluster types (5), and we can remove some unrelated types 
for those tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Permission to contribute to Apache Kafka

[Yash Mayya]

Hi Frédérik,

I've granted you the necessary permissions. Let me know if something
doesn't work as expected.

Cheers,
Yash

On Wed, May 22, 2024 at 1:38 PM Frédérik Rouleau
 wrote:

> Hi,
> As I now have my wiki Id: frouleau and my Jira Id: fred-ro, can I have the
> permission to contribute to KIP ?
>
> Regards,
>

[jira] [Created] (KAFKA-16813) Add global timeout for "@Test" and "@TestTemplate"

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16813:
--

 Summary: Add global timeout for "@Test" and "@TestTemplate"
 Key: KAFKA-16813
 URL: https://issues.apache.org/jira/browse/KAFKA-16813
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


in code base `@Test` is used by unit test and `@TestTemplate` is used by 
integration test. The later includes `ParameterizedTest`, `ClusterTest`, 
`ClusterTests`, and `ClusterTemplate`. Hence, we can add two different timeout 
for `@Test` and `@TestTemplate`. For example:

junit.jupiter.execution.timeout.default = 30s
junit.jupiter.execution.timeout.testtemplate.method.default = 120s

The accurate timeout value may need more discussion, but we can try it in small 
junit5 module first. For example: tools module and storage module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2925

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16814) KRaft broker cannot startup when `partition.metadata` is missing

[Luke Chen (Jira)]

Luke Chen created KAFKA-16814:
-

 Summary: KRaft broker cannot startup when `partition.metadata` is 
missing
 Key: KAFKA-16814
 URL: https://issues.apache.org/jira/browse/KAFKA-16814
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Luke Chen


When starting up kafka logManager, we'll check stray replicas to avoid some 
corner cases. But this check might cause broker unable to startup if 
`partition.metadata` is missing because when startup kafka, we load log from 
file, and the topicId of the log is coming from `partition.metadata` file. So, 
if `partition.metadata` is missing, the topicId will be None, and the 
`LogManager#isStrayKraftReplica` will fail with no topicID error.

The `partition.metadata` missing could be some storage failure, or another 
possible path is unclean shutdown after topic is created in the replica, but 
before data is flushed into `partition.metadata` file. This is possible because 
we do the flush, it's done async 
[here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L229].

 

 
{code:java}
ERROR Encountered fatal fault: Error starting LogManager 
(org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
java.lang.RuntimeException: The log dir 
Log(dir=/tmp/kraft-broker-logs/quickstart-events-0, topic=quickstart-events, 
partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, 
logEndOffset=0) does not have a topic ID, which is not allowed when running in 
KRaft mode.
    at 
kafka.log.LogManager$.$anonfun$isStrayKraftReplica$1(LogManager.scala:1609)
    at scala.Option.getOrElse(Option.scala:201)
    at kafka.log.LogManager$.isStrayKraftReplica(LogManager.scala:1608)
    at 
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1(BrokerMetadataPublisher.scala:294)
    at 
kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1$adapted(BrokerMetadataPublisher.scala:294)
    at kafka.log.LogManager.loadLog(LogManager.scala:359)
    at kafka.log.LogManager.$anonfun$loadLogs$15(LogManager.scala:493)
    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1623) {code}
 

Because if we don't do the isStrayKraftReplica check, the topicID and the 
`partition.metadata` will get recovered after getting topic partition update 
and becoming leader or follower later. I'm proposing we skip the 
`isStrayKraftReplica` check if topicID is None, instead of throwing exception 
to terminate the kafka. `isStrayKraftReplica` check is just for a corner case 
only, it should be fine IMO.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1048 Improve kafka-consumer-perf-test to benchmark single partition

[Harsh Panchal]

Hi,

I would like to propose a change in the kafka-consumer-perf-test tool to
support perf testing specific partitions.

kafka-consumer-perf-test is a great tool to quickly check raw consumer
performance. Currently, It subscribes to all the partitions and gives
overall cluster performance, however If we want to test performance of a
single broker/partition, existing tool does not support it.

I propose two optional flags --partitions and --offsets which gives
flexibility to benchmark only specific partitions optionally from specified
offsets.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1048%3A+Improve+kafka-consumer-perf-test+to+benchmark+single+partition

Regards,
Harsh Panchal

[DISCUSS] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

[jiang dou]

Hi


I would like to propose a change in the kafka-stream summary log。

Now the summary of stream-tread is record every two minutes, and not
support close  or update intervals.

When the kafka  is running， this is absolutely unnecessary and even harmful
since it fills the logs and thus storage space with unwanted and useless
data.

I propose adding a configuration to control the output interval or disable
it

KIP：
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1049%3A+Add+config+log.summary.interval.ms+to+Kafka+Streams

[jira] [Created] (KAFKA-16815) Handle FencedInstanceId on heartbeat for new consumer

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16815:
--

 Summary: Handle FencedInstanceId on heartbeat for new consumer
 Key: KAFKA-16815
 URL: https://issues.apache.org/jira/browse/KAFKA-16815
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans


With the new consumer group protocol, a member could receive a 
FencedInstanceIdError in the heartbeat response. This could be the case when an 
active member using a group instance id is removed from the group by an admin 
client. If a second member joins with the same instance id, the first member 
will receive a FencedInstanceId on the next heartbeat response. This should be 
treated as a fatal error (consumer should not attempt to rejoin). 

Currently, the FencedInstanceId is not explicitly handled by the client in the 
HeartbeatRequestManager. It ends up being treated as a fatal error, see 
[here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L417]
 (just because it lands on the "unexpected" error category). We should handle 
it explicitly, just to make sure that we express that it's is an expected 
error: log a proper message for it and fail (handleFatalFailure). We should 
also that the error is included in the tests that cover the HB request error 
handling 
([here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/clients/src/test/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManagerTest.java#L798])

    



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Frank Yang]

Hi team,

Chia-Ping Tsai and I would like to propose KIP-1047 to migrate 
kafka.serializer.Decoder from core module (scala) to tools module (java).

Feedback and comments are welcome.

KIP-1047: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
JIRA: https://issues.apache.org/jira/browse/KAFKA-16796

Thank you.
PoAn

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Dear all,

We know that  3.8.0 KIP is already frozen, but this is a small KIP and we need 
to ship it to 3.8.0 so as to remove the deprecated scala interface from 4.0.

Best,
Chia-Ping

On 2024/05/22 14:05:16 Frank Yang wrote:
> Hi team,
> 
> Chia-Ping Tsai and I would like to propose KIP-1047 to migrate 
> kafka.serializer.Decoder from core module (scala) to tools module (java).
> 
> Feedback and comments are welcome.
> 
> KIP-1047: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> 
> Thank you.
> PoAn

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Mickael Maison]

Hi,

Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
it would allow us to migrate this tool in 4.0.0. We're unfortunately
past the KIP deadline but maybe this is small enough to have an
exception.

I'm wondering whether we need to introduce a new Decoder interface and
instead if we could reuse Deserializer. We could deprecate the
key-decoder-class and value-decoder-class flags and introduce new
flags like key-deserializer-class and value-deserializer-class. One
benefit is that we already have many existing deserializer
implementations. WDYT?

One issue I also noted is that some of the existing Decoder
implementations (StringDecoder for example) can accept configurations
but currently DumpLogSegments does not provide a way to pass any
configurations, it creates an empty VerifiableProperties object each
time it instantiates a Decoder instance. If we were to use
Deserializer we would also need a way to provide configurations.

Thanks,
Mickael

On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai  wrote:
>
> Dear all,
>
> We know that  3.8.0 KIP is already frozen, but this is a small KIP and we 
> need to ship it to 3.8.0 so as to remove the deprecated scala interface from 
> 4.0.
>
> Best,
> Chia-Ping
>
> On 2024/05/22 14:05:16 Frank Yang wrote:
> > Hi team,
> >
> > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate 
> > kafka.serializer.Decoder from core module (scala) to tools module (java).
> >
> > Feedback and comments are welcome.
> >
> > KIP-1047: 
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> >
> > Thank you.
> > PoAn

[jira] [Created] (KAFKA-16816) Remove unneeded FencedInstanceId support on commit path for new consumer

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16816:
--

 Summary: Remove unneeded FencedInstanceId support on commit path 
for new consumer
 Key: KAFKA-16816
 URL: https://issues.apache.org/jira/browse/KAFKA-16816
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans


The new consumer contains logic related to handling FencedInstanceId exception 
received as a response to an OffsetCommit request (on the 
[consumer|https://github.com/apache/kafka/blob/028e7a06dcdca7d4dbeae83f2fce0a4120cc2753/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L776]
 and [commit 
manager|https://github.com/apache/kafka/blob/028e7a06dcdca7d4dbeae83f2fce0a4120cc2753/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L715]),
 but with the new group protocol, we will never get that error on a commit 
response. We should remove the code that expects the FencedInstanceId on the 
commit response, and also clean up the other related usages that we added to 
propagate the FencedInstanceId exception on the poll, commitSync and 
commitAsync API. Note that throwing that exception is part of the contract of 
the poll, commitSync and commitAsync APIs of the KafkaConsumer, but it changes 
with the new protocol. We should update the java doc for the new 
AsyncKafkaConsumer to reflect the change.  

 

With the new protocol If a consumer tries to commit offsets, there could be 2 
cases:
 # empty group -> commit succeeds, fencing an instance id would never happen 
because group is empty
 # non-empty group -> commit fails with UnknownMemberId, indicating that the 
member is not known to the group. The consumer needs to join the non-empty 
group in order to commit offsets to it. To complete the story, the moment the 
consumer attempts to join, it will receive an UnreleasedInstanceId error on the 
HB response, indicating it using a groupInstanceId that is already in use.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Thanks for Mickael response!

>I'm wondering whether we need to introduce a new Decoder interface and
instead if we could reuse Deserializer. We could deprecate the
key-decoder-class and value-decoder-class flags and introduce new
flags like key-deserializer-class and value-deserializer-class. One
benefit is that we already have many existing deserializer
implementations. WDYT?

I prefer to use different interface, since using the same interface
(Deserializer) may obstruct us from enhancing the interface used by
DumpLogSegments only in the future.

> One issue I also noted is that some of the existing Decoder
implementations (StringDecoder for example) can accept configurations
but currently DumpLogSegments does not provide a way to pass any
configurations, it creates an empty VerifiableProperties object each
time it instantiates a Decoder instance. If we were to use
Deserializer we would also need a way to provide configurations.

yep, that is a known issue:
https://issues.apache.org/jira/browse/KAFKA-12311

We will file PR to fix it

Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：

> Hi,
>
> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
> it would allow us to migrate this tool in 4.0.0. We're unfortunately
> past the KIP deadline but maybe this is small enough to have an
> exception.
>
> I'm wondering whether we need to introduce a new Decoder interface and
> instead if we could reuse Deserializer. We could deprecate the
> key-decoder-class and value-decoder-class flags and introduce new
> flags like key-deserializer-class and value-deserializer-class. One
> benefit is that we already have many existing deserializer
> implementations. WDYT?
>
> One issue I also noted is that some of the existing Decoder
> implementations (StringDecoder for example) can accept configurations
> but currently DumpLogSegments does not provide a way to pass any
> configurations, it creates an empty VerifiableProperties object each
> time it instantiates a Decoder instance. If we were to use
> Deserializer we would also need a way to provide configurations.
>
> Thanks,
> Mickael
>
> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
> wrote:
> >
> > Dear all,
> >
> > We know that  3.8.0 KIP is already frozen, but this is a small KIP and
> we need to ship it to 3.8.0 so as to remove the deprecated scala interface
> from 4.0.
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/05/22 14:05:16 Frank Yang wrote:
> > > Hi team,
> > >
> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
> kafka.serializer.Decoder from core module (scala) to tools module (java).
> > >
> > > Feedback and comments are welcome.
> > >
> > > KIP-1047:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> > >
> > > Thank you.
> > > PoAn
>

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

> One issue I also noted is that some of the existing Decoder
implementations (StringDecoder for example) can accept configurations
but currently DumpLogSegments does not provide a way to pass any
configurations, it creates an empty VerifiableProperties object each
time it instantiates a Decoder instance. If we were to use
Deserializer we would also need a way to provide configurations.

BTW, if the known bug gets fixed, we have to make new interface extend
`configurable`.

Or we can just ignore the known issue as `DumpLogSegments` has no options
to take custom configs for `Decoder`. That allow the `Decoder` more simple


Chia-Ping Tsai  於 2024年5月22日 週三 下午10:58寫道：

>
> Thanks for Mickael response!
>
> >I'm wondering whether we need to introduce a new Decoder interface and
> instead if we could reuse Deserializer. We could deprecate the
> key-decoder-class and value-decoder-class flags and introduce new
> flags like key-deserializer-class and value-deserializer-class. One
> benefit is that we already have many existing deserializer
> implementations. WDYT?
>
> I prefer to use different interface, since using the same interface
> (Deserializer) may obstruct us from enhancing the interface used by
> DumpLogSegments only in the future.
>
> > One issue I also noted is that some of the existing Decoder
> implementations (StringDecoder for example) can accept configurations
> but currently DumpLogSegments does not provide a way to pass any
> configurations, it creates an empty VerifiableProperties object each
> time it instantiates a Decoder instance. If we were to use
> Deserializer we would also need a way to provide configurations.
>
> yep, that is a known issue:
> https://issues.apache.org/jira/browse/KAFKA-12311
>
> We will file PR to fix it
>
> Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：
>
>> Hi,
>>
>> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
>> it would allow us to migrate this tool in 4.0.0. We're unfortunately
>> past the KIP deadline but maybe this is small enough to have an
>> exception.
>>
>> I'm wondering whether we need to introduce a new Decoder interface and
>> instead if we could reuse Deserializer. We could deprecate the
>> key-decoder-class and value-decoder-class flags and introduce new
>> flags like key-deserializer-class and value-deserializer-class. One
>> benefit is that we already have many existing deserializer
>> implementations. WDYT?
>>
>> One issue I also noted is that some of the existing Decoder
>> implementations (StringDecoder for example) can accept configurations
>> but currently DumpLogSegments does not provide a way to pass any
>> configurations, it creates an empty VerifiableProperties object each
>> time it instantiates a Decoder instance. If we were to use
>> Deserializer we would also need a way to provide configurations.
>>
>> Thanks,
>> Mickael
>>
>> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
>> wrote:
>> >
>> > Dear all,
>> >
>> > We know that  3.8.0 KIP is already frozen, but this is a small KIP and
>> we need to ship it to 3.8.0 so as to remove the deprecated scala interface
>> from 4.0.
>> >
>> > Best,
>> > Chia-Ping
>> >
>> > On 2024/05/22 14:05:16 Frank Yang wrote:
>> > > Hi team,
>> > >
>> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
>> kafka.serializer.Decoder from core module (scala) to tools module (java).
>> > >
>> > > Feedback and comments are welcome.
>> > >
>> > > KIP-1047:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
>> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
>> > >
>> > > Thank you.
>> > > PoAn
>>
>

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Josep Prat]

Hi all,

We can add this to 3.8.0, but keep in mind the KIP is not voted yet (as far
as I can see), so I would highly encourage to start the vote thread ASAP
and strat with the implementation right after.

Best,

-
Josep Prat
Open Source Engineering Director, aivenjosep.p...@aiven.io   |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Wed, May 22, 2024, 17:06 Chia-Ping Tsai  wrote:

> > One issue I also noted is that some of the existing Decoder
> implementations (StringDecoder for example) can accept configurations
> but currently DumpLogSegments does not provide a way to pass any
> configurations, it creates an empty VerifiableProperties object each
> time it instantiates a Decoder instance. If we were to use
> Deserializer we would also need a way to provide configurations.
>
> BTW, if the known bug gets fixed, we have to make new interface extend
> `configurable`.
>
> Or we can just ignore the known issue as `DumpLogSegments` has no options
> to take custom configs for `Decoder`. That allow the `Decoder` more simple
>
>
> Chia-Ping Tsai  於 2024年5月22日 週三 下午10:58寫道：
>
> >
> > Thanks for Mickael response!
> >
> > >I'm wondering whether we need to introduce a new Decoder interface and
> > instead if we could reuse Deserializer. We could deprecate the
> > key-decoder-class and value-decoder-class flags and introduce new
> > flags like key-deserializer-class and value-deserializer-class. One
> > benefit is that we already have many existing deserializer
> > implementations. WDYT?
> >
> > I prefer to use different interface, since using the same interface
> > (Deserializer) may obstruct us from enhancing the interface used by
> > DumpLogSegments only in the future.
> >
> > > One issue I also noted is that some of the existing Decoder
> > implementations (StringDecoder for example) can accept configurations
> > but currently DumpLogSegments does not provide a way to pass any
> > configurations, it creates an empty VerifiableProperties object each
> > time it instantiates a Decoder instance. If we were to use
> > Deserializer we would also need a way to provide configurations.
> >
> > yep, that is a known issue:
> > https://issues.apache.org/jira/browse/KAFKA-12311
> >
> > We will file PR to fix it
> >
> > Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：
> >
> >> Hi,
> >>
> >> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
> >> it would allow us to migrate this tool in 4.0.0. We're unfortunately
> >> past the KIP deadline but maybe this is small enough to have an
> >> exception.
> >>
> >> I'm wondering whether we need to introduce a new Decoder interface and
> >> instead if we could reuse Deserializer. We could deprecate the
> >> key-decoder-class and value-decoder-class flags and introduce new
> >> flags like key-deserializer-class and value-deserializer-class. One
> >> benefit is that we already have many existing deserializer
> >> implementations. WDYT?
> >>
> >> One issue I also noted is that some of the existing Decoder
> >> implementations (StringDecoder for example) can accept configurations
> >> but currently DumpLogSegments does not provide a way to pass any
> >> configurations, it creates an empty VerifiableProperties object each
> >> time it instantiates a Decoder instance. If we were to use
> >> Deserializer we would also need a way to provide configurations.
> >>
> >> Thanks,
> >> Mickael
> >>
> >> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
> >> wrote:
> >> >
> >> > Dear all,
> >> >
> >> > We know that  3.8.0 KIP is already frozen, but this is a small KIP and
> >> we need to ship it to 3.8.0 so as to remove the deprecated scala
> interface
> >> from 4.0.
> >> >
> >> > Best,
> >> > Chia-Ping
> >> >
> >> > On 2024/05/22 14:05:16 Frank Yang wrote:
> >> > > Hi team,
> >> > >
> >> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
> >> kafka.serializer.Decoder from core module (scala) to tools module
> (java).
> >> > >
> >> > > Feedback and comments are welcome.
> >> > >
> >> > > KIP-1047:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> >> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> >> > >
> >> > > Thank you.
> >> > > PoAn
> >>
> >
>

[jira] [Resolved] (KAFKA-15041) Source Connector auto topic creation fails when topic is deleted and brokers don't support auto topic creation

[Sagar Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sagar Rao resolved KAFKA-15041.
---
Resolution: Won't Fix

For now, setting the config `producer.override.max.block.ms` at a connector 
config level  or `producer.max.block.ms` at a worker config level to a lower 
value should fix this value. The problem is that the default value for the 
above config is[ set to Long.MAX_VALUE 
|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L820]in
 the configs and when topics are deleted manually, there's really no signal 
that is received to indicate the same. We could add some heuristics like 
checking if a topic is present or not periodically and refreshing the cache, or 
check the source topic metrics to see if the records are just being buffered 
and not being sent but that's outside the scope of runtime.

> Source Connector auto topic creation fails when topic is deleted and brokers 
> don't support auto topic creation
> --
>
> Key: KAFKA-15041
> URL: https://issues.apache.org/jira/browse/KAFKA-15041
> Project: Kafka
>  Issue Type: Bug
>  Components: connect
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
>   allows the source connectors to create topics even when the broker doesn't 
> allow to do so. It does so by checking for every record if a topic needs to 
> be created 
> [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
>  To not always keep checking for topic presence via admin topics, it also 
> maintains a cache of the topics that it has created and doesn't create those 
> anymore. This helps to create topics when brokers don't support automatic 
> topic creation.
> However, lets say the topic gets created initially and later on gets deleted 
> while the connector is still running and the brokers don't support automatic 
> topic creation. For such cases, the connector has cached the topic it has 
> already created and wouldn't recreate it because the cache never updates and 
> since the broker doesn't support topic creation, the logs would just be full 
> of messages like 
>  
> {code:java}
> Error while fetching metadata with correlation id 3260 : 
> {connect-test=UNKNOWN_TOPIC_OR_PARTITION}{code}
>  
> This can become a problem on environments where brokers don't allow topic 
> creation. We need a way to refresh the topics cache for such cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Ivan Yurchenko]

Hi!

I had a look at the KIP-932, and it seems KafkaShareConsumer is to be 
configured the same way as the normal consumer using key-value props. As I 
understand correctly, no adaptation is needed for it to benefit from KIP-899?

Meanwhile, the PR [1] is open for review. If there are comments that require 
changes, we can address them in the PR or in case it's already merged, 
afterwards.

Best,
Ivan

[1] https://github.com/apache/kafka/pull/13277

On Thu, May 16, 2024, at 01:52, Jun Rao wrote:
> Hi, Ivan,
> 
> You are right. StreamsConfigs can take all existing consumer configs, with
> or without prefixes. So, we don't need to add the new config to
> StreamsConfig explicitly.
> 
> For KIP-932, it says for each new consumer config, we need to determine
> whether it should be added to ShareConsumer config too.
> 
> Thanks,
> 
> Jun
> 
> On Wed, May 15, 2024 at 12:16 PM Ivan Yurchenko  wrote:
> 
> > Hi Jun,
> >
> > Thank you for you comment. I was thinking that this
> > `metadata.recovery.strategy` could be passed to the relevant consumer in
> > streams using the `restore.consumer.` prefix. I that what you meant or I
> > misunderstand?
> > As for the KIP-932, I'll have a closer look.
> >
> > Ivan
> >
> >
> > On Wed, May 15, 2024, at 20:14, Jun Rao wrote:
> > > Hi, Ivan,
> > >
> > > Thanks for the KIP. +1
> > >
> > > Just a minor comment. Should we add metadata.recovery.strategy to the
> > > Streams and the newly introduced ShareConsumer (KIP-932) too?
> > >
> > > Jun
> > >
> > > On Wed, May 8, 2024 at 11:35 AM Manikumar 
> > wrote:
> > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (binding).
> > > >
> > > > On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim  > >
> > > > wrote:
> > > > >
> > > > > Hi Ivan,
> > > > > Thanks for the KIP this is a very nice feature to have.
> > > > > +1(non-binding)
> > > > > Omnia
> > > > > > On 15 Apr 2024, at 14:33, Andrew Schofield <
> > andrew_schofi...@live.com>
> > > > wrote:
> > > > > >
> > > > > > Thanks for the KIP
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Andrew
> > > > > >
> > > > > >> On 15 Apr 2024, at 14:16, Chris Egerton 
> > > > wrote:
> > > > > >>
> > > > > >> Hi Ivan,
> > > > > >>
> > > > > >> Thanks for the KIP. After the recent changes, this LGTM. +1
> > (binding)
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Chris
> > > > > >>
> > > > > >> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
> > > > ivan0yurche...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hello,
> > > > > >>>
> > > > > >>> The discussion [1] for KIP-899 [2] has been open for quite some
> > > > time. I'd
> > > > > >>> like to put the KIP up for a vote.
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Ivan
> > > > > >>>
> > > > > >>> [1]
> > https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
> > > > > >>> [2]
> > > > > >>>
> > > > > >>>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
> > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> 

[jira] [Created] (KAFKA-16818) Move event-processing tests from ConsumerNetworkThreadTest to ApplicationEventProcessorTest

[Kirk True (Jira)]

Kirk True created KAFKA-16818:
-

 Summary: Move event-processing tests from 
ConsumerNetworkThreadTest to ApplicationEventProcessorTest
 Key: KAFKA-16818
 URL: https://issues.apache.org/jira/browse/KAFKA-16818
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer, unit tests
Affects Versions: 3.8.0
Reporter: Kirk True
Assignee: Kirk True
 Fix For: 3.8.0


The {{ConsumerNetworkThreadTest}} currently has a number of tests which do the 
following:
 # Add event of type _T_ to the event queue
 # Call {{ConsumerNetworkThread.runOnce()}} to dequeue the events and call 
{{ApplicationEventProcessor.process()}}
 # Verify that the appropriate {{ApplicationEventProcessor}} process method was 
invoked for the event

Those types of tests should be moved to {{{}ApplicationEventProcessorTest{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Thanks for Josep's response

> We can add this to 3.8.0, but keep in mind the KIP is not voted yet (as
far
as I can see), so I would highly encourage to start the vote thread ASAP
and strat with the implementation right after.

sure. We will file a draft PR at the same time!

Josep Prat  於 2024年5月23日 週四 上午12:31寫道：

> Hi all,
>
> We can add this to 3.8.0, but keep in mind the KIP is not voted yet (as far
> as I can see), so I would highly encourage to start the vote thread ASAP
> and strat with the implementation right after.
>
> Best,
>
> -
> Josep Prat
> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Wed, May 22, 2024, 17:06 Chia-Ping Tsai  wrote:
>
> > > One issue I also noted is that some of the existing Decoder
> > implementations (StringDecoder for example) can accept configurations
> > but currently DumpLogSegments does not provide a way to pass any
> > configurations, it creates an empty VerifiableProperties object each
> > time it instantiates a Decoder instance. If we were to use
> > Deserializer we would also need a way to provide configurations.
> >
> > BTW, if the known bug gets fixed, we have to make new interface extend
> > `configurable`.
> >
> > Or we can just ignore the known issue as `DumpLogSegments` has no options
> > to take custom configs for `Decoder`. That allow the `Decoder` more
> simple
> >
> >
> > Chia-Ping Tsai  於 2024年5月22日 週三 下午10:58寫道：
> >
> > >
> > > Thanks for Mickael response!
> > >
> > > >I'm wondering whether we need to introduce a new Decoder interface and
> > > instead if we could reuse Deserializer. We could deprecate the
> > > key-decoder-class and value-decoder-class flags and introduce new
> > > flags like key-deserializer-class and value-deserializer-class. One
> > > benefit is that we already have many existing deserializer
> > > implementations. WDYT?
> > >
> > > I prefer to use different interface, since using the same interface
> > > (Deserializer) may obstruct us from enhancing the interface used by
> > > DumpLogSegments only in the future.
> > >
> > > > One issue I also noted is that some of the existing Decoder
> > > implementations (StringDecoder for example) can accept configurations
> > > but currently DumpLogSegments does not provide a way to pass any
> > > configurations, it creates an empty VerifiableProperties object each
> > > time it instantiates a Decoder instance. If we were to use
> > > Deserializer we would also need a way to provide configurations.
> > >
> > > yep, that is a known issue:
> > > https://issues.apache.org/jira/browse/KAFKA-12311
> > >
> > > We will file PR to fix it
> > >
> > > Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：
> > >
> > >> Hi,
> > >>
> > >> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
> > >> it would allow us to migrate this tool in 4.0.0. We're unfortunately
> > >> past the KIP deadline but maybe this is small enough to have an
> > >> exception.
> > >>
> > >> I'm wondering whether we need to introduce a new Decoder interface and
> > >> instead if we could reuse Deserializer. We could deprecate the
> > >> key-decoder-class and value-decoder-class flags and introduce new
> > >> flags like key-deserializer-class and value-deserializer-class. One
> > >> benefit is that we already have many existing deserializer
> > >> implementations. WDYT?
> > >>
> > >> One issue I also noted is that some of the existing Decoder
> > >> implementations (StringDecoder for example) can accept configurations
> > >> but currently DumpLogSegments does not provide a way to pass any
> > >> configurations, it creates an empty VerifiableProperties object each
> > >> time it instantiates a Decoder instance. If we were to use
> > >> Deserializer we would also need a way to provide configurations.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
> > >> wrote:
> > >> >
> > >> > Dear all,
> > >> >
> > >> > We know that  3.8.0 KIP is already frozen, but this is a small KIP
> and
> > >> we need to ship it to 3.8.0 so as to remove the deprecated scala
> > interface
> > >> from 4.0.
> > >> >
> > >> > Best,
> > >> > Chia-Ping
> > >> >
> > >> > On 2024/05/22 14:05:16 Frank Yang wrote:
> > >> > > Hi team,
> > >> > >
> > >> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
> > >> kafka.serializer.Decoder from core module (scala) to tools module
> > (java).
> > >> > >
> > >> > > Feedback and comments are welcome.
> > >> > >
> > >> > > KIP-1047:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> > >> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> > >> > >
> > >> > > Thank you.
> > >> > > PoAn
> > >>
> > >
> >
>

[jira] [Resolved] (KAFKA-12399) Deprecate Log4J Appender KIP-719

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-12399.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Deprecate Log4J Appender KIP-719
> 
>
> Key: KAFKA-12399
> URL: https://issues.apache.org/jira/browse/KAFKA-12399
> Project: Kafka
>  Issue Type: Improvement
>  Components: logging
>Reporter: Dongjin Lee
>Assignee: Mickael Maison
>Priority: Major
>  Labels: needs-kip
> Fix For: 3.8.0
>
>
> As a following job of KAFKA-9366, we have to entirely remove the log4j 1.2.7 
> dependency from the classpath by removing dependencies on log4j-appender.
> KIP-719: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Jun Rao]

Hi, Ivan,

Thanks for the reply. KafkaShareConsumer doesn't take all configuration
values from KafkaConsumer. So, we need to make a note that this new config
will be part of KafkaShareConsumer too.

Jun

On Wed, May 22, 2024 at 9:45 AM Ivan Yurchenko  wrote:

> Hi!
>
> I had a look at the KIP-932, and it seems KafkaShareConsumer is to be
> configured the same way as the normal consumer using key-value props. As I
> understand correctly, no adaptation is needed for it to benefit from
> KIP-899?
>
> Meanwhile, the PR [1] is open for review. If there are comments that
> require changes, we can address them in the PR or in case it's already
> merged, afterwards.
>
> Best,
> Ivan
>
> [1] https://github.com/apache/kafka/pull/13277
>
> On Thu, May 16, 2024, at 01:52, Jun Rao wrote:
> > Hi, Ivan,
> >
> > You are right. StreamsConfigs can take all existing consumer configs,
> with
> > or without prefixes. So, we don't need to add the new config to
> > StreamsConfig explicitly.
> >
> > For KIP-932, it says for each new consumer config, we need to determine
> > whether it should be added to ShareConsumer config too.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, May 15, 2024 at 12:16 PM Ivan Yurchenko  wrote:
> >
> > > Hi Jun,
> > >
> > > Thank you for you comment. I was thinking that this
> > > `metadata.recovery.strategy` could be passed to the relevant consumer
> in
> > > streams using the `restore.consumer.` prefix. I that what you meant or
> I
> > > misunderstand?
> > > As for the KIP-932, I'll have a closer look.
> > >
> > > Ivan
> > >
> > >
> > > On Wed, May 15, 2024, at 20:14, Jun Rao wrote:
> > > > Hi, Ivan,
> > > >
> > > > Thanks for the KIP. +1
> > > >
> > > > Just a minor comment. Should we add metadata.recovery.strategy to the
> > > > Streams and the newly introduced ShareConsumer (KIP-932) too?
> > > >
> > > > Jun
> > > >
> > > > On Wed, May 8, 2024 at 11:35 AM Manikumar  >
> > > wrote:
> > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > +1 (binding).
> > > > >
> > > > > On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Ivan,
> > > > > > Thanks for the KIP this is a very nice feature to have.
> > > > > > +1(non-binding)
> > > > > > Omnia
> > > > > > > On 15 Apr 2024, at 14:33, Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Thanks for the KIP
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Andrew
> > > > > > >
> > > > > > >> On 15 Apr 2024, at 14:16, Chris Egerton
> 
> > > > > wrote:
> > > > > > >>
> > > > > > >> Hi Ivan,
> > > > > > >>
> > > > > > >> Thanks for the KIP. After the recent changes, this LGTM. +1
> > > (binding)
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >>
> > > > > > >> Chris
> > > > > > >>
> > > > > > >> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
> > > > > ivan0yurche...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Hello,
> > > > > > >>>
> > > > > > >>> The discussion [1] for KIP-899 [2] has been open for quite
> some
> > > > > time. I'd
> > > > > > >>> like to put the KIP up for a vote.
> > > > > > >>>
> > > > > > >>> Best,
> > > > > > >>> Ivan
> > > > > > >>>
> > > > > > >>> [1]
> > > https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
> > > > > > >>> [2]
> > > > > > >>>
> > > > > > >>>
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
> > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Andrew Schofield]

Hi,
I’ll add an item to the plan for KIP-932 to do this for the KafkaShareConsumer.
My guess is that there will be actually nothing to do in practice because of
code in common with the KafkaConsumer, but definitely worth me checking it out.

Thanks,
Andrew

> On 22 May 2024, at 18:28, Jun Rao  wrote:
>
> Hi, Ivan,
>
> Thanks for the reply. KafkaShareConsumer doesn't take all configuration
> values from KafkaConsumer. So, we need to make a note that this new config
> will be part of KafkaShareConsumer too.
>
> Jun
>
> On Wed, May 22, 2024 at 9:45 AM Ivan Yurchenko  wrote:
>
>> Hi!
>>
>> I had a look at the KIP-932, and it seems KafkaShareConsumer is to be
>> configured the same way as the normal consumer using key-value props. As I
>> understand correctly, no adaptation is needed for it to benefit from
>> KIP-899?
>>
>> Meanwhile, the PR [1] is open for review. If there are comments that
>> require changes, we can address them in the PR or in case it's already
>> merged, afterwards.
>>
>> Best,
>> Ivan
>>
>> [1] https://github.com/apache/kafka/pull/13277
>>
>> On Thu, May 16, 2024, at 01:52, Jun Rao wrote:
>>> Hi, Ivan,
>>>
>>> You are right. StreamsConfigs can take all existing consumer configs,
>> with
>>> or without prefixes. So, we don't need to add the new config to
>>> StreamsConfig explicitly.
>>>
>>> For KIP-932, it says for each new consumer config, we need to determine
>>> whether it should be added to ShareConsumer config too.
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Wed, May 15, 2024 at 12:16 PM Ivan Yurchenko  wrote:
>>>
 Hi Jun,

 Thank you for you comment. I was thinking that this
 `metadata.recovery.strategy` could be passed to the relevant consumer
>> in
 streams using the `restore.consumer.` prefix. I that what you meant or
>> I
 misunderstand?
 As for the KIP-932, I'll have a closer look.

 Ivan


 On Wed, May 15, 2024, at 20:14, Jun Rao wrote:
> Hi, Ivan,
>
> Thanks for the KIP. +1
>
> Just a minor comment. Should we add metadata.recovery.strategy to the
> Streams and the newly introduced ShareConsumer (KIP-932) too?
>
> Jun
>
> On Wed, May 8, 2024 at 11:35 AM Manikumar >>
 wrote:
>
>> Thanks for the KIP.
>>
>> +1 (binding).
>>
>> On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim <
>> o.g.h.ibra...@gmail.com
>
>> wrote:
>>>
>>> Hi Ivan,
>>> Thanks for the KIP this is a very nice feature to have.
>>> +1(non-binding)
>>> Omnia
 On 15 Apr 2024, at 14:33, Andrew Schofield <
 andrew_schofi...@live.com>
>> wrote:

 Thanks for the KIP

 +1 (non-binding)

 Andrew

> On 15 Apr 2024, at 14:16, Chris Egerton
>> 
>> wrote:
>
> Hi Ivan,
>
> Thanks for the KIP. After the recent changes, this LGTM. +1
 (binding)
>
> Cheers,
>
> Chris
>
> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
> wrote:
>
>> Hello,
>>
>> The discussion [1] for KIP-899 [2] has been open for quite
>> some
>> time. I'd
>> like to put the KIP up for a vote.
>>
>> Best,
>> Ivan
>>
>> [1]
 https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
>> [2]
>>
>>
>>

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
>>

>>>
>>
>

>>>
>>

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

[Chris Egerton]

Hi Greg,

Hope you had a nice weekend! Gonna try to keep things concise.

Concluded points:

RE version recommenders, I agree it's likely that programmatic UIs will
already be able to handle dynamic configuration definitions, and the detail
about SMTs is a great point. I still anticipate some awkwardness with
connector versions, though: if the latest version supports some new
properties, then a user switches to an earlier version, a UI may respond by
wiping values for these properties. I guess we can bite the bullet, though.

RE double-dinging during preflight validation for invalid versions, I like
the analogy with login credentials. I'm convinced that the proposal in the
KIP is best 👍

Continued points:

RE failing on worker startup, sorry, I should be clearer: there is no _new_
justification for it that doesn't also apply to existing behavior. We
shouldn't diverge from existing behavior solely for this new case. An
alternative could be to change existing behavior to fail fast on any
invalid default converter configuration instead of just for invalid
versions, but I'd vote to just stick to existing behavior and not
complicate things, especially since no other part of the KIP requires this
change.

RE exposing the version property in the /connector-plugins//config
endpoint, the behavior is inconsistent across plugin types. Hitting the
endpoint for the FileStreamSinkConnector on version 3.7.0 yields a response
that includes, among other things, the "topics", "topics.regex", and
"errors.tolerance" properties. I see that we don't do this everywhere (the
examples you cite for SMT and converter properties are accurate), but IMO
it's worth including this information somewhere directly accessible without
having to provide a full connector config. FWIW I'd be fine with GET
/connector-plugins//versions as a first-class endpoint either
instead of or in addition to adding recommended values for all plugin
versions.

Thanks for your continued work on this KIP, and with the progress we're
making I'm optimistic about its chances of appearing in 4.0.0.

Cheers,

Chris

On Wed, May 15, 2024 at 1:22 PM Greg Harris 
wrote:

> Hey Chris,
>
> Thanks for your quick follow up.
>
> > But this risk is already present with
> > existing error cases, and I don't see anything that justifies changing
> > existing behavior with an invalid converter class, or diverging from it
> in
> > the case of invalid converter versions.
>
> The justification is to fail-fast, and prevent REST API users from
> receiving errors from bad configs that they didn't write, or maybe
> don't even know apply to them.
> Up until recently errors in these configurations surfaced as failures
> to create the connector, or failures to start, and you made them
> fail-fast during validation. I think this change is in the same
> spirit, pulling the validation further forward and not letting errors
> lie dormant.
>
> And to call back to your original concern about interrupting
> connectors that explicitly provide these configurations and don't use
> the worker configs: I expect that operators with a majority of these
> sorts of clients aren't going to be setting the worker .version
> properties, because it would have no effect on the majority of their
> connectors. They would be able to rely on backwards-compatibility and
> continue to ignore the class properties.
>
> > CLI
> > and programmatic UI developers will want to develop their own tooling
> > layers.
>
> This is a very compelling argument. Snehashis do you want to figure
> out a REST API design for this use-case?
>
> > Regarding the GET /connector-plugins//config endpoint, I was
> > thinking about the response for non-connector plugins, e.g.,
> > GET /connector-plugins/RegexRouter/config. Would a "version" property
> > appear with recommended values?
>
> I intended for the ".version" property to be like other framework
> configs (".class", ".type", ".predicate", ".negate") where they are
> inside the plugin namespace, but not part of the plugin config itself.
> Perhaps we can deviate from those configs because it would aid in
> discovering other valid `GET
> /connector-plugins//config?version=` calls, without calling
> `GET /connector-plugins?connectorsOnly=false`.
> I don't really feel strongly either way.
>
> > if a user changes the
> > connector version in, e.g., a dropdown menu, then the UI either has to
> > re-fetch the ConfigDef for the new version, or risk operating on stale
> > information
>
> This is a really interesting situation, thanks for finding that! This
> is already a footgun with transformations and predicates; Once you
> fill out a transformation class (which includes a recommender) you
> will then see all of the transformations configurations.
> I think this means that UI developers should have already developed
> the infrastructure for handling dynamic recommenders, or else they've
> had this bug since KIP-66 in 2017. It may require some manual
> attention to roll-out support for the ".ver

[jira] [Created] (KAFKA-16819) CoordinatorRequestManager seems to return 0ms during the coordinator discovery

[Philip Nee (Jira)]

Philip Nee created KAFKA-16819:
--

 Summary: CoordinatorRequestManager seems to return 0ms during the 
coordinator discovery
 Key: KAFKA-16819
 URL: https://issues.apache.org/jira/browse/KAFKA-16819
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


In KAFKA-15250 we discovered the ConsumerNetworkThread is looping without much 
backoff.  The in-flight check PR fixed a lot of it; however, during the 
coordinator discovery phase, CoordinatorRequestManager would keep on returning 
0 before the coordinator node was found.

 

The impact is minor but we should be expecting the coordinator manager to 
backoff until the exp backoff expired (so it should return around 100ms).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16820) Kafka Broker fails to connect to Kraft Controller with no DNS matching

[Arushi Helms (Jira)]

Arushi Helms created KAFKA-16820:


 Summary: Kafka Broker fails to connect to Kraft Controller with no 
DNS matching 
 Key: KAFKA-16820
 URL: https://issues.apache.org/jira/browse/KAFKA-16820
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.6.1, 3.7.0, 3.8.0
Reporter: Arushi Helms
 Attachments: Screenshot 2024-05-22 at 1.09.11 PM-1.png

 

We are migrating our Kafka cluster from zookeeper to Kraft mode. We are running 
individual brokers and controllers with TLS enabled and IPs are given for 
communication. 
TLS enabled setup works fine among the brokers and the certificate looks 
something like:
h5.  
{noformat}
Common Name: *.kafka.service.consul
Subject Alternative Names: *.kafka.service.consul, IP 
Address:10.87.171.84{noformat}

Note: The DNS name for the node does not match the CN but since we are using 
IPs as communication, we have provided IPs as SAN. 

Same with the controllers, IPs are given as SAN in the certificate. 

In the current setup I am running 3 brokers and 3 controllers. 

Relevant controller configurations from one of the controllers:

{{}}
{noformat}
KAFKA_CFG_PROCESS_ROLES=controller 
KAFKA_KRAFT_CLUSTER_ID=5kztjhJ4SxSu-kdiEYDUow
KAFKA_CFG_NODE_ID=6 
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=4@10.87.170.83:9097,5@10.87.170.9:9097,6@10.87.170.6:9097
 
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER 
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INSIDE_SSL
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SSL,INSIDE_SSL:SSL 
KAFKA_CFG_LISTENERS=CONTROLLER://10.87.170.6:9097{noformat}
{{}}

 

Relevant broker configuration from one of the brokers:

 
{noformat}
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INSIDE_SSL 
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=4@10.87.170.83:9097,5@10.87.170.9:9097,6@10.87.170.6:9097
 
KAFKA_CFG_PROCESS_ROLES=broker 
KAFKA_CFG_NODE_ID=3 
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE_SSL:SSL,OUTSIDE_SSL:SSL,CONTROLLER:SSL
 
KAFKA_CFG_LISTENERS=INSIDE_SSL://10.87.170.78:9093,OUTSIDE_SSL://10.87.170.78:9096
 
KAFKA_CFG_ADVERTISED_LISTENERS=INSIDE_SSL://10.87.170.78:9093,OUTSIDE_SSL://10.87.170.78:9096{noformat}
{{}}

 

ISSUE 1: 
With this setup Kafka broker is failing to connect to the controller, see the 
following error:
{noformat}
2024-05-22 17:53:46,413] ERROR 
[broker-2-to-controller-heartbeat-channel-manager]: Request 
BrokerRegistrationRequestData(brokerId=2, clusterId='5kztjhJ4SxSu-kdiEYDUow', 
incarnationId=7741fgH6T4SQqGsho8E6mw, listeners=[Listener(name='INSIDE_SSL', 
host='10.87.170.81', port=9093, securityProtocol=1), Listener(name='INSIDE', 
host='10.87.170.81', port=9094, securityProtocol=0), Listener(name='OUTSIDE', 
host='10.87.170.81', port=9092, securityProtocol=0), 
Listener(name='OUTSIDE_SSL', host='10.87.170.81', port=9096, 
securityProtocol=1)], features=[Feature(name='metadata.version', 
minSupportedVersion=1, maxSupportedVersion=19)], rack=null, 
isMigratingZkBroker=false, logDirs=[TJssfKDD-iBFYfIYCKOcew], 
previousBrokerEpoch=-1) failed due to authentication error with controller 
(kafka.server.NodeToControllerRequestThread)org.apache.kafka.common.errors.SslAuthenticationException:
 SSL handshake failedCaused by: javax.net.ssl.SSLHandshakeException: No subject 
alternative DNS name matching 
cp-internal-onecloud-kfkc1.node.cp-internal-onecloud.consul found.  at 
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)  at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) at 
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) at 
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
  at 
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
  at 
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
   at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
   at 
java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) 
 at 
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
  at 
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
  at 
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
 at 
java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
  at 
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
  at 
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
at 
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslT

[jira] [Created] (KAFKA-16821) Create a new interface to store member metadata

[Ritika Reddy (Jira)]

Ritika Reddy created KAFKA-16821:


 Summary: Create a new interface to store member metadata
 Key: KAFKA-16821
 URL: https://issues.apache.org/jira/browse/KAFKA-16821
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ritika Reddy
Assignee: Ritika Reddy
 Attachments: Screenshot 2024-05-14 at 11.03.10 AM.png

!Screenshot 2024-05-14 at 11.03.10 AM.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16822) Abstract consumer group in coordinator to share functionality with share group

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-16822:
-

 Summary: Abstract consumer group in coordinator to share 
functionality with share group
 Key: KAFKA-16822
 URL: https://issues.apache.org/jira/browse/KAFKA-16822
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

[Elxan Eminov]

Hey Mickael,
Just checking to see if you have any thoughts on this.
thanks!

On Thu, 11 Apr 2024 at 15:11, Elxan Eminov  wrote:

> Hi Mickael!
> Any thoughts on this?
> Thanks!
>
> On Wed, 3 Apr 2024 at 13:21, Elxan Eminov  wrote:
>
>> Hi Mickael,
>> Thanks for your response and apologies for a huge delay in mine.
>>
>> My thinking is that any partition could go stale if there are no records
>> being produced into it. If enough of such partitions are present and are
>> owned by a single MM task, an OOM could happen.
>>
>> Regarding the scenario where the TTL value is lower than the refresh
>> interval - I believe that this is an edge that we need to document and
>> prevent against, for example either failing to start on such a combination
>> or resorting to a default value that would satisfy the constraint and
>> logging an error.
>>
>> Thanks,
>> Elkhan
>>
>> On Thu, 8 Feb 2024 at 14:17, Mickael Maison 
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the updates.
>>> I'm wondering whether we really need the ttl eviction mechanism. The
>>> motivation is to "avoid storing stale LRO entries which can cause an
>>> eventual OOM error". How could it contain stake entries? I would
>>> expect its cache to only contain entries for partitions assigned to
>>> the task that owns it. Also what is the expected behavior if there's
>>> no available LRO in the cache? If we keep this mechanism what happens
>>> if its value is lower than
>>> replication.record.lag.metric.refresh.interval?
>>>
>>> Thanks,
>>> Mickael
>>>
>>> On Mon, Feb 5, 2024 at 5:23 PM Elxan Eminov 
>>> wrote:
>>> >
>>> > Hi Mickael!
>>> > Any further thoughts on this?
>>> >
>>> > Thanks,
>>> > Elkhan
>>> >
>>> > On Thu, 18 Jan 2024 at 11:53, Mickael Maison >> >
>>> > wrote:
>>> >
>>> > > Hi Elxan,
>>> > >
>>> > > Thanks for the updates.
>>> > >
>>> > > We used dots to separate words in configuration names, so I think
>>> > > replication.offset.lag.metric.last-replicated-offset.ttl should be
>>> > > named replication.offset.lag.metric.last.replicated.offset.ttl
>>> > > instead.
>>> > >
>>> > > About the names of the metrics, fair enough if you prefer keeping the
>>> > > replication prefix. Out of the alternatives you mentioned, I think I
>>> > > prefer replication-record-lag. I think the metrics and configuration
>>> > > names should match too. Let's see what the others think about it.
>>> > >
>>> > > Thanks,
>>> > > Mickael
>>> > >
>>> > > On Mon, Jan 15, 2024 at 9:50 PM Elxan Eminov <
>>> elxanemino...@gmail.com>
>>> > > wrote:
>>> > > >
>>> > > > Apologies, forgot to reply on your last comment about the metric
>>> name.
>>> > > > I believe both replication-lag and record-lag are a little too
>>> abstract -
>>> > > > what do you think about either leaving it as
>>> replication-offset-lag or
>>> > > > renaming to replication-record-lag?
>>> > > >
>>> > > > Thanks
>>> > > >
>>> > > > On Wed, 10 Jan 2024 at 15:31, Mickael Maison <
>>> mickael.mai...@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi Elxan,
>>> > > > >
>>> > > > > Thanks for the KIP, it looks like a useful addition.
>>> > > > >
>>> > > > > Can you add to the KIP the default value you propose for
>>> > > > > replication.lag.metric.refresh.interval? In MirrorMaker most
>>> interval
>>> > > > > configs can be set to -1 to disable them, will it be the case
>>> for this
>>> > > > > new feature or will this setting only accept positive values?
>>> > > > > I also wonder if replication-lag, or record-lag would be clearer
>>> names
>>> > > > > instead of replication-offset-lag, WDYT?
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Mickael
>>> > > > >
>>> > > > > On Wed, Jan 3, 2024 at 6:15 PM Elxan Eminov <
>>> elxanemino...@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > Hi all,
>>> > > > > > Here is the vote thread:
>>> > > > > >
>>> https://lists.apache.org/thread/ftlnolcrh858dry89sjg06mdcdj9mrqv
>>> > > > > >
>>> > > > > > Cheers!
>>> > > > > >
>>> > > > > > On Wed, 27 Dec 2023 at 11:23, Elxan Eminov <
>>> elxanemino...@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi all,
>>> > > > > > > I've updated the KIP with the details we discussed in this
>>> thread.
>>> > > > > > > I'll call in a vote after the holidays if everything looks
>>> good.
>>> > > > > > > Thanks!
>>> > > > > > >
>>> > > > > > > On Sat, 26 Aug 2023 at 15:49, Elxan Eminov <
>>> > > elxanemino...@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > >> Relatively minor change with a new metric for MM2
>>> > > > > > >>
>>> > > > > > >>
>>> > > > >
>>> > >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-971%3A+Expose+replication-offset-lag+MirrorMaker2+metric
>>> > > > > > >>
>>> > > > > > >
>>> > > > >
>>> > >
>>>
>>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2926

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

[Greg Harris]

Hey Chris,

Thanks for your comments, and I'm glad that it seems like we're
aligning on the vision here.

> An
> alternative could be to change existing behavior to fail fast on any
> invalid default converter configuration instead of just for invalid
> versions

I suppose if this is landing in 4.0, we have the opportunity to break
compatibility and strictly validate the worker class configs, and
could have a new consistent behavior instead of inconsistent but
backwards-compatible behavior.

> no other part of the KIP requires this change.

This is correct, and a compelling argument. I'm fine leaving the
strict worker validation off of this configuration to potentially be
added later. If a KIP was raised to perform strict validation of the
worker config, it would include the version config, and can address
backwards compatibility for both configs together.

> RE exposing the version property in the /connector-plugins//config
> endpoint, the behavior is inconsistent across plugin types.

Yes it is, and that started in a bugfix, once people started using
this endpoint: https://issues.apache.org/jira/browse/KAFKA-14843 . I
wasn't planning on converging the behavior in this KIP, as I
considered it out-of-scope, and was going to follow the current
behavior. To summarize:

GET /connector-plugins//config will emit:
* connector.class (already implemented)
* connector.version (new)
* key.converter (already implemented)
* key.converter.version (new)
* value.converter (already implemented)
* value.converter.version (new)
* header.converter (already implemented)
* header.converter.version (new)
But will NOT emit:
* transforms..type
* transforms..version
* transforms..predicate
* predicates..type
* predicates..version
* predicates..negate

GET /connector-plugins//config will NOT emit:
* "" (there's not even a ".class" prefix!)
* version

GET /connector-plugins//config will NOT emit:
* type
* version
* predicate

GET /connector-plugins//config will NOT emit:
* type
* version
* negate

Do you want the converter, transform, and predicate endpoints changed?
Do you want just "version", or do you want all of the prefixed configs
including "type", "predicate" and "negate"? How would you want to
handle the converters?
And when I say the configs are "not part of the plugin config itself"
I mean that saying that GET
/connector-plugins/Flatten$Key/config?version=3.8.0 has a "version"
config that must be "3.8.0" is a little bit nonsense, as the version
is already specified.

> IMO
> it's worth including this information somewhere directly accessible without
> having to provide a full connector config. FWIW I'd be fine with GET
> /connector-plugins//versions as a first-class endpoint

You don't have to provide a configuration to call GET
/connector-plugins?connectorsOnly=false , is that endpoint not close
enough to what you have in mind? See also the Rejected Alternative
"Adding new REST API endpoints"

If you're calling /connector-plugins//config, you know the
name of a plugin right? That either comes from out-of-band knowledge,
validating a connector config, or calling GET
/connector-plugins?connectorsOnly=false.
* If you have out-of-band knowledge of plugin classes, perhaps you
have out-of-band knowledge of versions too.
* If you've just validated a connector config, there should be an
accompanying "version" field there with an accurate default value and
recommenders.
* If you've called GET /connector-plugins?connectorsOnly=false, that
endpoint includes version information.

Thanks,
Greg

On Wed, May 22, 2024 at 11:05 AM Chris Egerton  wrote:
>
> Hi Greg,
>
> Hope you had a nice weekend! Gonna try to keep things concise.
>
> Concluded points:
>
> RE version recommenders, I agree it's likely that programmatic UIs will
> already be able to handle dynamic configuration definitions, and the detail
> about SMTs is a great point. I still anticipate some awkwardness with
> connector versions, though: if the latest version supports some new
> properties, then a user switches to an earlier version, a UI may respond by
> wiping values for these properties. I guess we can bite the bullet, though.
>
> RE double-dinging during preflight validation for invalid versions, I like
> the analogy with login credentials. I'm convinced that the proposal in the
> KIP is best 👍
>
> Continued points:
>
> RE failing on worker startup, sorry, I should be clearer: there is no _new_
> justification for it that doesn't also apply to existing behavior. We
> shouldn't diverge from existing behavior solely for this new case. An
> alternative could be to change existing behavior to fail fast on any
> invalid default converter configuration instead of just for invalid
> versions, but I'd vote to just stick to existing behavior and not
> complicate things, especially since no other part of the KIP requires this
> change.
>
> RE exposing the version property in the /connector-plugins//config
> endpoint, the behavior is inconsistent across plugin types. Hitting the

Re: [DISCUSS] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

[Sophie Blee-Goldman]

Thanks for the KIP!

I'm not against adding this as a config for this per se, but if this is
causing you trouble right now you should be able to disable it via log4j
configuration so you don't need to wait for a fix in Kafka Streams itself.
Putting something like this in your log4j will shut off the offending log:

log4j.logger.org.apache.kafka.streams.processor.internals.StreamThread=WARN

On Wed, May 22, 2024 at 6:46 AM jiang dou  wrote:

> Hi
>
>
> I would like to propose a change in the kafka-stream summary log。
>
> Now the summary of stream-tread is record every two minutes, and not
> support close  or update intervals.
>
> When the kafka  is running， this is absolutely unnecessary and even harmful
> since it fills the logs and thus storage space with unwanted and useless
> data.
>
> I propose adding a configuration to control the output interval or disable
> it
>
> KIP：
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1049%3A+Add+config+log.summary.interval.ms+to+Kafka+Streams
>

[jira] [Created] (KAFKA-16823) Extract LegacyConsumer-specific unit tests from generic KafkaConsumerTest

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16823:
--

 Summary: Extract LegacyConsumer-specific unit tests from generic 
KafkaConsumerTest 
 Key: KAFKA-16823
 URL: https://issues.apache.org/jira/browse/KAFKA-16823
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans


Currently the KafkaConsumerTest file contains unit tests that apply to both 
consumer implementations, but also tests that apply to the legacy consumer 
only. We should consider splitting the tests that apply to the legacy only into 
their own LegacyConsumerTest file (aligning with the existing 
AsyncKafkaConsumerTest). End result would be: 

KafkaConsumerTest -> unit tests that apply to both consumers. 

LegacyKafkaConsumerTest -> unit tests that apply only to the 
LegacyKafkaConsumer, either because of the logic they test, or the way they are 
written (file to be created with this task)

AsyncKafkaConsumerTest -> unit tests that apply only to the AsyncKafkaConsumer 
(this file already exist)

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16824) Utils.getHost and Utils.getPort do not catch a lot of invalid host and ports

[Jira]

José Armando García Sancio created KAFKA-16824:
--

 Summary: Utils.getHost and Utils.getPort do not catch a lot of 
invalid host and ports
 Key: KAFKA-16824
 URL: https://issues.apache.org/jira/browse/KAFKA-16824
 Project: Kafka
  Issue Type: Bug
Reporter: José Armando García Sancio


For example it is not able to detect at least this malformed hosts and ports:
 # ho(st:9092
 # host:-92



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16160) AsyncKafkaConsumer is trying to connect to a disconnected node in a tight loop

[Philip Nee (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Nee resolved KAFKA-16160.

Resolution: Cannot Reproduce

> AsyncKafkaConsumer is trying to connect to a disconnected node in a tight loop
> --
>
> Key: KAFKA-16160
> URL: https://issues.apache.org/jira/browse/KAFKA-16160
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Phuc Hong Tran
>Priority: Major
>  Labels: consumer-threading-refactor
> Fix For: 3.8.0
>
>
> Observing some excessive logging running AsyncKafkaConsumer and observing 
> excessive logging of :
> {code:java}
> 1271 [2024-01-15 09:43:36,627] DEBUG [Consumer clientId=console-consumer, 
> groupId=concurrent_consumer] Node is not ready, handle the request in the 
> next event loop: node=worker4:9092 (id: 2147483644 rack: null), 
> request=UnsentRequest{requestBuil     
> der=ConsumerGroupHeartbeatRequestData(groupId='concurrent_consumer', 
> memberId='laIqS789StuhXFpTwjh6hA', memberEpoch=1, instanceId=null, 
> rackId=null, rebalanceTimeoutMs=30, subscribedTopicNames=[output-topic], 
> serverAssignor=null, topicP     
> artitions=[TopicPartitions(topicId=I5P5lIXvR1Cjc8hfoJg5bg, partitions=[0])]), 
> handler=org.apache.kafka.clients.consumer.internals.NetworkClientDelegate$FutureCompletionHandler@918925b,
>  node=Optional[worker4:9092 (id: 2147483644 rack: null)]     , 
> timer=org.apache.kafka.common.utils.Timer@55ed4733} 
> (org.apache.kafka.clients.consumer.internals.NetworkClientDelegate) {code}
> This seems to be triggered by a tight poll loop of the network thread.  The 
> right thing to do is to backoff a bit for that given node and retry later.
> This should be a blocker for 3.8 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16825) CVE vulnerabilities in Jetty and netty

[mooner (Jira)]

mooner created KAFKA-16825:
--

 Summary: CVE vulnerabilities in Jetty and netty
 Key: KAFKA-16825
 URL: https://issues.apache.org/jira/browse/KAFKA-16825
 Project: Kafka
  Issue Type: Task
Affects Versions: 3.7.0
Reporter: mooner


There is a vulnerability (CVE-2024-29025) in the passive dependency software 
Netty used by Kafka, which has been fixed in version 4.1.108.Final.

There is also a vulnerability (CVE-2024-22201) in the passive dependency 
software Jetty, which has been fixed in version 9.4.54.v20240208.

When will Kafka upgrade the versions of Netty and Jetty to fix these two 
vulnerabilities?

Reference website:

https://nvd.nist.gov/vuln/detail/CVE-2024-29025

https://nvd.nist.gov/vuln/detail/CVE-2024-22201



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16793) Heartbeat API for upgrading ConsumerGroup

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16793.
-
Fix Version/s: 3.8.0
 Reviewer: David Jacot
   Resolution: Fixed

> Heartbeat API for upgrading ConsumerGroup
> -
>
> Key: KAFKA-16793
> URL: https://issues.apache.org/jira/browse/KAFKA-16793
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)