SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

[Hamer, Bart]

Hi,

One of our Quarkus applications connects to Kafka topics using the 
kafka-clients library. Quarkus provides support for Apache Kafka through the 
SmallRye Reactive Messaging 
framework. According to the SmallRye documentation the bootstrap.servers 
property in the application.properties file should contain a comma separated 
list (see the Configuration Reference section on this page: 
https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.4/kafka/kafka.html).
 With kafka-clients v3.7.1 this worked fine, but after upgrading to v3.8.0 
starting the application fails with this exception:

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in 
bootstrap.servers: kafka://test1:9094,kafka://test2:9094

This is caused by the changed regex pattern named HOST_PORT_PATTERN in the 
org.apache.kafka.common.utils.Utils class, which no longer accepts a comma 
separated list.  Could you please look into this?
Thanks!


Met vriendelijke groeten / Kind regards / Mit freundlichen Grüßen,



Bart Hamer

Software Specialist

BTO-SM-BAM



Aanwezig: ma/di/wo/do







M

+31 (0)6 46 94 13 75

E

bart.ha...@tennet.eu

I

www.tennet.eu



[cid:c1e674e6-6276-4ebd-a18b-737c05a2d22f]

TenneT TSO B.V.
Utrechtseweg 310
Arnhem
Postbus 718
6800 AS Arnhem
Nederland



Handelsregister: Arnhem  09155985

Denk aan het milieu. Print dit bericht alleen als het noodzakelijk is.

Re: SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

[Josep Prat]

Hi Bart,

I'm not an expert in this area, but looking at the change [1] in question
and the usage of this regex, I'd say it was never the intention to accept
comma separated hosts with ports (and optional protocol).
The only 2 usages of HOST_PORT_PATTERN can be found in lines 538 [2] and
548 [3] in the same Utils class. Also looking at the tests it seems the
expectation has always been to parse a single host with a port pattern
(with optional protocol).

[1]: https://github.com/apache/kafka/pull/16048
[2]:
https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L538
[3]:
https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L548
[4]:
https://github.com/apache/kafka/blob/3.8/clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java#L104

Best,

On Wed, Sep 11, 2024 at 4:23 PM Hamer, Bart  wrote:

> Hi,
>
> One of our Quarkus applications connects to Kafka topics using the
> *kafka-clients* library. Quarkus provides support for Apache Kafka
> through the SmallRye Reactive Messaging
>  framework. According
> to the SmallRye documentation the *bootstrap.servers* property in the
> application.properties file should contain a comma separated list (see the 
> *Configuration
> Reference* section on this page:
> https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.4/kafka/kafka.html).
> With kafka-clients v3.7.1 this worked fine, but after upgrading to v3.8.0
> starting the application fails with this exception:
>
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
> Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in
> bootstrap.servers: kafka://test1:9094,kafka://test2:9094
>
> This is caused by the changed regex pattern named HOST_PORT_PATTERN in the
> *org.apache.kafka.common.utils.Utils* class, which no longer accepts a
> comma separated list.  Could you please look into this?
> Thanks!
>
> Met vriendelijke groeten / Kind regards / Mit freundlichen Grüßen,
>
>
>
> *Bart Hamer*
>
> Software Specialist
>
> BTO-SM-BAM
>
>
>
> *Aanwezig: ma/di/wo/do*
>
>
>
>
>
>
>
> *M*
>
> +31 (0)6 46 94 13 75
>
> *E*
>
> *bart.ha...@tennet.eu *
>
> *I*
>
> *www.tennet.eu  *
>
>
>
> TenneT TSO B.V.
> Utrechtseweg 310
> Arnhem
> Postbus 718
> 6800 AS Arnhem
> Nederland
>
>
>
> *Handelsregister:* Arnhem  09155985
>
> Denk aan het milieu. Print dit bericht alleen als het noodzakelijk is.
>
>
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [ANNOUNCE] New committer: Jeff Kim

[Philip Nee]

congrats!

On Tue, Sep 10, 2024 at 5:02 PM Kirk True  wrote:

> Congratulations Jeff!
>
> > On Sep 9, 2024, at 6:25 PM, Matthias J. Sax  wrote:
> >
> > Congrats!
> >
> > On 9/9/24 12:34 PM, José Armando García Sancio wrote:
> >> Congratulations Jeff!
> >> On Mon, Sep 9, 2024 at 11:45 AM Justine Olshan
> >>  wrote:
> >>>
> >>> Congratulations Jeff!
> >>>
> >>> On Mon, Sep 9, 2024 at 8:33 AM Satish Duggana <
> satish.dugg...@gmail.com>
> >>> wrote:
> >>>
>  Congratulations Jeff!
> 
>  On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna 
> wrote:
> >
> > Congrats! Well deserved!
> >
> > Best,
> > Bruno
> >
> >
> >
> > On 9/9/24 2:28 PM, Bill Bejeck wrote:
> >> Congrats Jeff!!
> >>
> >> On Mon, Sep 9, 2024 at 7:50 AM Lianet M. 
> wrote:
> >>
> >>> Congrats Jeff!!!
> >>>
> >>> On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton <
> fearthecel...@gmail.com
> >
> >>> wrote:
> >>>
>  Congrats!
> 
>  On Mon, Sep 9, 2024, 06:36 Rajini Sivaram <
> rajinisiva...@gmail.com>
> >>> wrote:
> 
> > Congratulations, Jeff!
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, Sep 9, 2024 at 10:49 AM Luke Chen 
>  wrote:
> >
> >> Congrats, Jeff!
> >>
> >> On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
> >>  wrote:
> >>
> >>> Congrats Jeff!
> >>>
> >>> On Mon, Sep 9, 2024, 11:02 Yash Mayya 
> >>> wrote:
> >>>
>  Congratulations Jeff!
> 
>  On Mon, 9 Sept, 2024, 12:13 David Jacot, 
> >>> wrote:
> 
> > Hi all,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka
> > committer,
>  Jeff
> > Kim.
> >
> > Jeff has been a Kafka contributor since May 2020. In addition
> >>> to
> >> being
> > a regular contributor and reviewer, he has made significant
> > contributions to the next generation of the consumer
> rebalance
> > protocol (KIP-848) and to the new group coordinator. He
> >>> authored
> > KIP-915 which improved how coordinators can be downgraded. He
>  also
> > contributed multiple fixes/improvements to the fetch from
>  follower
> > feature.
> >
> > Congratulations, Jeff!
> >
> > Thanks,
> > David (on behalf of the Apache Kafka PMC)
> >
> 
> >>>
> >>
> >
> 
> >>>
> >>
> 
>
>

Re: SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

[James Olsen]

Looking at 
org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(List, 
ClientDnsLookup) shows that error message is referring to a singular url from 
the list.  So the issue is earlier in the chain - the list is not being decoded 
from the CSV.

> On 12 Sep 2024, at 03:03, Josep Prat  wrote:
> 
> Hi Bart,
> 
> I'm not an expert in this area, but looking at the change [1] in question
> and the usage of this regex, I'd say it was never the intention to accept
> comma separated hosts with ports (and optional protocol).
> The only 2 usages of HOST_PORT_PATTERN can be found in lines 538 [2] and
> 548 [3] in the same Utils class. Also looking at the tests it seems the
> expectation has always been to parse a single host with a port pattern
> (with optional protocol).
> 
> [1]: https://github.com/apache/kafka/pull/16048
> [2]:
> https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L538
> [3]:
> https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L548
> [4]:
> https://github.com/apache/kafka/blob/3.8/clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java#L104
> 
> Best,
> 
> On Wed, Sep 11, 2024 at 4:23 PM Hamer, Bart  wrote:
> 
>> Hi,
>> 
>> One of our Quarkus applications connects to Kafka topics using the
>> *kafka-clients* library. Quarkus provides support for Apache Kafka
>> through the SmallRye Reactive Messaging
>>  framework. According
>> to the SmallRye documentation the *bootstrap.servers* property in the
>> application.properties file should contain a comma separated list (see the 
>> *Configuration
>> Reference* section on this page:
>> https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.4/kafka/kafka.html).
>> With kafka-clients v3.7.1 this worked fine, but after upgrading to v3.8.0
>> starting the application fails with this exception:
>> 
>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>> Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in
>> bootstrap.servers: kafka://test1:9094,kafka://test2:9094
>> 
>> This is caused by the changed regex pattern named HOST_PORT_PATTERN in the
>> *org.apache.kafka.common.utils.Utils* class, which no longer accepts a
>> comma separated list.  Could you please look into this?
>> Thanks!
>> 
>> Met vriendelijke groeten / Kind regards / Mit freundlichen Grüßen,
>> 
>> 
>> 
>> *Bart Hamer*
>> 
>> Software Specialist
>> 
>> BTO-SM-BAM
>> 
>> 
>> 
>> *Aanwezig: ma/di/wo/do*
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> *M*
>> 
>> +31 (0)6 46 94 13 75
>> 
>> *E*
>> 
>> *bart.ha...@tennet.eu *
>> 
>> *I*
>> 
>> *www.tennet.eu  *
>> 
>> 
>> 
>> TenneT TSO B.V.
>> Utrechtseweg 310
>> Arnhem
>> Postbus 718
>> 6800 AS Arnhem
>> Nederland
>> 
>> 
>> 
>> *Handelsregister:* Arnhem  09155985
>> 
>> Denk aan het milieu. Print dit bericht alleen als het noodzakelijk is.
>> 
>> 
>> 
> 
> 
> -- 
> [image: Aiven] 
> 
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>     
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: Facing issue while producing

[Vikram Singh]

Hello Akash,

I am getting below error logs sometimes when a kafka producer is going to
produce some messages to the topic.

*Logs:-*
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not
the current leader. For requests intended for any replica, this error
indicates that the broker is not a replica of the topic partition

`Error :  org.apache.kafka.common.errors.OutOfOrderSequenceException: The
broker received an out of order sequence number

On Fri, Aug 30, 2024 at 9:04 AM Vikram Singh 
wrote:

> Hello Akash,
>
> I will try by changing compression type and test if the issue persists,
> but for now the issue is resolved I guess. I just changed
> replication_factor of the topic to 3. It was like a 3-node kafka cluster
> and replication_factor was configured 2 for the topic.
>
> On Wed, Aug 28, 2024 at 2:55 PM Akash Jain 
> wrote:
>
>> Also what is the version of Java you are running?
>>
>> On Wed, Aug 28, 2024 at 11:44 AM Akash Jain 
>> wrote:
>>
>> > Hi Vikram, please share some more details:
>> >
>> >1. producer version
>> >2. broker version
>> >3. broker side config you mentioned is "snappy", can you check what
>> is
>> >it on topic level as well
>> >4. do you face this issue when you use any other algorithm instead of
>> >snappy, say lz4? Try with topic compression.type set to producer and
>> >producer compression.type to anything other than snappy
>> >5. Do you face this issue when you disable compression?
>> >6. You also mention that you are facing this issue randomly, can you
>> >elaborate more? Like you have different producer versions - and a
>> >specific producer shows this problem? Or does a specific broker show
>> this
>> >problem? Have you observed any pattern at all?
>> >
>> > Do you see these exceptions in broker? I guess yes. Broker will
>> decompress
>> > the messages. It is likely because the version of snappy that is used in
>> > producer is 'far away' from the version of snappy in broker.
>> >
>> > On Tue, Aug 27, 2024 at 8:00 AM Vikram Singh
>> >  wrote:
>> >
>> >> Hello Akash,
>> >>
>> >> It's the same on the broker side as well.
>> >>
>> >> *Broker side config:-* compression.type=snappy
>> >>
>> >> On Mon, Aug 26, 2024 at 9:30 PM Akash Jain 
>> >> wrote:
>> >>
>> >> > And what is it on broker?
>> >> >
>> >> > On Sunday, August 25, 2024, Vikram Singh > >> > .invalid>
>> >> > wrote:
>> >> >
>> >> > > Hello Akash,
>> >> > >
>> >> > >
>> >> > > Yes, i am using compression type in producer side configuration as
>> >> > > mentioned below,
>> >> > >
>> >> > > And the compression type I am using is *snappy*.
>> >> > >
>> >> > >
>> >> > > *Compression on producer :-*
>> >> > >
>> >> > > producerProperties.put("compression.type",CloudKafkaProducerConfig.
>> >> > > COMPRESSION_TYPE);
>> >> > >
>> >> > > On Fri, Aug 23, 2024 at 5:56 PM Akash Jain <
>> akashjain0...@gmail.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Vikram. Can you share you code snippet? Are you using
>> >> compression on
>> >> > > > producer/broker side?
>> >> > > >
>> >> > > > On Friday, August 23, 2024, Vikram Singh <
>> >> vikram.si...@nciportal.com
>> >> > > > .invalid>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Hello,
>> >> > > > >
>> >> > > > > I am facing issues while producing messages on kafka topics. I
>> am
>> >> > > facing
>> >> > > > > this issue randomly. Please help me by referring to the logs
>> >> below.
>> >> > > > >
>> >> > > > > Logs :-
>> >> > > > > 1). ERROR [ReplicaManager broker=0] Error processing append
>> >> operation
>> >> > > on
>> >> > > > > partition MHM_CLZ_COM-AWS_123-7 (kafka.server.ReplicaManager)
>> >> > > > >
>> >> > > > > org.apache.kafka.common.KafkaException: Failed to decompress
>> >> record
>> >> > > > stream
>> >> > > > >
>> >> > > > > 2). Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>