Re: Consumer failure after rolling Broker upgrade

[Luke Chen]

Hi James,

I've filed a bug in JIRA: KAFKA-13563
.
I'll investigate this issue.

Thank you.
Luke

On Wed, Dec 22, 2021 at 2:49 AM James Olsen  wrote:

> This failure occurred again during this month's rolling OS security
> updates to the Brokers (no change to Broker version).  I have also been
> able to reproduce it locally with the following process:
>
> 1. Start a 3 Broker cluster with a Topic having Replicas=3.
> 2. Start a Client with Producer and Consumer communicating over the Topic.
> 3. Stop the Broker that is acting as the Group Coordinator.
> 4. Observe successful Rediscovery of new Group Coordinator.
> 5. Restart the stopped Broker.
> 6. Stop the Broker that became the new Group Coordinator at step 4.
> 7. Observe "Rediscovery will be attempted" message but no "Discovered
> group coordinator" message.
>
> In short, Group Coordinator Rediscovery only works for the first Broker
> failover not any subsequent failover.
>
> I conducted tests using 2.7.1 servers.  The issue occurs with 2.7.1 and
> 2.7.2 Clients.  The issue does not occur with 2.5.1 and 2.7.0 Clients.
> This make me suspect that
> https://issues.apache.org/jira/browse/KAFKA-10793 introduced this issue.
>
> Regards, James.
>
> On 24/11/2021, at 14:35, James Olsen  wrote:
>
> Luke,
>
> We did not upgrade to resolve the issue.  We simply restarted the failing
> clients.
>
> Regards, James.
>
> On 23/11/2021, at 16:10, Luke Chen  wrote:
>
> Hi James,
> > Bouncing the clients resolved the issue
> Could you please describe which version you upgrade to, to resolve this
> issue? That should also help other users encountering the same issue.
>
> And the code snippet you listed, existed since 2018, I don't think there
> is any problem there.
> Maybe there are bugs existed in other places, and got fixed indirectly.
>
> Thank you.
> Luke
>
> On Tue, Nov 23, 2021 at 10:27 AM James Olsen  wrote:
>
>> We had a 2.5.1 Broker/Client system running for some time with regular
>> rolling OS upgrades to the Brokers without any problems.  A while ago we
>> upgraded both Broker and Clients to 2.7.1 and now on the first rolling OS
>> upgrade to the 2.7.1 Brokers we encountered some Consumer issues.  We have
>> a 3 Broker setup with min-ISRs configured to avoid any outage.
>>
>> So maybe we just got lucky 6 times in a row with the 2.5.1 or maybe there
>> is an issue with the 2.7.1.
>>
>> The observable symptom is a continuous stream of "The coordinator is not
>> available" messages when trying to commit offsets.  It starts with the
>> usual messages you might expect during a rolling upgrade...
>>
>> 2021-11-22 04:41:25,269 WARN
>> [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator]
>> 'pool-7-thread-132' [Consumer clientId=consumer-MyService-group-58,
>> groupId=MyService-group] Offset commit failed on partition MyTopic-0 at
>> offset 866799313: The coordinator is loading and hence can't process
>> requests.
>>
>> ... then 5 minutes of all OK, then ...
>>
>> 2021-11-22 04:46:33,258 WARN
>> [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator]
>> 'pool-7-thread-132' [Consumer clientId=consumer-MyService-group-58,
>> groupId=MyService-group] Offset commit failed on partition MyTopic-0 at
>> offset 866803953: This is not the correct coordinator.
>>
>> 2021-11-22 04:46:33,258 INFO
>> [org.apache.kafka.clients.consumer.internals.AbstractCoordinator]
>> 'pool-7-thread-132' [Consumer clientId=consumer-MyService-group-58,
>> groupId=MyService-group] Group coordinator b-2.xxx.com:9094<
>> http://b-2.xxx.com:9094> (id: 2147483645 rack: null) is unavailable or
>> invalid due to cause: error response NOT_COORDINATOR.isDisconnected: false.
>> Rediscovery will be attempted.
>>
>> 2021-11-22 04:46:33,258 WARN  [xxx.KafkaConsumerRunner]
>> 'pool-7-thread-132' Offset commit with offsets
>> {MyTopic-0=OffsetAndMetadata{offset=866803953, leaderEpoch=null,
>> metadata=''}} failed:
>> org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
>> commit failed with a retriable exception. You should retry committing the
>> latest consumed offsets.
>> Caused by: org.apache.kafka.common.errors.NotCoordinatorException: This
>> is not the correct coordinator.
>>
>> ... then the following message for every subsequent attempt to commit
>> offsets ...
>>
>> 2021-11-22 04:46:33,284 WARN  [xxx.KafkaConsumerRunner]
>> 'pool-7-thread-132' Offset commit with offsets
>> {MyTopic-0=OffsetAndMetadata{offset=866803954, leaderEpoch=82,
>> metadata=''}, MyOtherTopic-0=OffsetAndMetadata{offset=12654756,
>> leaderEpoch=79, metadata=''}} failed:
>> org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset
>> commit failed with a retriable exception. You should retry committing the
>> latest consumed offsets.
>> Caused by:
>> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
>> coordinator is not available.
>>
>> In the above example we are doing manual async-commits but we also had

Log4j 2.x preview for Kafka

[Deepak Jain]

Hi Luke,

We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the Log4j 
vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104 and CVE-2021-45105, 
we are waiting for kafka to upgrade to Log4j 2.17. However, we came across 
following link in which there is a preview for the same.

http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/

Please let us know if it's safe and stable to upgrade our prod env with the 
preview or do we wait for Kafka official release (Log4j 2.x support with Java 
8) for the same.

Thanks in advance.

Regards,
Deepak

Re: Log4j 2.x preview for Kafka

[Israel Ekpo]

Currently, the core Apache Kafka components do not have any dependencies on
log4j2

There may be Kafka connectors that use log4j2  so you would need to check
with your connector vendors to see if this applies to those connectors.

If you do not use Kafka connect, then this may not apply to you.

Here is the official announcement from the Kafka project on this issue

https://kafka.apache.org/cve-list

If you are using non-upstream Kafka distro that includes log4j2, then check
with that vendor for additional information

I hope this helps

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://izzyacademy.com/


On Wed, Dec 22, 2021 at 10:58 AM Deepak Jain <
deepak.j...@cumulus-systems.com> wrote:

> Hi Luke,
>
> We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the
> Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104 and
> CVE-2021-45105, we are waiting for kafka to upgrade to Log4j 2.17. However,
> we came across following link in which there is a preview for the same.
>
> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
>
> Please let us know if it's safe and stable to upgrade our prod env with
> the preview or do we wait for Kafka official release (Log4j 2.x support
> with Java 8) for the same.
>
> Thanks in advance.
>
> Regards,
> Deepak
>
>

Re: Log4j 2.x preview for Kafka

[Luke Chen]

Hi Deepak,

As Israel mentioned, the core Apache Kafka components do not have any
dependencies on log4j2.
The only CVE that core Apache Kafka got impacted is CVE-2021-4104.
You can check the official announcement for the mitigation methods here:
https://kafka.apache.org/cve-list

For the log4j 2.x preview, I don't suggest applying to the production
environment.
After all, the PR haven't completed code reviewed yet.
But you can try it on staging environment, and welcome to report any issue
to us.

Thank you.
Luke






On Thu, Dec 23, 2021 at 3:39 AM Israel Ekpo  wrote:

> Currently, the core Apache Kafka components do not have any dependencies
> on log4j2
>
> There may be Kafka connectors that use log4j2  so you would need to check
> with your connector vendors to see if this applies to those connectors.
>
> If you do not use Kafka connect, then this may not apply to you.
>
> Here is the official announcement from the Kafka project on this issue
>
> https://kafka.apache.org/cve-list
>
> If you are using non-upstream Kafka distro that includes log4j2, then
> check with that vendor for additional information
>
> I hope this helps
>
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://izzyacademy.com/
>
>
> On Wed, Dec 22, 2021 at 10:58 AM Deepak Jain <
> deepak.j...@cumulus-systems.com> wrote:
>
>> Hi Luke,
>>
>> We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the
>> Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104 and
>> CVE-2021-45105, we are waiting for kafka to upgrade to Log4j 2.17. However,
>> we came across following link in which there is a preview for the same.
>>
>> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
>>
>> Please let us know if it's safe and stable to upgrade our prod env with
>> the preview or do we wait for Kafka official release (Log4j 2.x support
>> with Java 8) for the same.
>>
>> Thanks in advance.
>>
>> Regards,
>> Deepak
>>
>>

Re: Log4j 2.x preview for Kafka

[Deepak Jain]

Hi Israel,
Thanks for your prompt response but it didn't resolve my query.
We are mainly concern about the CVE-2021-4104 vulnerability as Log4j 1.x is use 
by the core components which are being currently used in our prod env.
I just want to know whether the preview in the following link (which uses 
latest kafka with Log4j2) is the stable and official release from the Apache 
kafka community and can we use it in our production environment.

If not, then since we do not use the JMS Appender so do we wait for the Apache 
kafka to officially release the Kafka version which uses Log4j2 and is 
compatible with Java 8.

Regards,
Deepak


From: Israel Ekpo 
Sent: Thursday, December 23, 2021 1:09:13 AM
To: Users 
Cc: Luke Chen 
Subject: Re: Log4j 2.x preview for Kafka

Currently, the core Apache Kafka components do not have any dependencies on
log4j2

There may be Kafka connectors that use log4j2  so you would need to check
with your connector vendors to see if this applies to those connectors.

If you do not use Kafka connect, then this may not apply to you.

Here is the official announcement from the Kafka project on this issue

https://kafka.apache.org/cve-list

If you are using non-upstream Kafka distro that includes log4j2, then check
with that vendor for additional information

I hope this helps

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://izzyacademy.com/


On Wed, Dec 22, 2021 at 10:58 AM Deepak Jain <
deepak.j...@cumulus-systems.com> wrote:

> Hi Luke,
>
> We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to the
> Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104 and
> CVE-2021-45105, we are waiting for kafka to upgrade to Log4j 2.17. However,
> we came across following link in which there is a preview for the same.
>
> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
>
> Please let us know if it's safe and stable to upgrade our prod env with
> the preview or do we wait for Kafka official release (Log4j 2.x support
> with Java 8) for the same.
>
> Thanks in advance.
>
> Regards,
> Deepak
>
>

RE: Log4j 2.x preview for Kafka

[Deepak Jain]

Hi Luke,

Thanks for your clarification. Just one more query:

Since, we do not use the JMS Appender so do we need to apply the mitigation 
mentioned in the below link for CVE-2021-4104.
https://kafka.apache.org/cve-list

Regards,
Deepak

-Original Message-
From: Luke Chen  
Sent: 23 December 2021 09:15
To: Israel Ekpo 
Cc: Users 
Subject: Re: Log4j 2.x preview for Kafka

Hi Deepak,

As Israel mentioned, the core Apache Kafka components do not have any 
dependencies on log4j2.
The only CVE that core Apache Kafka got impacted is CVE-2021-4104.
You can check the official announcement for the mitigation methods here:
https://kafka.apache.org/cve-list

For the log4j 2.x preview, I don't suggest applying to the production 
environment.
After all, the PR haven't completed code reviewed yet.
But you can try it on staging environment, and welcome to report any issue to 
us.

Thank you.
Luke






On Thu, Dec 23, 2021 at 3:39 AM Israel Ekpo  wrote:

> Currently, the core Apache Kafka components do not have any 
> dependencies on log4j2
>
> There may be Kafka connectors that use log4j2  so you would need to 
> check with your connector vendors to see if this applies to those connectors.
>
> If you do not use Kafka connect, then this may not apply to you.
>
> Here is the official announcement from the Kafka project on this issue
>
> https://kafka.apache.org/cve-list
>
> If you are using non-upstream Kafka distro that includes log4j2, then 
> check with that vendor for additional information
>
> I hope this helps
>
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://izzyacademy.com/
>
>
> On Wed, Dec 22, 2021 at 10:58 AM Deepak Jain < 
> deepak.j...@cumulus-systems.com> wrote:
>
>> Hi Luke,
>>
>> We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to 
>> the Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104 
>> and CVE-2021-45105, we are waiting for kafka to upgrade to Log4j 
>> 2.17. However, we came across following link in which there is a preview for 
>> the same.
>>
>> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
>>
>> Please let us know if it's safe and stable to upgrade our prod env 
>> with the preview or do we wait for Kafka official release (Log4j 2.x 
>> support with Java 8) for the same.
>>
>> Thanks in advance.
>>
>> Regards,
>> Deepak
>>
>>

Re: Log4j 2.x preview for Kafka

[Luke Chen]

Hi Deepak,

> Since, we do not use the JMS Appender so do we need to apply the
mitigation mentioned in the below link for CVE-2021-4104.
https://kafka.apache.org/cve-list

>From an application security perspective*, *I'd suggest applying it, to
avoid mis-configuration in the future by someone else.
As you know, one month later, when everyone forgets about the CVE, someone
could accidentally enable the JMS Appender.

Thank you.
Luke

On Thu, Dec 23, 2021 at 11:51 AM Deepak Jain <
deepak.j...@cumulus-systems.com> wrote:

> Hi Luke,
>
> Thanks for your clarification. Just one more query:
>
> Since, we do not use the JMS Appender so do we need to apply the
> mitigation mentioned in the below link for CVE-2021-4104.
> https://kafka.apache.org/cve-list
>
> Regards,
> Deepak
>
> -Original Message-
> From: Luke Chen 
> Sent: 23 December 2021 09:15
> To: Israel Ekpo 
> Cc: Users 
> Subject: Re: Log4j 2.x preview for Kafka
>
> Hi Deepak,
>
> As Israel mentioned, the core Apache Kafka components do not have any
> dependencies on log4j2.
> The only CVE that core Apache Kafka got impacted is CVE-2021-4104.
> You can check the official announcement for the mitigation methods here:
> https://kafka.apache.org/cve-list
>
> For the log4j 2.x preview, I don't suggest applying to the production
> environment.
> After all, the PR haven't completed code reviewed yet.
> But you can try it on staging environment, and welcome to report any issue
> to us.
>
> Thank you.
> Luke
>
>
>
>
>
>
> On Thu, Dec 23, 2021 at 3:39 AM Israel Ekpo  wrote:
>
> > Currently, the core Apache Kafka components do not have any
> > dependencies on log4j2
> >
> > There may be Kafka connectors that use log4j2  so you would need to
> > check with your connector vendors to see if this applies to those
> connectors.
> >
> > If you do not use Kafka connect, then this may not apply to you.
> >
> > Here is the official announcement from the Kafka project on this issue
> >
> > https://kafka.apache.org/cve-list
> >
> > If you are using non-upstream Kafka distro that includes log4j2, then
> > check with that vendor for additional information
> >
> > I hope this helps
> >
> > Israel Ekpo
> > Lead Instructor, IzzyAcademy.com
> > https://izzyacademy.com/
> >
> >
> > On Wed, Dec 22, 2021 at 10:58 AM Deepak Jain <
> > deepak.j...@cumulus-systems.com> wrote:
> >
> >> Hi Luke,
> >>
> >> We are using Kafka 2.8.1 Broker/Client system in our prod env. Due to
> >> the Log4j vulnerability CVE-2021-44228, CVE-2021-45046, CVE-2021-4104
> >> and CVE-2021-45105, we are waiting for kafka to upgrade to Log4j
> >> 2.17. However, we came across following link in which there is a
> preview for the same.
> >>
> >> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
> >>
> >> Please let us know if it's safe and stable to upgrade our prod env
> >> with the preview or do we wait for Kafka official release (Log4j 2.x
> >> support with Java 8) for the same.
> >>
> >> Thanks in advance.
> >>
> >> Regards,
> >> Deepak
> >>
> >>
>