[jira] [Resolved] (KAFKA-13033) coordinator not available error should cause add into unmap list to do a new lookup

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13033.
-
  Reviewer: David Jacot
Resolution: Fixed

> coordinator not available error should cause add into unmap list to do a new 
> lookup
> ---
>
> Key: KAFKA-13033
> URL: https://issues.apache.org/jira/browse/KAFKA-13033
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Blocker
> Fix For: 3.0.0
>
>
> In KIP-699, we add some handler to handle different types of operation. In 
> the `handleError`, we didn't make the `COORDINATOR_NOT_AVAILABLE` as 
> unmapped, to do a re-lookup. In 
> [DescribeTransactionsHandler|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/internals/DescribeTransactionsHandler.java#L172-L186],
>  there's already explained by [~hachikuji] why `COORDINATOR_NOT_AVAILABLE` 
> and `NOT_COORDINATOR` should be listed in unmapped, and 
> `COORDINATOR_LOAD_IN_PROGRESS` should not.
>  
> {code:java}
> case COORDINATOR_LOAD_IN_PROGRESS:
> // If the coordinator is in the middle of loading, then we just need to 
> retry
> log.debug("DescribeTransactions request for transactionalId `{}` failed 
> because the " +
> "coordinator is still in the process of loading state. Will 
> retry",
> transactionalIdKey.idValue);
> break;
> case NOT_COORDINATOR:
> case COORDINATOR_NOT_AVAILABLE:
> // If the coordinator is unavailable or there was a coordinator change, 
> then we unmap
> // the key so that we retry the `FindCoordinator` request
> unmapped.add(transactionalIdKey);
> log.debug("DescribeTransactions request for transactionalId `{}` returned 
> error {}. Will attempt " +
> "to find the coordinator again and retry", 
> transactionalIdKey.idValue, error);
> break;
> {code}
> We should be consistent with it. Fix it, add logs and comments, and also 
> update the tests.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build became unstable: Kafka » Kafka Branch Builder » 3.0 #37

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #329

[Apache Jenkins Server]

See 

Zookeeper : Throttling connections

[Kafka Life]

Dear KAFKA & Zookeeper experts.

1/ What is zookeeper Throttling ? Is it done at zookeepr ? How is it set
configured ?
2/ Is it helpful ?

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

[Omnia Ibrahim]

Hi,
Can I get 2 more +1 binding for this KIP?
Thanks

On Fri, Jul 2, 2021 at 5:14 PM Omnia Ibrahim 
wrote:

> Hi All,
>
> Just thought of bumping this voting thread again to see if we can form a
> consensus around this.
>
> Thanks
>
> On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison 
> wrote:
>
>> +1 (binding)
>> Thanks for the KIP!
>>
>> On Tue, May 4, 2021 at 3:23 PM Igor Soarez 
>> wrote:
>> >
>> > Another +1 here, also non-binding.
>> >
>> > Thank you Omnia!
>> >
>> > --
>> > Igor
>> >
>> >
>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
>> > > +1 (non-binding), thanks!
>> > >
>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim 
>> wrote:
>> > >
>> > >> Hi
>> > >> Can I get a vote on this, please?
>> > >>
>> > >> Best
>> > >> Omnia
>> > >>
>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
>> o.g.h.ibra...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> If anyone interested in reading the discussions you can find it here
>> > >>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
>> > >>>
>> > >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim <
>> o.g.h.ibra...@gmail.com>
>> > >>> wrote:
>> > >>>
>> >  Hi everyone,
>> >  I’m proposing a new KIP for MirrorMaker 2 to add the ability to
>> control
>> >  internal topics naming convention. The proposal details are here
>> > 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>> > 
>> >  Please vote in this thread.
>> >  Thanks
>> >  Omnia
>> > 
>> > >>>
>> > >
>>
>

Re: Zookeeper : Throttling connections

[Israel Ekpo]

Hello,

I am assuming you are using Zookeeper because of your Kafka brokers. What
version of Kafka are you using.

I would like to start by stating that very soon this will no longer be an
issue as the project is taking steps to decouple Kafka from Zookeeper. Take
a look at KIP-500 for additional information. In Kafka 2.8.0 you should be
able to configure the early access version of running Kafka without
Zookeeper.

When running Kafka in KRaft Mode (Without Zookeeper) you do not need to
worry about this throttling since Zookeeper is not part of the architecture.

To answer you questions, some changes were made to prevent Zookeeper from
getting overwhelmed hence to avoid recurring crashes during stressful loads
restrictions can be configured to manage the load on Zookeeper via the
throttling settings

https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-3243

You should be able to see additional information in the administration
guides on how this is configured

https://zookeeper.apache.org/doc/r3.2.2/zookeeperAdmin.html

For example, the configuration setting below

globalOutstandingLimit

(Java system property: *zookeeper.globalOutstandingLimit.*)

Clients can submit requests faster than ZooKeeper can process them,
especially if there are a lot of clients. To prevent ZooKeeper from running
out of memory due to queued requests, ZooKeeper will throttle clients so
that there is no more than globalOutstandingLimit outstanding requests in
the system. The default limit is 1,000.

I hope this gets you started

Feel free to reach out if you have additional questions

Have a great day


On Fri, Jul 16, 2021 at 7:44 AM Kafka Life  wrote:

> Dear KAFKA & Zookeeper experts.
>
> 1/ What is zookeeper Throttling ? Is it done at zookeepr ? How is it set
> configured ?
> 2/ Is it helpful ?
>

Re: Zookeeper : Throttling connections

[Kafka Life]

Thank you very much Mr. Israel Ekpo. Really appreciate it.

We are using the 0.10 version of kafka and in the process of upgrading to
2.6.1 . Planning in process and Yes, these connections to zookeepers are
for Kafka functionality.

frequently there are incidents where zookeepers get bombarded with around
8,000 to 10,000 connections from multiple clients.
Do you suggest we can configure globalOutstandingLimit to 8000 ?
should this be set in zoo.cfg file in Zookeeper ?


On Fri, Jul 16, 2021 at 7:11 PM Israel Ekpo  wrote:

> Hello,
>
> I am assuming you are using Zookeeper because of your Kafka brokers. What
> version of Kafka are you using.
>
> I would like to start by stating that very soon this will no longer be an
> issue as the project is taking steps to decouple Kafka from Zookeeper. Take
> a look at KIP-500 for additional information. In Kafka 2.8.0 you should be
> able to configure the early access version of running Kafka without
> Zookeeper.
>
> When running Kafka in KRaft Mode (Without Zookeeper) you do not need to
> worry about this throttling since Zookeeper is not part of the
> architecture.
>
> To answer you questions, some changes were made to prevent Zookeeper from
> getting overwhelmed hence to avoid recurring crashes during stressful loads
> restrictions can be configured to manage the load on Zookeeper via the
> throttling settings
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-3243
>
> You should be able to see additional information in the administration
> guides on how this is configured
>
> https://zookeeper.apache.org/doc/r3.2.2/zookeeperAdmin.html
>
> For example, the configuration setting below
>
> globalOutstandingLimit
>
> (Java system property: *zookeeper.globalOutstandingLimit.*)
>
> Clients can submit requests faster than ZooKeeper can process them,
> especially if there are a lot of clients. To prevent ZooKeeper from running
> out of memory due to queued requests, ZooKeeper will throttle clients so
> that there is no more than globalOutstandingLimit outstanding requests in
> the system. The default limit is 1,000.
>
> I hope this gets you started
>
> Feel free to reach out if you have additional questions
>
> Have a great day
>
>
> On Fri, Jul 16, 2021 at 7:44 AM Kafka Life  wrote:
>
> > Dear KAFKA & Zookeeper experts.
> >
> > 1/ What is zookeeper Throttling ? Is it done at zookeepr ? How is it set
> > configured ?
> > 2/ Is it helpful ?
> >
>

[jira] [Resolved] (KAFKA-13017) Excessive logging on sink task deserialization errors

[Tom Bentley (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Bentley resolved KAFKA-13017.
-
Fix Version/s: 3.1.0
   Resolution: Fixed

> Excessive logging on sink task deserialization errors
> -
>
> Key: KAFKA-13017
> URL: https://issues.apache.org/jira/browse/KAFKA-13017
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 3.0.0, 2.7.0, 2.8.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.1.0
>
>
> Even with {{errors.log.enable}} set to {{false}}, deserialization failures 
> are still logged at {{ERROR}} level by the 
> {{org.apache.kafka.connect.runtime.WorkerSinkTask}} namespace. This becomes 
> problematic in pipelines with {{errors.tolerance}} set to {{all}}, and can 
> generate excessive logging of stack traces when deserialization errors are 
> encountered for most if not all of the records being consumed by a sink task.
> The logging added to the {{WorkerSinkTask}} class in KAFKA-9018 should be 
> removed and, if necessary, any valuable information from it not already 
> present in the log messages generated by Connect with {{errors.log.enable}} 
> and {{errors.log.include.messages}} set to {{true}} should be added in that 
> place instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13097) Handle the requests gracefully to publish the events in TopicBasedRemoteLogMetadataManager when it is not yet initialized.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-13097:
--

 Summary: Handle the requests gracefully to publish the events in 
TopicBasedRemoteLogMetadataManager when it is not yet initialized.
 Key: KAFKA-13097
 URL: https://issues.apache.org/jira/browse/KAFKA-13097
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana
Assignee: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13098) No such file exception when recovering snapshots in metadata log dir

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-13098:
--

 Summary: No such file exception when recovering snapshots in 
metadata log dir
 Key: KAFKA-13098
 URL: https://issues.apache.org/jira/browse/KAFKA-13098
 Project: Kafka
  Issue Type: Bug
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio
 Fix For: 3.0.0


{code:java}
RaftClusterTest > testCreateClusterAndCreateListDeleteTopic() FAILED
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: 
/tmp/kafka-286994548094074875/broker_0_data0/@metadata-0/partition.metadata.tmp
at 
java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at 
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at 
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)   

at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at 
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at 
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)


   
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at 
kafka.raft.KafkaMetadataLog$.recoverSnapshots(KafkaMetadataLog.scala:616)
at kafka.raft.KafkaMetadataLog$.apply(KafkaMetadataLog.scala:583)   
   
at kafka.raft.KafkaRaftManager.buildMetadataLog(RaftManager.scala:257)
at kafka.raft.KafkaRaftManager.(RaftManager.scala:132)


 
at 
kafka.testkit.KafkaClusterTestKit$Builder.build(KafkaClusterTestKit.java:227)
at 
kafka.server.RaftClusterTest.testCreateClusterAndCreateListDeleteTopic(RaftClusterTest.scala:87)
 {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13099) Message too large error when expiring transactionalIds

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13099:
---

 Summary: Message too large error when expiring transactionalIds
 Key: KAFKA-13099
 URL: https://issues.apache.org/jira/browse/KAFKA-13099
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


We have seen a couple reports of MESSAGE_TOO_LARGE errors when writing 
tombstones for expired transactionalIds. This is possible because we collect 
all expired IDs into a single batch. We should ensure that the created batches 
are smaller than the max message size. Any expired IDs that cannot fit can be 
expired later.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #38

[Apache Jenkins Server]

See 

[DISCUSS] KIP-732 Delete Committed Connect Records

[Ryanne Dolan]

Hey y'all, please review the following small proposal:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-762%3A+Delete+Committed+Connect+Records

The idea is to enable Source->Sink pipelines to clean up after themselves
by automatically truncating intermediate topics.

Ryanne

[jira] [Resolved] (KAFKA-12777) AutoTopicCreationManager does not handle response errors

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-12777.
--
Resolution: Fixed

Committed #11069, thanks!

> AutoTopicCreationManager does not handle response errors
> 
>
> Key: KAFKA-12777
> URL: https://issues.apache.org/jira/browse/KAFKA-12777
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: David Arthur
>Priority: Critical
> Fix For: 3.0.0
>
>
> The request completion callback in AutoTopicCreationManager assumes the 
> response is present. 
> {code:scala}
> override def onComplete(response: ClientResponse): Unit = {
> debug(s"Auto topic creation completed for ${creatableTopics.keys} 
> with response ${response.responseBody.toString}.")
> clearInflightRequests(creatableTopics)
>   }
> {code}
> We should at least check to see if the response exists before logging it and 
> clearing the in-flight flag. 
> This was found while debugging a separate issue. Here is a log snippet:
> {code}
> [2021-05-13 11:21:03,890] DEBUG [BrokerToControllerChannelManager broker=1 
> name=forwarding] Version mismatch when attempting to send 
> EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0 lim=43 cap=43], 
> requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 10, 65, 78, 79, 78, 89, 77, 79, 
> 85, 83, 0, 0], clientHostAddress=[127, 0, 0, 1]) with correlation id 2 to 0 
> (org.apache.kafka.clients.NetworkClient:495)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support ENVELOPE
> [2021-05-13 11:21:03,893] ERROR [BrokerToControllerChannelManager broker=1 
> name=forwarding]: Request 
> EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0 lim=43 cap=43], 
> requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 10, 65, 78, 79, 78, 89, 77, 79, 
> 85, 83, 0, 0], clientHostAddress=[127, 0, 0, 1]) failed due to unsupported 
> version error (kafka.server.BrokerToControllerRequestThread:76)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support ENVELOPE
> [2021-05-13 11:21:03,894] ERROR [BrokerToControllerChannelManager broker=1 
> name=forwarding] Uncaught error in request completion: 
> (org.apache.kafka.clients.NetworkClient:576)
> java.lang.NullPointerException
>   at 
> kafka.server.DefaultAutoTopicCreationManager$$anon$1.$anonfun$onComplete$1(AutoTopicCreationManager.scala:179)
>   at kafka.utils.Logging.debug(Logging.scala:62)
>   at kafka.utils.Logging.debug$(Logging.scala:62)
>   at 
> kafka.server.DefaultAutoTopicCreationManager.debug(AutoTopicCreationManager.scala:67)
>   at 
> kafka.server.DefaultAutoTopicCreationManager$$anon$1.onComplete(AutoTopicCreationManager.scala:179)
>   at 
> kafka.server.BrokerToControllerRequestThread.handleResponse(BrokerToControllerChannelManager.scala:355)
>   at 
> kafka.server.BrokerToControllerRequestThread.$anonfun$generateRequests$1(BrokerToControllerChannelManager.scala:339)
>   at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
>   at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:545)
>   at 
> kafka.common.InterBrokerSendThread.pollOnce(InterBrokerSendThread.scala:74)
>   at 
> kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManager.scala:374)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> {code}
> I suspect the NPE is due to {{response.responseBody}} being null.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #39

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #40

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #331

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #41

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #332

[Apache Jenkins Server]

See