Re: [DISCUSS] Broker behaviour when returning records

Hi Megan, Answering your questions below: 1. Why can't the broker remove messages from the first batch? So that fetch.max.bytes is not violated. The documentation has answered your question: If the first record batch in the first non-empty partition of the fetch is larger than this limit, the b

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

Hi Tom, 1. > I think you mean tools-log4j.properties there, right? Thanks for reading carefully. Yes, you are right. It was an error. 2. > Can the log4j-appender log a warning about its own deprecation? That's a good suggestion. I hadn't thought of this part - adding a WARNing message to the Lo

Re: State store guarantees - KStreams 2.8.1

Well, it's unclear what the remote lookup does... As Kafka Streams does not implement this part, my best guess at the moment is to blame it on a bug in the remote request implementation. Are you using some of-the-shelf implementation for the remove lookup part or are you using something build

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #626

See

[jira] [Resolved] (KAFKA-13502) Support configuring BROKER_LOGGER on controller-only KRaft nodes

[ https://issues.apache.org/jira/browse/KAFKA-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio resolved KAFKA-13502. Resolution: Fixed This issue was fixed by KAFKA-13552. > Support

[jira] [Resolved] (KAFKA-13552) Unable to dynamically change broker log levels on KRaft

[ https://issues.apache.org/jira/browse/KAFKA-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Armando Garcia Sancio resolved KAFKA-13552. Resolution: Fixed > Unable to dynamically change broker log levels

Re: [kafka-clients] Re: [VOTE] 3.1.0 RC1

Thanks for managing and running this release, David! I am excited about this particular release and I appreciate every effort and contribution that made this possible. On Fri, Jan 21, 2022 at 8:39 AM 'David Jacot' via kafka-clients < kafka-clie...@googlegroups.com> wrote: > Thanks for all the vo

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #625

See

[DISCUSS] Broker behaviour when returning records

Hi all, I have some questions about how brokers batch records to send to consumers. One of the configuration properties for a consumer is fetch.max.bytes. Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about fetch.max.bytes : "This property lets you specify the maximum bytes

[GitHub] [kafka-site] dajac commented on a change in pull request #393: MINOR: Update site for Apache Kafka 3.1.0 release

dajac commented on a change in pull request #393: URL: https://github.com/apache/kafka-site/pull/393#discussion_r790003684 ## File path: downloads.html ## @@ -6,12 +6,58 @@ Download -3.0.0 is the latest release. The current stable version is 3.0.0. +3.1.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #624

See Changes: -- [...truncated 508953 lines...] [2022-01-21T21:14:07.519Z] > Task :connect:api:javadocJar [2022-01-21T21:14:07.519Z] > Task :connect:api:compileTestJava UP-TO

[GitHub] [kafka-site] bbejeck commented on a change in pull request #393: MINOR: Update site for Apache Kafka 3.1.0 release

bbejeck commented on a change in pull request #393: URL: https://github.com/apache/kafka-site/pull/393#discussion_r78637 ## File path: downloads.html ## @@ -6,12 +6,58 @@ Download -3.0.0 is the latest release. The current stable version is 3.0.0. +3.

Re: State store guarantees - KStreams 2.8.1

I agree it sounds a bit off, but it seems that even a host that is not marked as active allows me to query it's store and gives me a result that is not null. This application has an API that either queries local or remote store (basically via HTTP API of active host), but the weird part is I get t

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #68

See

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

For a little background on why `AlterIsr` returns the state back in the response, originally the idea was that the partition leader could use the response to reset its own state after a failed request. The tricky thing for ISR changes is always ensuring that the partition leader reflects a worst-ca

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.0 #169

See

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

Hi all, The following suggestions are not strictly required to implement this KIP but what do we think about: 1. Changing the name of the AlterIsr RPC to AlterPartition RPC. 2. Change the name of the field "CurrentIsrVersion" to "PartitionEpoch". This is the name that we use in the KRaft impleme

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

Thanks Raman and Colin for your feedback. Raman wrote: > - Could you please explain the following about backward compatibility. > If a leader has been elected unclean. And we decide to roll the > cluster back when the leader is in the middle of recovery, leader will > simply not be able to recover

[GitHub] [kafka-site] junrao merged pull request #392: adding new vulnerability info - CVE-2022-23307

junrao merged pull request #392: URL: https://github.com/apache/kafka-site/pull/392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr..

Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment

Hey David, Thanks for the proposal. This was a tricky bug and I think your approach is probably the best way forward. It would be helpful to add a little more detail to the proposal. When the coordinator detects that the static leader is returning, it will set `skipAssignment` to true in the `Joi

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #623

See Changes: -- [...truncated 507111 lines...] [2022-01-21T18:40:21.986Z] [2022-01-21T18:40:21.986Z] org.apache.kafka.streams.integration.EosIntegrationTest > shouldBeAbl

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #67

See

[jira] [Resolved] (KAFKA-13412) Retry of initTransactions after timeout may cause invalid transition

[ https://issues.apache.org/jira/browse/KAFKA-13412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-13412. - Fix Version/s: 3.2.0 3.1.1 Resolution: Fixed > Retry of initTr

[jira] [Created] (KAFKA-13611) Failed reconfiguration of tasks can cause missing offset replications in MirrorCheckpointConnector

Aljoscha Pörtner created KAFKA-13611: Summary: Failed reconfiguration of tasks can cause missing offset replications in MirrorCheckpointConnector Key: KAFKA-13611 URL: https://issues.apache.org/jira/browse/KAF

Re: State store guarantees - KStreams 2.8.1

but instance A returns result X for a partition I and instance B returns result Y for the same partition I. This sound a little off. As you stated, if both instances agree on the active host, the active host must either be instance A or instance B, and thus you can query partition I only on i

[jira] [Resolved] (KAFKA-13388) Kafka Producer nodes stuck in CHECKING_API_VERSIONS

[ https://issues.apache.org/jira/browse/KAFKA-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-13388. - Fix Version/s: 3.0.1 3.2.0 3.1.1 Reviewer: Rajini

[jira] [Resolved] (KAFKA-13591) Fix flaky test ControllerIntegrationTest.testTopicIdCreatedOnUpgrade

[ https://issues.apache.org/jira/browse/KAFKA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-13591. - Fix Version/s: 3.2.0 3.1.1 Reviewer: Jason Gustafson Resolutio

[jira] [Created] (KAFKA-13610) Make log.cleaner.enable dynamically configurable

Colin McCabe created KAFKA-13610: Summary: Make log.cleaner.enable dynamically configurable Key: KAFKA-13610 URL: https://issues.apache.org/jira/browse/KAFKA-13610 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-13609) Fix the exception type thrown from dynamic broker config validation

Colin McCabe created KAFKA-13609: Summary: Fix the exception type thrown from dynamic broker config validation Key: KAFKA-13609 URL: https://issues.apache.org/jira/browse/KAFKA-13609 Project: Kafka

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #66

See

[jira] [Created] (KAFKA-13608) Implement Position restoration for all in-memory state stores

Vicky Papavasileiou created KAFKA-13608: --- Summary: Implement Position restoration for all in-memory state stores Key: KAFKA-13608 URL: https://issues.apache.org/jira/browse/KAFKA-13608 Project:

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #392: adding new vulnerability info - CVE-2022-23307

scott-confluent commented on a change in pull request #392: URL: https://github.com/apache/kafka-site/pull/392#discussion_r789758488 ## File path: cve-list.html ## @@ -9,6 +9,31 @@ Apache Kafka Security Vulnerabilities This page lists all security vulnerabilities fixed in re

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #168

See Changes: -- [...truncated 418447 lines...] [2022-01-21T14:52:19.667Z] ControllerIntegrationTest > testPreemptionOnControllerShutdown() STARTED [2022-01-21T14:52:21.539Z]

[jira] [Created] (KAFKA-13607) Cannot use PEM certificate coding when parent defined file-based

Piotr Smolinski created KAFKA-13607: --- Summary: Cannot use PEM certificate coding when parent defined file-based Key: KAFKA-13607 URL: https://issues.apache.org/jira/browse/KAFKA-13607 Project: Kafka

__consumer_offsets auto creation with `auto.create.topics.enable = false`

Hi, Kafka 2.8.x changed the behaviour of auto topic creation (KAFKA-9751) so that it's now done asynchronously by a forwarded request to the controller rather than being immediately started by the broker that received the MetadataRequest. In 2.7.x and earlier the client used to get LEADER_NOT_A

[RESULTS] [VOTE] Release Kafka version 3.1.0

This vote passes with 7 +1 votes (4 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Randall Hauch * Bill Bejeck * Mickael Maison * David Jacot Committers: * No votes Community: * Michal Tóth * Jakub Scholz * Israel Ekpo 0 votes * No votes -1 votes * No votes Vote thread: https://lists

Re: [VOTE] 3.1.0 RC1

Thanks for all the votes! I'm also +1 (binding) given that: * Ran the release and generated 3.1.0 RC1. * Verified all checksums and signatures. * Built from source and ran unit tests. * Installed 3.1.0 RC1 and performed the quickstart for both ZK and KRaft. * Spotchecked the Javadocs and the site

State store guarantees - KStreams 2.8.1

Hi everyone, I'm trying for a while to answer myself a question about what are actually guarantees for state stores in regards to consistency when connected to transformers. I have an application where a single (persistent, rocksdb backed) state store is connected to multiple transformers. Each t

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #167

See Changes: -- [...truncated 416052 lines...] [2022-01-21T12:34:47.748Z] KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED [2022-01-21T12:34:47.748Z] [2022-

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

Hi Dongjin, Many thanks for the KIP. I have just a couple of minor points. 1. In the Proposed Changes section there is the sentence "For backward compatibility, these tools will use the log4j configuration file (tools-log4j2.properties) by default.", I think you mean tools-log4j.properties there,

Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT

Hi Julien, Thanks! A couple of other points sprang to mind: 1. seconds is a unit, but millis etc are really just prefixes. I can imagine what my old physics teacher would have to say about mixing these concepts :-). I would have preferred to use the abbreviations s, ms, µs and ns (ms in particula

[jira] [Created] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

Stanislav Savulchik created KAFKA-13606: --- Summary: MirrorCheckpointTask doesn't check offsets sync result Key: KAFKA-13606 URL: https://issues.apache.org/jira/browse/KAFKA-13606 Project: Kafka