[DISCUSS] KIP-1097: error record reporter

[Anton Liauchuk]

Hi all,

I have opened 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1097+error+record+reporter

POC: https://github.com/apache/kafka/pull/17493

Please review KIP and PR, feedbacks and suggestions are welcome.

[jira] [Created] (KAFKA-17785) Kafka protocol documentation should include tagged field information

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-17785:


 Summary: Kafka protocol documentation should include tagged field 
information
 Key: KAFKA-17785
 URL: https://issues.apache.org/jira/browse/KAFKA-17785
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 4.0.0


The Kafka documentation includes a description of the Kafka protocol 
([https://kafka.apache.org/protocol]). This is useful to someone wanting to 
understand the details of the protocol. However, in the area of tagged fields, 
the documentation is not helpful.

Tagged fields provide a way of supplying optional data in a request or 
response. Modern requests and responses usually permit tagged fields, although 
they are relatively rarely used in practice.

One RPC in which tagged fields contain important information is the 
APIVersionsResponse. In v3 and above, the response contains the set of 
supported and finalised features, which can be used by a client to determine 
whether a broker supports a feature. This information is carried as tagged 
fields, but if you read the documentation of the protocol, you would be 
entirely in the dark.

This improvement adds tagged field information into the protocol documentation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17783) Remove sharePartition from SharePartitionManager if the partition partition is deleted or becomes a follower

[Abhinav Dixit (Jira)]

Abhinav Dixit created KAFKA-17783:
-

 Summary: Remove sharePartition from SharePartitionManager if the 
partition partition is deleted or becomes a follower
 Key: KAFKA-17783
 URL: https://issues.apache.org/jira/browse/KAFKA-17783
 Project: Kafka
  Issue Type: Sub-task
Reporter: Abhinav Dixit


In reference to comment 
[https://github.com/apache/kafka/pull/17437#discussion_r1795840636] , in the 
case when a partition is deleted or becomes a follower, we probably also want 
to remove the sharePartition from {{SharePartitionManager}} to free up the space



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #203

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #100

[Apache Jenkins Server]

See 

Re: [VOTE] 3.9.0 RC2

[Federico Valeri]

Hi Colin, thanks for putting out the new RC.

According to KIP-950, remote.log.manager.thread.pool.size should be
marked as deprecated and replaced by two new configurations:
remote.log.manager.copier.thread.pool.size and
remote.log.manager.expiration.thread.pool.size. I don't see that
reflected in the documentation, but I guess it can be fixed without
another RC.





On Sat, Oct 12, 2024 at 10:50 AM Chia-Ping Tsai  wrote:
>
> hi Colin
>
> I'm sending this email to remind developers who plan to test E2E for RC2.
>
> There is an issue with the E2E tests due to the new release of Ducktape. I've 
> filed KAFKA-17781 to address it. Only developers who recently rebuilt the E2E 
> image will encounter this issue, so I assume it is NOT a blocker for RC2.
>
> Best,
> Chia-Ping
>
>
> On 2024/10/10 21:14:55 Colin McCabe wrote:
> > This is the second candidate for the release of Apache Kafka 3.9.0. I have 
> > titled it rc2 since I had an rc1 which got very far, even to the point of 
> > pushing tags and docker images, before I spotted an issue. So rather than 
> > mutate the tags, I decided to skip over rc1.
> >
> > - This is a major release, the final one in the 3.x line. (There may of 
> > course be other minor releases in this line, such as 3.9.1.)
> > - Tiered storage will be considered production-ready in this release.
> > - This will be the final major release to feature the deprecated ZooKeeper 
> > mode.
> >
> > This release includes the following KIPs:
> > - KIP-853: Support dynamically changing KRaft controller membership
> > - KIP-1057: Add remote log metadata flag to the dump log tool
> > - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> > - KIP-1040: Improve handling of nullable values in InsertField, 
> > ExtractField, and other transformations
> > - KIP-1031: Control offset translation in MirrorSourceConnector
> > - KIP-1033: Add Kafka Streams exception handler for exceptions occurring 
> > during processing
> > - KIP-1017: Health check endpoint for Kafka Connect
> > - KIP-1025: Optionally URL-encode clientID and clientSecret in 
> > authorization header
> > - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> > - KIP-950: Tiered Storage Disablement
> > - KIP-956: Tiered Storage Quotas
> >
> > Release notes for the 3.9.0 release:
> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by October 16, 2024.
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/
> >
> > * Docker release artifacts to be voted upon:
> > apache/kafka:3.9.0-rc2
> > apache/kafka-native:3.9.0-rc2
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/javadoc/
> >
> > * Documentation:
> > https://kafka.apache.org/39/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/39/protocol.html
> >
> > * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc2 tag:
> > https://github.com/apache/kafka/releases/tag/3.9.0-rc2
> >
> > * Successful Docker Image Github Actions Pipeline for 3.9 branch:
> > Docker Build Test Pipeline (JVM): 
> > https://github.com/apache/kafka/actions/runs/11281563007
> > Docker Build Test Pipeline (Native): 
> > https://github.com/apache/kafka/actions/runs/11281608809
> >
> > Thanks to everyone who helped with this release candidate, either by 
> > contributing code, testing, or documentation.
> >
> > Regards,
> > Colin
> >

[jira] [Created] (KAFKA-17784) Mirror Maker2 Pod CrashLoopbackOff

[George Yang (Jira)]

George Yang created KAFKA-17784:
---

 Summary: Mirror Maker2 Pod CrashLoopbackOff
 Key: KAFKA-17784
 URL: https://issues.apache.org/jira/browse/KAFKA-17784
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.7.1
Reporter: George Yang
 Attachments: connect-1.log

When I use k8s deployment with mirrormaker v3.7.1, and deploy one kafka node in 
each data center, I always got the crashloopbackoff error, please see the 
attachment. 

 

The configuration of mirrormaker is:

```configuration

# specify any number of cluster aliases
clusters = idca, idcb

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
idca.bootstrap.servers = 192.168.2.146:13399
idcb.bootstrap.servers = 192.168.2.147:13399

# enable and configure individual replication flows
idca->idcb.enabled = true

# regex which defines which topics gets replicated.
idca->idcb.topics = .*

idcb->idca.enabled = true
idcb->idca.topics = .*

# Setting replication factor of newly created remote topics
replication.factor=1

tasks.max=6
emit.checkpoints.interval.seconds=5
idca.producer.acks=all
idca.producer.batch.size=5
#idcb.producer.enable.idempotence=true

idca.consumer.auto.offset.reset=latest
idcb.consumer.auto.offset.reset=latest
idca.consumer.max.poll.interval.ms=2
idcb.consumer.max.poll.interval.ms=2

refresh.topics.enabled=true
refresh.topics.interval.seconds=5

refresh.groups.enabled=true
refresh.groups.interval.seconds=5

dedicated.mode.enable.internal.rest = true
idca.scheduled.rebalance.max.delay.ms=2
idcb.scheduled.rebalance.max.delay.ms=2

# Internal Topic Settings  
#
# The replication factor for mm2 internal topics "heartbeats", 
"idcb.checkpoints.internal" and
# "mm2-offset-syncs.idcb.internal"
# For anything other than development testing, a value greater than 1 is 
recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3

# The replication factor for connect internal topics 
"mm2-configs.idcb.internal", "mm2-offsets.idcb.internal" and
# "mm2-status.idcb.internal"
# For anything other than development testing, a value greater than 1 is 
recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3

```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-17520) align the low bound of ducktape version

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai reopened KAFKA-17520:


> align the low bound of ducktape version 
> 
>
> Key: KAFKA-17520
> URL: https://issues.apache.org/jira/browse/KAFKA-17520
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Eric Chang
>Priority: Minor
> Fix For: 4.0.0, 3.7.2, 3.8.1
>
>
> There was a discussion for it: 
> https://github.com/apache/kafka/pull/14216#issuecomment-2117519600
> and I noticed this issue when trying to upgrade base image for e2e



--
This message was sent by Atlassian Jira
(v8.20.10#820010)