lianetm commented on code in PR #668:
URL: https://github.com/apache/kafka-site/pull/668#discussion_r1990138511
##########
blog.html:
##########
@@ -22,6 +22,286 @@
<!--#include virtual="includes/_nav.htm" -->
<div class="right">
<h1 class="content-title">Blog</h1>
+ <article>
+ <h2 class="bullet">
+ <a id="apache_kafka_400_release_announcement"></a>
+ <a href="#apache_kafka_400_release_announcement">Apache
Kafka 4.0.0 Release Announcement</a>
+ </h2>
+ 17 March 2025 - David Jacot
+
+ <p>
+ We are proud to announce the release of Apache Kafka®
4.0 This release contains many new
+ features and improvements. This blog post will highlight
some of the more prominent features.
+ For a full list of changes, be sure to check the <a
href="https://archive.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html";>release
notes</a>.
+ </p>
+ <p>
+ See the <a
href="https://kafka.apache.org/documentation.html#upgrade_4_0_0";>Upgrading to
4.0 from any versions 0.8.x through 3.9.x</a> section in the
+ documentation for the list of notable changes and detailed
upgrade steps.
+ </p>
+ <p>
+ Apache Kafka 4.0 is a significant milestone, marking the
first major release to operate entirely
+ without Apache ZooKeeper®. By running in KRaft mode
by default, Kafka simplifies
+ deployment and management, eliminating the complexity of
maintaining a separate ZooKeeper
+ ensemble. This change significantly reduces operational
overhead, enhances scalability, and
+ streamlines administrative tasks. We want to take this as
an opportunity to express our gratitude
+ to the ZooKeeper community and say, thank you! ZooKeeper
was the backbone of Kafka for
+ more than 10 years, and it did serve Kafka very well.
Kafka would most likely not be what it is
+ today without it. We don’t take this for granted and
highly appreciate all the hard work the
+ community invested to build ZooKeeper. Thank you!
+ </p>
+ <p>
+ Kafka 4.0 also brings the general availability of <a
href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848</a> introducing a
powerful new consumer
+ group protocol designed to dramatically improve rebalance
performance. This optimization
+ significantly reduces downtime and latency, enhancing the
reliability and responsiveness of
+ consumer groups, especially in large-scale deployments.
+ </p>
+ <p>
+ Additionally, we are excited to offer <a
href="https://cwiki.apache.org/confluence/x/i4ogF";>early access</a> to
+ Queues for Kafka (<a
href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932</a>), enabling
Kafka to support traditional queue semantics directly. This
+ feature extends Kafka’s versatility, making it an ideal
messaging platform for a wider range of
+ use cases, particularly those requiring point-to-point
messaging patterns.
+ </p>
+ <p>
+ These highlights illustrate Kafka's continued evolution to
meet the demands of modern,
+ scalable, and efficient data streaming and messaging
architectures.
+ </p>
+
+ <h3>Kafka Broker, Controller, Producer, Consumer and Admin
Client</h3>
+
+ <ul>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/HhD1D";>KIP-848: The Next Generation
of the Consumer Rebalance Protocol</a>
+ <br>
+ Apache Kafka also says goodbye to “stop-the-world”
rebalances with the general availability of the
+ next generation of the consumer rebalance protocol. It
increases the stability and the performance
+ of consumer groups while simplifying clients. The new
protocol is enabled by default on the server
+ side. Consumers must opt-in by setting
<code>group.protocol=consumer</code>.
+ See <a
href="https://kafka.apache.org/40/documentation.html#consumer_rebalance_protocol";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/B40ODg";>KIP-890: Transactions
Server-Side Defense</a>
+ <br>
+ The second phase of KIP-890 has been completed. It
reduces the chances of “zombie transactions”
+ during producer failures. See <a
href="https://kafka.apache.org/40/documentation.html#transaction_protocol";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/4hA0Dw";>KIP-932: Queues for Kafka
(Early Access)</a>
+ <br>
+ This KIP introduces the concept of a share group as a
way of enabling cooperative consumption
+ using Kafka topics. It does not add the concept of a
“queue” to Kafka per se, but rather that
+ introduces cooperative consumption to accommodate
these queuing use-cases using regular Kafka
+ topics. Share groups make this possible. You can think
of a share group as roughly equivalent to a
+ “durable shared subscription” in existing systems.
+ Please see the <a
href="https://cwiki.apache.org/confluence/x/i4ogF";>Early Access Release
Notes</a>
+ for limitations and configurations.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/mpOzDw";>KIP-966: Eligible Leader
Replicas (Preview)</a>
+ <br>
+ KIP-966 introduces Eligible Leader Replicas (ELR) in
preview in 4.0. ELR is a subset of the ISR
+ replicas guaranteed to have complete data up to the
high-watermark. ELRs are safe for leader
+ election, preventing data loss. See <a
href="https://kafka.apache.org/40/documentation.html#eligible_leader_replicas";>here</a>
for details.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/MYtEE";>KIP-996: Pre-Vote</a>
+ <br>
+ KIP-996 introduces a "Pre-Vote" mechanism to reduce
unnecessary KRaft leader elections in Apache
+ Kafka clusters. By allowing nodes to check their
eligibility for leadership before initiating an election,
+ this approach minimizes disruptions caused by network
partitions or transient issues.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/joqzDw";>KIP-950: Tiered Storage
Disablement</a>
+ <br>
+ This feature will allow customers to selectively
disable Tiered Storage on a per-topic basis.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/XA-OEg";>KIP-1076: Metrics for
client applications KIP-714 extension</a>
+ <br>
+ With KIP-714, cluster operators can now collect client
metrics directly from brokers using a plugin.
+ This makes it much easier to monitor client behavior.
However, KIP-714 only gathers metrics from
+ Kafka clients (admin, consumer, and producer) and
doesn't cover application-level metrics for
+ embedded clients, such as Kafka Streams. To fully
understand performance, it's important to collect
+ both client metrics and application-specific metrics.
This KIP introduces a way for applications that
+ use Kafka clients to include their own metrics
alongside the existing client metrics.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/NIyMEw";>KIP-1106: Add duration
based offset reset option for consumer clients</a>
+ <br>
+ Kafka consumer supports auto.offset.reset config
option, which is used when there is no initial
+ offset in Kafka or if the current offset does not
exist any more on the server. This config currently
+ supports earliest/latest/none options. Currently
consumer resets might force applications to
+ reprocess large amounts of data from earlier offsets.
With infinite storage, its beneficial to have a
+ duration based offset reset strategy. This KIP allows
applications to consume/initialise from a fixed
+ duration when there is no initial offset in Kafka.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/XoowEg";>KIP-1043: Administration of
groups</a>
+ <br>
+ KIP-848 and KIP-932 introduce new group types,
consumer and share respectively. The Admin Client
+ API has limitations when dealing with newer group
types, sometimes returning errors or incorrectly
+ stating that groups don't exist. To address this
challenge a new command-line tool, kafka-groups.sh,
+ and updates to existing command line tools,
kafka-consumer-groups.sh and kafka-share-groups.sh,
+ were made in Kafka 4.0 to enable users to view all
groups in a cluster, along with their types and
+ protocols, providing accurate information even when
the Admin Client API fails.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/w4mMEw";>KIP-1099: Extend
kafka-consumer-groups command line tool to support new consumer group</a>
+ <br>
+ This KIP extends the kafka-consumer-groups and the
kafka-share-groups command line tools to
+ provide more information related to consumer and share
groups. This helps when it comes to
+ troubleshooting consumer and share groups,
respectively introduced by KIP-848 and KIP-932.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/KYuMEw";>KIP-1102: Enable clients to
rebootstrap based on timeout or error code</a>
+ <br>
+ KIP-1102 enhances Kafka client resilience by
proactively triggering metadata rebootstrap when no
+ updates occur within a timeout period and allowing
servers to explicitly signal clients to rebootstrap.
+ This addresses previous limitations in KIP-899, where
clients could become stuck with outdated
+ metadata unless all brokers were unreachable.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/K5sODg";>KIP-896: Remove old client
protocol API versions in Kafka 4.0</a>
+ <br>
+ For the first time, old protocol API versions have
been removed. Users should ensure brokers are
+ version 2.1 or higher before upgrading Java clients
(including Connect and Streams to 4.0. Similarly,
+ users should ensure their Java clients version is 2.1
or higher before upgrading brokers to 4.0.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/y4kgF";>KIP-1124: Providing a clear
Kafka Client upgrade path for 4.x</a>
+ <br>
+ This KIP outlines the upgrade patch for Kafka Clients,
Streams and Connect to 4.0. It is a must read
+ before upgrading your clusters and clients.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/GCx4CQ";>KIP-653: Upgrade log4j to
log4j2</a>
+ <br>
+ The logging framework has been migrated from Log4j to
Log4j2. Users can use the
+ log4j-transform-cli tool to automatically convert
their existing Log4j configuration files to Log4j2
+ format.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/RomNCg";>KIP-724: Drop support for
message formats v0 and v1</a>
+ <br>
+ The message formats v0 and v1 were deprecated in
Apache Kafka 3.0. They were removed in 4.0.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/P4vOCg";>KIP-750: Drop support for
Java 8 in Kafka 4.0 (deprecate in 3.0)</a> &
+ <a
href="https://cwiki.apache.org/confluence/x/Bov5E";>KIP-1013: Drop broker and
tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)</a>
+ <br>
+ Kafka Clients and Kafka Streams require Java 11 while
Kafka Brokers, Connect and Tools now requires Java 17.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/FAqpEQ";>KIP-1030: Change
constraints and default values for various configurations</a>
+ <br>
+ KIP-1030 has changed the default values of some
configurations in order to provide better default to
+ users starting using Apache Kafka. Check out the KIP
for the details.
+ </li>
+ </ul>
+
+ <h3>Kafka Streams</h3>
+
+ <ul>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/gIuMEw";>KIP-1104: Allow Foreign Key
Extraction from Both Key and Value in KTable Joins</a>
+ <br>
+ KIP-1104 enhances Kafka Streams by allowing foreign
keys to be extracted directly from both
+ record keys and values, removing the need to duplicate
keys into values for foreign-key joins.
+ This improvement simplifies joins, reduces storage
overhead, and provides a more intuitive
+ developer experience.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/TZCMEw";>KIP-1112: allow custom
processor wrapping</a>
+ <br>
+ KIP-1112 simplifies applying cross-cutting logic in
Kafka Streams by introducing the
+ ProcessorWrapper interface, enabling seamless
injection of custom logic around Processor API
+ and DSL processors. This eliminates previous
redundancy and reduces the maintenance
+ overhead caused by manually integrating logic into
each processor individually.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/LQ6TEg";>KIP-1065: Add "retry"
return-option to ProductionExceptionHandler</a>
+ <br>
+ KIP-1065 addresses persistent errors in Kafka Streams
by allowing users to break retry loops
+ via a new "RETRY" option in the
ProductionExceptionHandler. This enhancement provides
+ customizable error handling, enabling retries,
graceful failures, or dropping problematic records
+ to continue processing efficiently.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/IgstEw";>KIP-1091: Improved Kafka
Streams operator metrics</a>
+ <br>
+ Kafka Streams exposes its metrics with KIP-1076. The
KIP especially adds a state metric for
+ each StreamThread and the client instance itself. It
also provides detailed visibility into the
+ application state.
+ </li>
+ </ul>
+
+ <h3>Kafka Connect</h3>
+
+ <ul>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/rZWzDw";>KIP-970: Deprecate and
remove Connect's redundant task configurations endpoint</a>
+ <br>
+ The endpoint <code>GET
/connectors/{connector}/tasks-config</code> added in KIP-661 was deprecated in
+ Apache Kafka 3.7 and is finally removed in 4.0. The
endpoint
+ </code>GET /connectors/{connector}/tasks</code> must
be used as a replacement.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/jA3OEg";>KIP-1074: Allow the
replication of user internal topics</a>
+ <br>
+ Currently, Mirror Maker 2 automatically excludes
topics ending in ".internal" or "-internal",
+ mistakenly identifying them as internal topics and
preventing replication of legitimate business
+ topics unless a custom replication policy is
implemented. This KIP introduces configurable
+ behavior, allowing users to replicate these topics
without needing custom code.
+ </li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/x/UwotEw";>KIP-1089: Allow disabling
heartbeats replication in MirrorSourceConnector</a>
+ <br>
+ Currently, MirrorSourceConnector always replicates
heartbeat topics, which can cause issues
+ when multiple connectors with different configurations
replicate topics between the same
+ clusters, resulting in duplicate heartbeat
replication. This KIP introduces a configurable option to
+ disable heartbeat topic replication, allowing users
greater flexibility, especially when using
+ multiple connectors with distinct configurations.
+ </li>
+ </ul>
+
+ <h3>Summary</h3>
+
+ <p>Ready to get started with Apache Kafka 4.0.0? Check out all
the details in the <a
href="https://downloads.apache.org/kafka/4.0.0/RELEASE_NOTES.html";>release
notes</a>
+ and <a
href="https://kafka.apache.org/downloads";>download</a> Apache Kafka 4.0.0.</p>
+
+ <p>This was a community effort, so thank you to everyone who
contributed to this release, including all our users and our 196
contributors:<br>
+ A. Sophie Blee-Goldman, abhi-ksolves, Abhijeet Kumar,
Abhinav Dixit, Abhishek Giri, Alieh
+ Saeedi, Almog Gavra, Alyssa Huang, Andrew Schofield, Anna
Sophie Blee-Goldman, Anshul
+ Goyal, Ao Li, Apoorv Mittal, Arnav Dadarya, Arpit Goyal,
Artem Livshits, Ayoub Omari,
+ bachmanity1, bboyleonp666, Bill Bejeck, brenden20, Bruno
Cadonna, Caio Guedes, Calvin Liu,
+ Chengyan, Cheryl Simmons, Chia Chuan Yu, Chia-Chuan Yu,
Chia-Ping Tsai, Chirag Wadhwa,
Review Comment:
`Chia Chuan Yu, Chia-Chuan Yu` also twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
