Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

[Ismael Juma]

Perhaps we can start a wiki page where we collect these ideas as a
precursor to a KIP for record format v3?

Ismael

On Mon, May 15, 2023, 8:19 PM Luke Chen  wrote:

> Hi Divij and Ismael,
>
> Thanks for your great comments.
> Yes, I know record format changes are _extremely expensive_ for the
> ecosystem.
> But on the other hand, it's not clear "what kind of change" is worth
> changing it.
> That's why I posted the KIP for discussion.
>
> It looks like the benefit of this KIP is still not strong enough, and we
> have more further changes planned for message format v3.
> I'll move this KIP into "discarded" state and add some reasons there.
> Please remember to take this KIP (and Divij's proposal) into consideration
> when we plan to propose a new message format.
>
> Thank you.
> Luke
>
>
> On Mon, May 15, 2023 at 10:55 PM Ismael Juma  wrote:
>
> > Hi Luke,
> >
> > Thanks for the KIP. A few things:
> >
> > 1. Record format changes are _extremely expensive_ for the ecosystem, so
> we
> > need to have very strong motivation for them. There is a reason why we
> have
> > had so few of them and the last one was in 0.11.
> > 2. It was a conscious decision to make the record header fixed size - it
> > would be a lot more complicated to set some of the fields after writing
> the
> > actual records otherwise. If we want the record header to be variable
> size,
> > then we would probably want to move some fields to a "trailer".
> > 3. v3 of the record format should make it cheaper to make changes in the
> > future (perhaps it could support tagged fields or similar)
> > 4. We'd want to fix other known issues at the same time (eg log append
> time
> > should always be available, there may be others)
> > 5. We should consider whether we would want to introduce a user header
> that
> > is at the batch level vs record level for efficiency reasons
> >
> > Ismael
> >
> > On Fri, May 12, 2023 at 12:04 AM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > message attribute field. This KIP is to add a flag in the batch header
> to
> > > indicate if messages inside the batch have attribute field or not, to
> > > reduce the message size, thus, save network traffic and storage size
> (and
> > > money, of course).
> > >
> > > Please check the link for more detail:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > >
> > > Any feedback is welcome.
> > >
> > > Thank you.
> > > Luke
> > >
> >
>

[jira] [Created] (KAFKA-14998) Kafka server start fails with the following error java.lang.NumberFormatException: For input string: "a"

[Nikilesh (Jira)]

Nikilesh created KAFKA-14998:


 Summary: Kafka server start fails with the following error 
java.lang.NumberFormatException: For input string: "a"
 Key: KAFKA-14998
 URL: https://issues.apache.org/jira/browse/KAFKA-14998
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.5.1
Reporter: Nikilesh


Kafka fails to restart because of the following error. The Kafka cluster was in 
a healthy state all along in a kubernetes environment and this issue started 
happening as part of upgrade to k8s cluster and no changes to kafka as such in 
terms of configuration.

 

 
{code:java}
[2023-05-16 02:29:32,159] ERROR [KafkaServer id=0] Fatal error during 
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.NumberFormatException: For input string: "a"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at scala.collection.immutable.StringLike.toLong(StringLike.scala:309)
at scala.collection.immutable.StringLike.toLong$(StringLike.scala:309)
at scala.collection.immutable.StringOps.toLong(StringOps.scala:33)
at kafka.log.Log$.offsetFromFileName(Log.scala:2568)
at kafka.log.Log$.offsetFromFile(Log.scala:2572)
at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:597)
at 
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
at kafka.log.Log.loadSegmentFiles(Log.scala:586)
at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:697)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2329)
at kafka.log.Log.loadSegments(Log.scala:691)
at kafka.log.Log.(Log.scala:297)
at kafka.log.Log$.apply(Log.scala:2463)
at kafka.log.LogManager.loadLog(LogManager.scala:272)
at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:342)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Feature Discussion: Enhancing Kafka Connect's transformation interface to 1 -> Many / Many -> Many

[Rabin Banerjee]

Hi Team, 

We have a few use cases where nested record with array needs to be flattened / 
exploded to multiple rows and write it to destination like Cassandra. 

Now current Transformation interface in Kafka connect takes one record and 
returns one / zero record.
Also current flatten ignores the array 
https://issues.apache.org/jira/browse/KAFKA-12305.

We would like to know about your thought on enhancing the transformation 
interface to produce more than one record , 1 -> Many or Many -> Many. 

We understand an alternative is to use an upstream KStream pipeline but that 
has multiple challenges like adding extra hop, more pipelines to maintain etc.

Enhancing the Transformation interface would allow us to have a Generic SMT to 
handle this like  Explode, similar to 
https://github.com/apache/spark/blob/007c4593058537251d83a4cb44efe31e394aee22/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L4542
Thanks 
Rabin

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #133

[Apache Jenkins Server]

See 

Re: Feature Discussion: Enhancing Kafka Connect's transformation interface to 1 -> Many / Many -> Many

[Chris Egerton]

Hi Rabin,

I like this idea but it presents challenges for delivery semantics,
especially (though not exclusively) with sink connectors.

If a sink connector uses the SinkTask::preCommit method [1] to explicitly
notify the Connect runtime about which offsets are safe to commit, it may
mistakenly report that an offset is safe to commit after successfully
writing a single record with that offset to the sink system, even if there
a 1->many transformation has created multiple records with that offset.

If a sink connector doesn't use that method, then we'd also have to do some
extra bookkeeping in the Connect runtime to track progress within a batch
of records that were all derived from a single original record in Kafka. We
currently use consumer group offsets to track sink connector progress
(unless the connector chooses to use an alternative mechanism), which
doesn't support any finer granularity than one record at a time, so there
would have to be some non-trivial design work to figure things out on that
front.

There's also a question of how we'd want to apply this across entire chains
of transformations. You mention the possibility of many->many; does this
mean that there would be an opt-in interface for accepting an entire batch
of records that was produced by a prior transformation?

Overall my sense of this feature so far has been that there are a lot of
edge cases that can bite our users if we're not careful about addressing
them with clever design work, which is at least part of the reason that we
haven't implemented it yet. I'd be interested in your thoughts about how we
can make this feature safe and easy to use without adding footguns for
existing connectors and deployments.

[1] -
https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/sink/SinkTask.html#preCommit(java.util.Map)

Cheers,

Chris

On Tue, May 16, 2023 at 6:26 AM Rabin Banerjee
 wrote:

> Hi Team,
>
> We have a few use cases where nested record with array needs to be
> flattened / exploded to multiple rows and write it to destination like
> Cassandra.
>
> Now current Transformation interface in Kafka connect takes one record and
> returns one / zero record.
> Also current flatten ignores the array
> https://issues.apache.org/jira/browse/KAFKA-12305.
>
> We would like to know about your thought on enhancing the transformation
> interface to produce more than one record , 1 -> Many or Many -> Many.
>
> We understand an alternative is to use an upstream KStream pipeline but
> that has multiple challenges like adding extra hop, more pipelines to
> maintain etc.
>
> Enhancing the Transformation interface would allow us to have a Generic
> SMT to handle this like  Explode, similar to
> https://github.com/apache/spark/blob/007c4593058537251d83a4cb44efe31e394aee22/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L4542
> Thanks
> Rabin

Re: [DISCUSS] Re-visit end of life policy

[Igor Soarez]

My impression is also that a lot of users run older,
out of EOL, versions of Kafka.

The final 3.x version is particularly concerning, as it will be
the last bridge to migrate away from ZK. If a big portion of users
only upgrade after its EOL period, we might only then discover an
important bug and potentially be cutting off the migration path (or
just making it really difficult) for a lot of users.

Could the release efforts be re-shuffled perhaps to allow a longer
bug fix policy for the last minor release in each release version?
Or does the KRaft migration perhaps merit an exceptional case?

--
Igor

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Kamal Chandraprakash]

Hi,

Inline with the *backup-only* replica approach, KIP-491: Preferred Leader
Deprioritized List

was
proposed to mark a replica as the lowest priority to become leader.




On Sat, May 13, 2023 at 6:53 AM 67  wrote:

> Hi Luke,
>
>
> It's a good point that add this config and get better P99 latency, but is
> this changing the meaning of "in sync replicas"? consider a situation with
> "replica=3 acks=2", when two broker fail and left only the broker that
> does't have the message, it is in sync, so will be elected as leader, will
> it cause a NOT NOTICED lost of acked messages?
>
>
> qiangLiu
>
>
>
>
> -- 原始邮件 --
> 发件人:
>   "dev"
> <
> ism...@juma.me.uk>;
> 发送时间: 2023年5月10日(星期三) 中午1:28
> 收件人: "dev"
> 主题: Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config
>
>
>
> Hi Luke,
>
> As discussed in the other KIP, there are some subtleties when it comes to
> the semantics of the system if we don't wait for all members of the isr
> before we ack. I don't understand why you say the leader election question
> is out of scope - it seems to be a core aspect to me.
>
> Ismael
>
>
> On Wed, May 10, 2023, 8:50 AM Luke Chen 
> > Hi Ismael,
> >
> > No, I didn't know about this similar KIP! I hope I've known that so
> that I
> > don't need to spend time to write it again! :(
> > I checked the KIP and all the discussions (here
> > <
> https://lists.apache.org/list?dev@kafka.apache.org:gte=100d:KIP-250>;).
> I
> > think the consensus is that adding a client config to `acks=quorum` is
> > fine.
> > This comment
> > ;
> from
> > Guozhang pretty much concluded what I'm trying to do.
> >
> >
> >
> >
> >
> >
> >
> >
> > *1. Add one more value to client-side acks config:   0: no
> acks needed at
> > all.   1: ack from the leader.   all: ack from
> ALL the ISR replicas
> >  quorum: this is the new value, it requires ack from enough
> number of ISR
> > replicas no smaller than majority of the replicas AND no smaller
> > than{min.isr}.2. Clarify in the docs that if a user wants to tolerate
> X
> > failures, she needs to set client acks=all or acks=quorum (better tail
> > latency than "all") with broker {min.sir} to be X+1; however, "all"
> is not
> > necessarily stronger than "quorum".*
> >
> > Concerns from KIP-250 are:
> > 1. Introducing a new leader LEO based election method. This is not
> clear in
> > the KIP-250 and needs more discussion
> > 2. The KIP-250 also tried to optimize the consumer latency to read
> messages
> > beyond high watermark, which also has some discussion about how to
> achieve
> > that, and no conclusion
> >
> > Both of the above 2 concerns are out of the scope of my current KIP.
> > So, I think it's good to provide this `acks=quorum` or
> > `acks=min.insync.replicas` option to users to give them another
> choice.
> >
> >
> > Thank you.
> > Luke
> >
> >
> > On Wed, May 10, 2023 at 8:54 AM Ismael Juma  wrote:
> >
> > > Hi Luke,
> > >
> > > Are you aware of
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-250+Add+Support+for+Quorum-based+Producer+Acknowledgment
> >
> >;
> > ?
> > >
> > > Ismael
> > >
> > > On Tue, May 9, 2023 at 10:14 PM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion for the KIP-926: introducing
> > > > acks=min.insync.replicas config. This KIP is to introduce
> > > > `acks=min.insync.replicas` config value in producer, to
> improve the
> > write
> > > > throughput and still guarantee high durability.
> > > >
> > > > Please check the link for more detail:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> >
> >;
> > >
> > > > Any feedback is welcome.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >

Re: [VOTE] KIP-927: Improve the kafka-metadata-quorum output

[Divij Vaidya]

+1 (non binding)

Divij Vaidya



On Tue, May 16, 2023 at 4:35 AM ziming deng 
wrote:

> Thanks for this improvement, +1 from me(binging)
>
> —
> Best,
> Ziming
>
> > On May 16, 2023, at 00:43, Federico Valeri  wrote:
> >
> > Hi all,
> >
> > I'd like to start a vote on KIP-927: Improve the kafka-metadata-quorum
> output.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> >
> > Discussion thread:
> > https://lists.apache.org/thread/pph59hxvz5jkk709x53p44xrpdqwv8qc
> >
> > Thanks
> > Fede
>
>

Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field

[Luke Chen]

Good idea!
I've created a wiki for the ideas for message format v.3, and added the
link in this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3

Thanks.
Luke

On Tue, May 16, 2023 at 4:30 PM Ismael Juma  wrote:

> Perhaps we can start a wiki page where we collect these ideas as a
> precursor to a KIP for record format v3?
>
> Ismael
>
> On Mon, May 15, 2023, 8:19 PM Luke Chen  wrote:
>
> > Hi Divij and Ismael,
> >
> > Thanks for your great comments.
> > Yes, I know record format changes are _extremely expensive_ for the
> > ecosystem.
> > But on the other hand, it's not clear "what kind of change" is worth
> > changing it.
> > That's why I posted the KIP for discussion.
> >
> > It looks like the benefit of this KIP is still not strong enough, and we
> > have more further changes planned for message format v3.
> > I'll move this KIP into "discarded" state and add some reasons there.
> > Please remember to take this KIP (and Divij's proposal) into
> consideration
> > when we plan to propose a new message format.
> >
> > Thank you.
> > Luke
> >
> >
> > On Mon, May 15, 2023 at 10:55 PM Ismael Juma  wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for the KIP. A few things:
> > >
> > > 1. Record format changes are _extremely expensive_ for the ecosystem,
> so
> > we
> > > need to have very strong motivation for them. There is a reason why we
> > have
> > > had so few of them and the last one was in 0.11.
> > > 2. It was a conscious decision to make the record header fixed size -
> it
> > > would be a lot more complicated to set some of the fields after writing
> > the
> > > actual records otherwise. If we want the record header to be variable
> > size,
> > > then we would probably want to move some fields to a "trailer".
> > > 3. v3 of the record format should make it cheaper to make changes in
> the
> > > future (perhaps it could support tagged fields or similar)
> > > 4. We'd want to fix other known issues at the same time (eg log append
> > time
> > > should always be available, there may be others)
> > > 5. We should consider whether we would want to introduce a user header
> > that
> > > is at the batch level vs record level for efficiency reasons
> > >
> > > Ismael
> > >
> > > On Fri, May 12, 2023 at 12:04 AM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > > message attribute field. This KIP is to add a flag in the batch
> header
> > to
> > > > indicate if messages inside the batch have attribute field or not, to
> > > > reduce the message size, thus, save network traffic and storage size
> > (and
> > > > money, of course).
> > > >
> > > > Please check the link for more detail:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > >
> > > > Any feedback is welcome.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >
>

[VOTE] 3.4.1 RC0

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0. A
few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK session
timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
- connect
KAFKA-12558  MM2 may not
sync partition offsets correctly
KAFKA-14666  MM2 should
translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug: State
stores lose state when tasks are reassigned under EOS

Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by May 24, 2023
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc0

* Documentation:
https://kafka.apache.org/34/documentation.html

* Protocol:
https://kafka.apache.org/34/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/133/

System tests:
Will update the results later

Thank you.
Luke