Hi,
thanks for reporting this issue. I did look into it, and it is indeed a
bug: https://issues.apache.org/jira/browse/KAFKA-18478
The workaround should be to set the StreamsConfig value serde to the
stream-side value serde to make it work.
-Matthias
On 1/10/25 5:36 AM, Hagiu Alexandru wr
Hello,
Kind reminder :)
Also, I want to add that I am trying to add a grace period for my
kstream-ktable join, but I encounter an exception.
It seems the Serde that I am providing in the join description is not taken
into consideration when creating the *RocksDBTimeOrderedKeyValueBuffer*.
*https
Seems this question was cross-posted on SO:
https://stackoverflow.com/questions/79299571/inconsistent-results-with-kstream-ktable-join-under-load
Left an answer there.
On 12/28/24 11:45 PM, Hagiu Alexandru wrote:
Hello,
I'm developing a Quarkus microservice that utilizes Kafka Streams to
proc
Hi Matthias,
Yes, it's just a matter of adding the [DISCUSS] prefix in the subject.
By the way, I didn't say this won't need a KIP, just that I won't be
pushing for it, but other maintainers might think it's needed.
For the discuss thread, you should write down what changes in the build and
what
Hey Josep,
I'm glad you agree that a KIP is not needed here, and I agree with you that
how to publish these artifacts should be discussed with the Kafka team. In
fact, this is what I created this thread for π This is my first time
contributing to Kafka, so I'm going to have to ask what a DISCUSS t
Hi Matthias,
It's not adding a new functionality but it's changing the way to generate
artifacts. In the end we are talking about generating a new binary.
I could live with not having a KIP, but a DISCUSS thread I think it's
necessary. This signals the community members and maintainers that their
Hi Matthias, Hi Josep,
I'm afraid I can't do the KIP thing as the signup process for Apache
Confluence requires sending me a password reset link via E-Mail and said
E-Mail doesn't seem to reach me for some reason. I've contacted the Apache
infrastructure team but haven't yet heard back from them.
Josep,
thanks for helping with this. I was also thinking if we might need a KIP
for this change. Since you had the same though, I would say, yes, let's
do a KIP.
@Matthias: can you prepare a KIP? You can read up on the details on the
wiki page:
https://cwiki.apache.org/confluence/display/KA
Hey Josep et al,
I've created a ticket regarding this.
https://issues.apache.org/jira/browse/KAFKA-16237
All the best,
Matthias
Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
:
>
> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
> your accounts are created and w
Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
your accounts are created and we'll properly set them up so you can create
and assign tickets to you.
Best,
On Thu, Feb 8, 2024 at 11:32β―AM Matthias Berndt
wrote:
> Thanks Josep, I've applied for a JIRA account and addr
Thanks Josep, I've applied for a JIRA account and addressed your
review comments.
Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
:
>
> Hi Matthias,
>
> I think for this particular case it would be worth creating a JIRA ticket
> for this as it's a new "feature".
> Regarding the change itself,
Hi Matthias,
I think for this particular case it would be worth creating a JIRA ticket
for this as it's a new "feature".
Regarding the change itself, I think we need to clarify how the release
process would work. Right now, the script `gradlewAll` is used (which
basically runs the build with Scala
Hi Matthias J., Hi Lucas, Hi Josep,
Thank you for your encouraging responses regarding a Scala 3 port of
Kafka-Streams-Scala, and apologies for the late response from my side.
I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
retaining support for 2.13 and 2.12). Almost no chan
Hi,
For reference, prior work on this:
https://github.com/apache/kafka/pull/11350
https://github.com/apache/kafka/pull/11432
Best,
On Thu, Feb 1, 2024, 15:55 Lucas Brutschy
wrote:
> Hi Matthiases,
>
> I know Scala 2 fairly well, so I'd be happy to review changes that add
> Scala 3 support. How
Hi Matthiases,
I know Scala 2 fairly well, so I'd be happy to review changes that add
Scala 3 support. However, as Matthias S. said, it has to be driven by
people who use Scala day-to-day, since I believe most Kafka Streams
committers are working with Java.
Rewriting the tests to not use Embedded
Thanks for raising this. The `kafka-streams-scala` module seems to be an
important feature for Kafka Streams and I am generally in favor of your
proposal to add Scala 3 support. However, I am personally no Scala
person and it sounds like quite some overhead.
If you are willing to drive and own
`KGroupedStream` is just an "intermediate representation" to get a
better flow in the DSL. It's not a "top level" abstraction like
KStream/KTable.
For `KTable` there is `transformValue()` -- there is no `transform()`
because keying must be preserved -- if you want to change the keying you
ne
Hi Daan,
First of all, it does sound like that is a correct
implementation of QueryableStoreProvider. Kudos for taking
that on; the complexity of that API was one of my top
motivations for replacing it with IQv2!
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-796%3A+Interactive+Query+v2
)
Guozhang - thank you for your thoughts.
You are right - this is more about the producer client than the streams
client.
caching the metadata outside producer, e.g. in an admin client would not
> be a perfect solution since in either way your metadata cache inside the
> producer or inside the admi
Hello,
Thanks for reaching out to the community for this. I think (maybe you've
also suggested) it is rather an observation on producer client than on
streams client. Generally speaking we want to know if we can fail fast if
the metadata cannot be found in producer.send() call. And here are my
tho
I am not sure if I understand your scenario. What do you exactly mean by
"back ground job" and by "sweeps the records in the store using range
calls" ?
As you noted, only the `Processor` that have access to the store should
write/delete anything from it. You should not try to modify the store
from
> This leads to lockexception in
>> consumer2 and consumer2 remains in livelock to create state directories for
>> those two partitons.
There is a fix for a `LockException` during rebalance in 0.11.0.1:
https://issues.apache.org/jira/browse/KAFKA-5167
Maybe upgrading helps? Note, that you can upg
It is correct, that Kafka Streams does not create input/output/through()
topics. You should create those topics upfront.
> We've *set auto.create.topics.enable=true *in our kafka
>> server properties file and we'd expect that the topics get created, but the
>> consumer goes into SHUTDOWN mode and
After adding more logging details, we found that we hadn't created topics
yet for the consumer & producer for the streams application. We've added a
check when starting up to verify that the topics exist, otherwise we exit
our app. We're not dynamically creating topics and we want to create them
up
Does the application transit to RUNNING state? Also check
`KafkaStreams#localThreadsMetadata()` what tasks are assigned?
You might also enable DEBUG logs for
`org.apache.kafka.clients.consumer.**` classes to see if the consumer
sends fetch request to the broker.
-Matthias
On 6/11/19 7:03 PM, Br
The application just hangs (we let it sit for ~1 hour, small dataset as
we're testing), we can restart it listening to 1 of the 3 topics we start
it with and it chugs along, no problem. The same code is executed as
separate application.ids listening to other topics without any issues.
We'll try to
What do you exactly observe?
- Does the application rebalance correctly?
- Does it start processing?
- Anything in the logs about the status of the application?
-Matthias
On 6/10/19 6:19 PM, Brian Putt wrote:
> Hello,
>
> I'm working with the kafka streams api and am running into issues whe
Hi Jay,
I'm not aware of a Go implementation of Kafka Streams in Apache Kafka. If
anyone here knows something I don't, feel free to correct me.
goka (https://github.com/lovoo/goka), while not part of Apache Kafka, seems
to be pretty popular. You may also want to check out KSQL which, although
it'
If your producer is using transactions, it's possible that the producer was
killed in the middle of a transaction, in which case any un-committed
records would be logged on the broker but skipped by downstream consumers.
Otherwise, it's likely that the leader for the partition crashed before the
r
Thanks, I understand how consumer works. The main question is related to
why the join did not work and how it happened that only one message
remained uncommitted.
ΠΏΡ, 15 ΠΌΠ°Ρ. 2019 Π³. Π² 16:29, Ryanne Dolan :
> Hello! When using exactly-once semantics, uncommitted or aborted records
> are skipped b
Hello! When using exactly-once semantics, uncommitted or aborted records
are skipped by the consumer as if they don't exist.
When inspecting the topic manually, use isolation.level=read_committed to
get the same behavior.
Ryanne
On Fri, Mar 15, 2019, 6:08 AM Π€Π΅Π΄ΠΎΡ Π§Π΅ΡΠ½ΠΈΠ»ΠΈΠ½
wrote:
> I also noti
I also noticed another important thing now. Message which used for join is
uncommitted. I understood it with the help of consumer's setting isolation
level - read_committed. The message got into the topic using the same
stream app. Remind that stream app has processing guarantee
= exactly_once. How
>From the release plan:
" While the target release date is fixed at ~2w after code freeze, RCs
will roll out as needed until the release vote passes"
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
You can follow the voting threads on this mailing list - the current t
e Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________
> From: Damian Guy
> Sent: Tuesd
From: Damian Guy
Sent: Tuesday, August 1, 2017 2:11 PM
To: dev@kafka.apache.org
Subject: Re: Kafka Streams debugging with "no fluent" API choice
Hi Paolo,
The change would require a KIP as it is a public API change. I don't see
any harm in making the
<http://paolopatierno.wordpress.com/>
>
>
>
> From: Damian Guy
> Sent: Tuesday, August 1, 2017 12:11 PM
> To: dev@kafka.apache.org
> Subject: Re: Kafka Streams debugging with "no fluent" API choice
>
> I don't know s
ev@kafka.apache.org
Subject: Re: Kafka Streams debugging with "no fluent" API choice
I don't know specifically why this is removed, however if you want to get
the same functionality you can use peek, i.e:
stream.map(...).peek(...).filter(..)
You can log the key values out in t
I don't know specifically why this is removed, however if you want to get
the same functionality you can use peek, i.e:
stream.map(...).peek(...).filter(..)
You can log the key values out in the peek call.
On Tue, 1 Aug 2017 at 11:48 Paolo Patierno wrote:
> Hi guys,
>
>
> I was thinking about
I thought, we drop records with null key? No?
-Matthias
On 6/23/17 12:25 AM, Damian Guy wrote:
> My guess is it is because the record doesn't have a key, i.e., the key is
> null. We have a fix for this in 0.11, in that we will skip records with a
> null key during restore.
>
> On Fri, 23 Jun 201
My guess is it is because the record doesn't have a key, i.e., the key is
null. We have a fix for this in 0.11, in that we will skip records with a
null key during restore.
On Fri, 23 Jun 2017 at 03:57 Matthias J. Sax wrote:
> Hi,
>
> can you reproduce the error reliably? Are use using 0.10.2.0
Hi,
can you reproduce the error reliably? Are use using 0.10.2.0 or 0.10.2.1?
It's unclear to me, how an NPE can occur. It seems to happen within
Streams library. Might be a bug. Not sure atm.
-Matthias
On 6/22/17 9:43 AM, Shekar Tippur wrote:
> Hello,
>
> I am trying to perform a simple join
Your observation is correct.
The paragraph you quote is not very precise but also not necessarily
wrong. The example is simplified and assumes that there is no
re-partitioning even if it is not mentioned explicitly.
-Matthias
On 6/20/17 9:32 AM, Paolo Patierno wrote:
> Hi devs,
>
>
> at foll
p://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
From: Michal Borowiecki
Sent: Sunday, June 18, 2017 9:34 AM
To: dev@kafka.apache.org; Jay Kreps
Cc: us...@kafka.apache.org; Matthias J. Sax
Subject: Re: Kafka Streams vs Spark
DevExperience<http://paolopatierno.wordpress.com/>
<http://paolopatierno.wordpress.com/>
From: Eno Thereska <mailto:eno.there...@gmail.com>
Sent: Thursday, June 15, 2017 3:57 PM
To:us...@kafka.apache.org <mailto:us..
> your article ! Thanks !
>>>
>>>
>>> Paolo
>>>
>>>
>>> Paolo Patierno
>>> Senior Software Engineer (IoT) @ Red Hat
>>> Microsoft MVP on Windows Embedded & IoT
>>> Microsoft Azure Advisor
>>>
>>> Twitter
o even because the
> usage is pretty simple but this scenario could have a lack against Spark.
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> T
o even because the
> usage is pretty simple but this scenario could have a lack against Spark.
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> T
!
Paolo
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopati
Kyle,
What's your apache id? I can grant you the permission.
Guozhang
On Sat, Apr 29, 2017 at 7:33 AM, Kyle Winkelman
wrote:
> I don't seem to have permission. When logged in I can neither edit the
> main page nor create an additional KIP.
>
> Thanks,
> Kyle
>
> On Thu, Apr 27, 2017 at 12:35
I don't seem to have permission. When logged in I can neither edit the main
page nor create an additional KIP.
Thanks,
Kyle
On Thu, Apr 27, 2017 at 12:35 PM, Eno Thereska
wrote:
> Hi Kyle,
>
> I believe Guozhang has now given you permission to edit the KIP wiki at
> https://cwiki.apache.org/con
Hi Kyle,
I believe Guozhang has now given you permission to edit the KIP wiki at
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals.
Could you see if you can add this there?
Many thanks
Eno
On Wed, Apr 26, 2017 at 6:00 PM, Kyle Winkelman
wrote:
> Thank you for your r
Thank you for your reply.
I have attached my first attempt at writing a KIP and I was wondering if
you could review it and share your thoughts.
Going forward I would like to create this KIP. I was wondering whom I
should ask to get the necessary permissions on the wiki. Username:
winkelman.kyle
Hi Kyle,
Sorry for the delay in replying. I think it's worth doing a KIP for this
one. One super helpful thing with KIPs is to list a few more scenarios that
would benefit from this approach. In particular it seems the main benefit
is from reducing the number of state stores. Does this necessarily
Eno,
Thanks for the response. The figure was just a restatement of my questions.
I have made an attempt at a low level processor and it appears to work but
it isn't very pretty and was hoping for something at the streams api level.
I have written some code to show an example of how I see the Cogro
Hi Kyle, (cc-ing user list as well)
This could be an interesting scenario. Two things to help us think through it
some more: 1) it seems you attached a figure, but I cannot seem to open it. 2)
what about using the low level processor API instead of the DSL as approach 3?
Do you have any thought
Hi Misha,
If it works locally, then I would still suspect that it is due to a
transient timing issue: note that the create-topic script is non-blocking,
i.e. even when it returns it does not necessarily guarantee that the leader
metadata information has been completed propagating to brokers.
Gu
Thanks Guozhang,
Yes, I rely on auto-create, and it works locally. Maybe I need to tweak some
timeout conf for that?
Also I identified, that even I manually create a topic, it lists but I can not
produce messages to this topic with the same exception.
Producing to other topics works well, so it
Misha,
Did you pre-create the sink topic before starting your application or you
are relying on the broker-side auto-create for that topic?
If you are relying on auto-create, then there is a transient period where
the topic is created but the metadata has not been propagated to the
brokers so the
Also, in the next version of Kafka / Kafka Streams such "intermediate"
topics will automatically be created for you when you do joins or
aggregations:
https://issues.apache.org/jira/browse/KAFKA-3561
So my previous message explained your options today when using the current
release of Kafka Strea
Poul,
to add to what Matthias said: If you are wondering how to manually create
a topic, you have basically two options.
A. Use Kafka's CLI tools to create the topic "from the outside".
# Example
$ kafka-topics.sh --create --topic my-custom-toipc --zookeeper
localhost:2181 --partitions
Hi,
you can manually create a topic with the number of partitions you want
to have and use this topic via through()
KStream input = ...
input.map().through("manually-created-topic").join(...)
However, both KStream and KTable need to have the same number of
partitions for perform the join. Thus,
Dirk,
we included the note "be careful when using Kafka Streams in production"
because Kafka Streams as shipped in Kafka 0.10.0.0 is the first-ever
release of Kafka Streams. In practice, users are running Streams
applications in a variety of stages -- some are doing pilots or
evaluations, some ar
Thanks Matthias!
Got it working with through, still use 0.10.0.0 version.
> On Jun 29, 2016, at 22:42, Matthias J. Sax wrote:
>
> Hi,
>
> for joins, data of both inputs must be co-located, ie, partitioned on
> the same key and have the same number of partitions:
>
> See "Note" box at:
> http:
Hi,
for joins, data of both inputs must be co-located, ie, partitioned on
the same key and have the same number of partitions:
See "Note" box at:
http://docs.confluent.io/3.0.0/streams/developer-guide.html?highlight=join#joining-streams
From an older email thread about the same issue:
>>> If yo
Hi Alexander,
I haven't tried Kafka Streams on Windows but did notice that Microsoft has
merged code into github to make RocksDB available on Windows. Perhaps this is
useful:
https://blogs.msdn.microsoft.com/bingdevcenter/2015/07/22/open-source-contribution-from-bing-rocksdb-is-now-available-in-
Great, thanks:
https://issues.apache.org/jira/browse/KAFKA-3619
On Mon, Apr 25, 2016 at 11:13 AM, Guozhang Wang wrote:
> Thanks for reporting that Greg.
>
> I have just added you as a Kafka contributor so you should be able to
> create JIRAs now. Could you create a new JIRA for this and change
Thanks for reporting that Greg.
I have just added you as a Kafka contributor so you should be able to
create JIRAs now. Could you create a new JIRA for this and change the RB
title with the JIRA number?
Thanks!
Guozhang
On Mon, Apr 25, 2016 at 12:13 AM, Greg Fodor wrote:
> Apologies for a sec
Hello Maria,
We have some thoughts about supporting finer grained flow controls in Kafka
Streams https://issues.apache.org/jira/browse/KAFKA-3478 as part of a big
effort to improve re-processing user experience, which covers this use
case. We are shooting to have this post 0.10.0.0.
As for now, o
Also, have you looked at Kafka Connect released in 0.9? It has a MySQL
binlog connector implementation in progress, just thinking maybe you would
be interested to check it out and see if there are any feedbacks that you
want to give.
https://github.com/wushujames/kafka-mysql-connector
Guozhang
O
Hello Ben,
1. Currently Kafka Streams high-level DSL does not take the
StreamPartitioner yet, please feel free to file a JIRA so that we can keep
track and discuss of whether / how to incorporate it into Kafka Streams DSL.
2. As for now, you can do two work arounds:
1) use `process()` function t
Thanks Bill,
The example code looks great! I will read through it, and am expecting your
high-level examples already!
Guozhang
On Sat, Mar 5, 2016 at 12:39 PM, Bill Bejeck wrote:
> Oops, forgot that little bit of information -
> http://codingjunkie.net/kafka-processor-part1/
>
> A review wou
Oops, forgot that little bit of information -
http://codingjunkie.net/kafka-processor-part1/
A review would be greatly appreciated to make sure I'm on track.
Thanks
On Sat, Mar 5, 2016 at 3:12 PM, Neha Narkhede wrote:
> Great. Mind sharing the draft or linking to the post if already publishe
Great. Mind sharing the draft or linking to the post if already published?
Happy to review if that is useful.
On Sat, Mar 5, 2016 at 12:07 PM, Bill Bejeck wrote:
> All,
>
> I'm pretty excited about kafka-streams and I've written a blog (first of
> 2-3) on the subject, and I'd thought I'd share.
73 matches
Mail list logo
