Hi Philip, Colin, Chris, Matthias, Kirk, David, Xiangyuan LI,
KIP-944 was extended a bit more to explain why effect systems like Zio
and Cats-effects make it impossible to run code on a specific thread.
I understand that using an effect system is pretty far removed from
writing Java in transa
Hello Xiangyuan LI,
I am not familiar with coroutines, nor with Kotlin. You will have to
work with the documentation:
https://kotlinlang.org/docs/coroutines-overview.html
However, I am familiar with Zio and Cats-effects (both Scala libraries).
In both Zio and Cats-effects one creates effects
Hi Erik:
I read KIP-944 and email list roughly, it seems most Java developer not
familiar with the conception of "coroutine" so cannot imagine why code of
one function without Thread.start() may run in separate threads and even
developer couldn't control it. Maybe you need a more elaborate desc
Hi David,
> Could you elaborate a bit more on why the callbacks must be ran in
another thread vs in the invoker thread?
I have been thinking on how to explain this for 2 months now and it is
not easy. It has something to do with that you cannot control what a
thread is doing if you have to a
Hi Erik,
Thanks for the KIP. I would like to better understand the motivation of
this KIP. I am not familiar with async runtimes so please excuse me if I
ask stupid questions.
Could you elaborate a bit more on why the callbacks must be ran in another
thread vs in the invoker thread? This is not c
Colin, Matthias, Chris,
I have expanded the use case description in KIP-944. I hope it is more
clear what we're trying to achieve.
https://cwiki.apache.org/confluence/x/chw0Dw
Kind regards,
Erik.
Op 22-07-2023 om 17:23 schreef Erik van Oosten:
Hello Chris,
Thanks for elaborating Matth
Hi Erik,
I'm glad that this alternative is agreeable to you! Just to be clear
though, it does still technically require a KIP since it's still a change
to public API (just a much smaller one). And if we do decide to pursue this
approach, we'll have to define the expected contract for this method s
Hi all,
I have created https://github.com/apache/kafka/pull/14071 to implement
Chris' idea.
Kind regards,
Erik.
Op 22-07-2023 om 16:39 schreef Chris Egerton:
Hi Erik,
I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka
Hello Chris,
Thanks for elaborating Matthias' words. Apparently the use case
description is too terse. Indeed, that is not FUD and that is something
I can work with.
> It's also worth mentioning that what's proposed in the KIP is only
blocked by the private access modifier on the KafkaConsum
Hi Erik,
I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals and the
Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code snippet o
Hi Matthias,
I am getting a bit frustrated here. All the concerns and questions I
have seen so far are addressed in KIP-944.
Please let me know if they are not clear enough, but please do not come
with FUD.
Kind regards,
Erik.
Op 21-07-2023 om 21:13 schreef Matthias J. Sax:
I am not a
I am not a clients (or threading) expert, but I tend to agree to Colin's
concerns.
In particular, it would be nice to see an example how you intent to use
the API (I am not familiar with Kotlin or it's co-routins), to better
understand what this changes help to solve to begin with.
Opening u
Hi Erik,
Thanks for the KIP!
I empathize with your frustration over the radio silence. It gets like that
sometimes, and I apologize for my lack of feedback.
I’d personally like to see this lively exchange move over to the DISCUSS thread
you’d created before.
Thanks,
Kirk
> On Jul 14, 2023, a
Hi Colin,
The way I understood Philp's message is that KIP-944 also plays nice
with KIP-945. But I might be mistaken.
Regardless, KIP-945 does /not/ resolve the underlying problem (the need
for nested consumer invocations) because it has the explicit goal of not
changing the user facing API.
HI Philip & Erik,
Hmm... if we agree that KIP-945 addresses this use case, I think it would be
better to just focus on that KIP. Fundamentally it's a better and cleaner model
than a complex scheme involving thread-local variables. I really don't want to
be debugging complex interactions between
Hi Colin, Philip,
I have added a section to KIIP-944 to address your concerns around
memory consistency over multiple threads.
You can read them here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-944%3A+Support+async+runtimes+in+consumer#KIP944:Supportasyncruntimesinconsumer-Threadsa
Thanks Philip,
> I think this can be demonstrated via diagrams and some code in the KIP.
There are some diagrams in KIP-944. How can they be improved?
I will add some code to address the concerns around memory barriers.
> We are in-process of re-writing the KafkaConsumer
Nice! I will read th
Hey Erik - Another thing I want to add to my comment is. We are in-process
of re-writing the KafkaConsumer, and I think your proposal would work in
the new consumer because we are going to separate the user thread and the
background thread. Here is the 1-pager, and we are in process of
converting
Hey Erik,
Sorry for holding up this email for a few days since Colin's response
includes some of my concerns. I'm in favor of this KIP, and I think your
approach seems safe. Of course, I probably missed something therefore I
think this KIP needs to cover different use cases to demonstrate it doe
Hello Colin,
>> In KIP-944, the callback thread can only delegate to another thread
after reading from and writing to a threadlocal variable, providing the
barriers right there.
> I don't see any documentation that accessing thread local variables
provides a total store or load barrier. Do y
On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote:
> Hi Colin,
>
> Thanks for your thoughts and taking the time to reply.
>
> Let me take away your concerns. None of your worries are an issue with
> the algorithm described in KIP-944. Here it goes:
>
> > It's not clear ot me that it's safe to
Hi Colin,
Thanks for your thoughts and taking the time to reply.
Let me take away your concerns. None of your worries are an issue with
the algorithm described in KIP-944. Here it goes:
> It's not clear ot me that it's safe to access the Kafka consumer or
producer concurrently from different
Hi Erik,
It's not clear ot me that it's safe to access the Kafka consumer or producer
concurrently from different threads. There are data structures that aren't
protected by locks, so I wouldn't necessarily expect accessing and mutating
them in a concurrent way to work. This is true even if the
Dear PMCs,
So far there have been 0 responses to KIP-944. I understand this may not
be something that keeps you busy, but this KIP is important to people
that use async runtimes like Zio, Cats and Kotlin.
Is there anything you need to come to a decision?
Kind regards,
Erik.
Op 05-07-20
24 matches
Mail list logo
