Thanks for the KIP Peter! Curious to see where this one goes, I think it's
good to start a discussion around this though perhaps we'll need to split
it up into more focused improvements as there's a lot bundled in this one
idea!
A0. I'd like to see some folk that are more familiar with the broker
t;
> > Thanks for the KIP!
> >
> > On Wed, Feb 5, 2025 at 8:06 AM Sophie Blee-Goldman
> > wrote:
> >>
> >> +1 (binding)
> >>
> >> Thanks for the KIP! Neat and practical idea
> >>
> >> On Tue, Feb 4, 2025 at 10:52 AM A
ed and called out. Might be worth to add
> a short sentence/paragraph about it.
>
>
> Thoughts?
>
>
> -Matthias
>
> On 2/6/25 4:49 PM, Matthias J. Sax wrote:
> > Sounds good. Thanks for clarifying.
> >
> >
> > -Matthias
> >
> > On 2/6/25 3:1
t;
> -Matthias
>
>
> On 2/6/25 12:01 PM, Matthias J. Sax wrote:
> > BatchWindows works for me.
> >
> >
> > On 2/6/25 7:34 AM, Almog Gavra wrote:
> >> Happy to name it BatchWindows. Will give some people time to chime in
> and
> >> then change th
the established naming pattern and
> grammar used by other Windows classes: eg TimeWindows, SessionWindows,
> SlidingWindows
>
> Not a big deal though, won't redact my +1 on the voting thread if you
> prefer to keep it as BatchedWindows
>
> On Tue, Feb 4, 2025 at 10:51 A
Hello All,
I'd like to start a vote on KIP-1127. Please take a look at the KIP and
Discussion thread and let me know what you think.
Note that the discussion thread was incorrectly prefixed with KIP-1124
instead of KIP-1127 (oops!).
Link to KIP:
https://cwiki.apache.org/confluence/display/KAFKA/
orse" (for the lack
> of a better word...); I don't think its really worse, it just inherently
> non-deterministic, and that's fine.
>
> Guess we are overall on the same page and share common understanding of
> the trade-off. So I think we are good :)
>
>
&g
to full
> event-time based windows we support so far).
>
> So I don't think that is an actual difference between stream-time or
> count-based `BatchWindows` with regard to determinism.
>
>
>
> > which is important for EOS and even ALOS
>
> I don't see
iscussed.
> >>
> >> For `TimeWindow`, I am not sure as I said, and your interpretation as
> >> "just a container" might be fine, too. I agree to the problem, that if
> >> we add something new, we might just leak it again, and thus not gain
> >>
ndering if we should go with `extends` or add something
> new, which we hopefully can keep internal...)
>
> Not sure if we ever intent to make `StreamTimeWindows` overlapping?
> Given the described use-cases it seems very unlikely? So effectively
> it's a "tumbling window&
Hello!
I'd like to initiate a discussion thread on KIP-1127:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1127+Flexible+Windows+for+Late+Arriving+Data
This KIP aims to make it easier to specify windowing semantics that are
more tolerable to late arriving data, particularly with suppressi
Almog Gavra created KAFKA-18626:
---
Summary: Support for windows that are more flexible to late
arriving data
Key: KAFKA-18626
URL: https://issues.apache.org/jira/browse/KAFKA-18626
Project: Kafka
Almog Gavra created KAFKA-18326:
---
Summary: Cached stores may return deleted values
Key: KAFKA-18326
URL: https://issues.apache.org/jira/browse/KAFKA-18326
Project: Kafka
Issue Type: Bug
config map, and only
> one
> >>>> that takes in a StreamsConfig. Rather than adding a new constructor
> for
> >>>> TopologyConfig I think it makes sense to just add this new config to
> >> both
> >>>> StreamsConfig and TopologyConfig, as
Thanks Sophie! This KIP will certainly make it easier to implement any kind
of custom functionality across all processors in the DSL, I can imagine
quite a few use cases for this.
One suggestion, we should consider including a configure() method that
takes in Map configs, so that it can be configu
Hi Sebastien,
Thanks for the KIP! In general, I'm a fan of giving users the tools they
need to protect their organization so I'm supportive of this proposal. A
few nits and comments:
A1. [nit] consider 'topics.internal.require.explicit.naming' so that (a) we
can group anything else we introduce f
Hi Frederik - thanks for the KIP, this will be a fantastic and elegant
addition to Kafka Streams.
I have a higher level question about this, which is that the `poll`
interface returns multiple records and yet the DeserializationException
will be thrown if any record in the batch cannot be processe
+1 Not binding :)
On Mon, Mar 25, 2024 at 11:11 AM Walker Carlson
wrote:
> Hello everybody,
>
> I think we have had some pretty good discussion on this kip and it seems
> that we are close if not yet settled on the final version.
>
> So I would like to open up the voting for KIP-1024:
> https://
am in favor of deferring this to
> a separate KIP.
>
> Best,
> Bruno
>
> On 3/26/24 12:49 AM, Almog Gavra wrote:
> > Hello Folk!
> >
> > Glad to see improvements to the GlobalKTables in discussion! I think they
> > deserve more love :)
> >
> > Sco
Hello Folk!
Glad to see improvements to the GlobalKTables in discussion! I think they
deserve more love :)
Scope creep alert (which I'm generally against and certainly still support
this KIP without but I want to see if there's an elegant way to address
both problems). The KIP mentions that "Now
Hello Stan,
I wanted to give you a heads up that
https://github.com/apache/kafka/pull/15073 (
https://issues.apache.org/jira/browse/KAFKA-16046) was identified as a
blocker regression and should be merged to trunk by EOD.
Cheers,
Almog
On Tue, Jan 2, 2024 at 4:20 AM Stanislav Kozlovski
wrote:
[
https://issues.apache.org/jira/browse/KAFKA-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Almog Gavra reopened KAFKA-16046:
-
> Stream Stream Joins fail after restoration with deserialization excepti
[
https://issues.apache.org/jira/browse/KAFKA-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Almog Gavra resolved KAFKA-16046.
-
Resolution: Fixed
> Stream Stream Joins fail after restoration with deserialization excepti
40 PM Almog Gavra wrote:
> Sorry for the late response to the late reply, hah! I didn't give any
> thought about how we would want to integrate this custom store supplier
> with querying of the stores. My initial intuition suggests that we'd
> probably want a separate API
Almog Gavra created KAFKA-16046:
---
Summary: Stream Stream Joins fail after restoration with
deserialization exceptions
Key: KAFKA-16046
URL: https://issues.apache.org/jira/browse/KAFKA-16046
Project
Almog Gavra created KAFKA-16038:
---
Summary: Periodic Logging fo Current Assignment
Key: KAFKA-16038
URL: https://issues.apache.org/jira/browse/KAFKA-16038
Project: Kafka
Issue Type: Improvement
face from KS to expose the
> store instance directly.
>
>
> Guozhang
>
>
> On Sun, Nov 19, 2023 at 5:26 PM Almog Gavra wrote:
> >
> > Hello Guozhang,
> >
> > Thanks for the feedback! For 1 there are tests verifying this and I did
> so
> > manually as we
[
https://issues.apache.org/jira/browse/KAFKA-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Almog Gavra resolved KAFKA-15774.
-
Fix Version/s: 3.7.0
Resolution: Fixed
Note that we decided not to have
[
https://issues.apache.org/jira/browse/KAFKA-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Almog Gavra resolved KAFKA-15215.
-
Fix Version/s: 3.7.0
Resolution: Fixed
> The default.dsl.store config is not compati
custom logic, like an
> external controller / scheduler which is developed by a different
> group of people rather than the Streams app developers themselves,
> that can only turn on certain features after learning the actual store
> impl suppliers used?
>
> Guozhang
>
> On Sat
t; I don't have any objection to the goal; contrary! It's very annoying
> what we have right now, and if we can improve it, I am totally in favor
> of it.
>
>
> -Matthias
>
> On 11/3/23 8:47 AM, Almog Gavra wrote:
> > Good question :) I have a PR for it already h
Hello Rohan,
Thanks for the KIP! Thoughts below (seems I have similar comments to
Guozhang, but I had already written this before reading his reply haha!).
They're basically all minor suggestions for improvements, I wouldn't
consider any of them blocking.
1. For the API, thoughts on changing the
this?
>
> When we call `StreasmBuilder.build()` it will give us a already fully
> wired `Topology`, including all store suppliers needed. I don't see a
> clean way how we could change the store supplier after the fact?
>
>
> -Matthias
>
> On 11/2/23 5:11 PM, Almog Gavra
Almog Gavra wrote:
> OK! I think I got everything, but I'll give the KIP another read with
> fresh eyes later. Just a reminder that the voting is open, so go out and
> exercise your civic duty! ;)
>
> - Almog
>
> On Fri, Jul 28, 2023 at 10:38 AM Almog Gavra
> wr
Almog Gavra created KAFKA-15774:
---
Summary: Respect default.dsl.store Configuration Without Passing
it to StreamsBuilder
Key: KAFKA-15774
URL: https://issues.apache.org/jira/browse/KAFKA-15774
Project
+1 (non-binding) - great improvement, thanks Colt & Eduwer!
On Tue, Oct 17, 2023 at 11:25 AM Guozhang Wang
wrote:
> +1 from me.
>
> On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy
> wrote:
> >
> > Hi,
> >
> > thanks again for the KIP!
> >
> > +1 (binding)
> >
> > Cheers,
> > Lucas
> >
> >
> >
>
Thanks Almog! I made a pass over the updated wiki and have no more
> questions. +1
> >>
> >> Guozhang
> >>
> >> On Wed, Jul 26, 2023 at 8:14 AM Almog Gavra
> wrote:
> >>>
> >>> Hello Everyone,
> >>>
> >>>
OK! I think I got everything, but I'll give the KIP another read with fresh
eyes later. Just a reminder that the voting is open, so go out and exercise
your civic duty! ;)
- Almog
On Fri, Jul 28, 2023 at 10:38 AM Almog Gavra wrote:
> Thanks Guozhang & Sophie:
>
> A2. Will
> >
> > > > > A5. I also wasn't crazy about "Plugin" at first, but I will admit
> it's
> > > > > grown on me. I think it rubbed me the wrong
> > > > > way at first because it's just not part of the standard
> vocabulary in
>
Hello Everyone,
Opening the voting for KIP-954. The discussion is converging, but please
feel free to chime in on the last few conversation points if you aren't
happy with where it settled.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+cust
rename some occurrences in section
> "Materialized API" especially in the code section "Stores.java".
>
> A6: Actually I am not sure if I completely follow here. Is this about
> the static methods in class Stores? If yes, I agree with Almog to keep
> this o
open the voting tomorrow! Thanks again everyone for the discussion.
Cheers,
Almog
On Tue, Jul 25, 2023 at 9:20 AM Almog Gavra wrote:
> Glad you like my KIP-secretary skills ;)
>
> A2. I'm definitely happy to take your suggestion here and not do anything
> special w.r.t. Versione
pproach makes
> sense in the context of that feature's future plans.
>
> A3: sounds reasonable to me
>
> A5: Also sounds fine to me, though I'll let others chime in with/if they
> have an alternative suggestion/preference. I guess the other contender
> would be some
ess Stores class and seeing a
> >> timestamped
> >> > > version of the persistent but not in-memory stores. One could easily
> >> assume
> >> > > there was no timestamped option for IM stores and that this feature
> >> was
> >> > > inc
ion in Streams: where can we put guarantees of the
> >> public contract for RocksDB (or other store implementations) when all
> the
> >> RocksDB stuff is technically internal.
> >>
> >> Basically, I'm suggesting two things: first, call out in some way
> (perhaps
Hi All,
I would like to propose a KIP to expand support for default store types
(KIP-591) to encompass custom store implementations:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+custom+types
Looking forward to your feedback!
Cheers,
Almog
>
> Of course, for this KIP's scope itself, we can just do it for one specific
> rule of "source-changelog". I'm just wondering about the general framework
> for extending the current configuration here.
>
> Guozhang
>
>
> On Wed, Jan 13, 2
Hello,
I would like to start the discussion thread for KIP-705:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-705%3A+Selectively+Disable+Topology+Optimizations
The KIP is proposing an additional streams configuration that allows to
selectively disable optimizations under the topology.opti
Almog Gavra created KAFKA-12192:
---
Summary: Add Configuration to Selectively Disable Topology
Optimizations
Key: KAFKA-12192
URL: https://issues.apache.org/jira/browse/KAFKA-12192
Project: Kafka
Another thanks from me! +1 (non-binding)
On Tue, Jan 21, 2020 at 11:04 AM Randall Hauch wrote:
> Thanks again for the KIP and this improvement for Connect.
>
> +1 (binding)
>
> Randall
>
> On Tue, Jan 21, 2020 at 10:45 AM Tom Bentley wrote:
>
> > +1 (non-binding). Thanks for the KIP Konstantine
gt; >> > > > >
> > >> > > >
> > >> > > > I'll try to see how they both fit. Sure.
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
Hi Konstantine,
Thanks for the KIP! This is going to make automatic integration with
Connect much more powerful.
My thoughts are mostly around freshness of the data and being able to
expose that to users. Riffing on Randall's timestamp question - have we
considered adding some interval at which p
Thanks Cyrus! I think this change is a good step in hardening the API. I do
believe that APIs should be defined by functionality and not performance
characteristics, so I'd prefer using List<> over ArrayList<> (the
alternative you mention in rejected). That also gives us leeway in the
future to swa
; > >> +1 (binding)
> > > >>
> > > >> Randall
> > > >>
> > > >> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> > > >> konstant...@confluent.io> wrot
tine Karantasis <
> > >> konstant...@confluent.io> wrote:
> > >>
> > >>> Thanks Almog!
> > >>> Nicely designed and concise KIP.
> > >>>
> > >>> +1 non-binding
> > >>>
> > >>> Konstanti
; 4: "If desired, update any source connector configs that use the
> JsonConverter for key and/or value converters to use
> `decimal.format=NUMERIC`."
>
> Finally, should we add a short discussion in the Rejected Alternatives
> about the option of leaving JsonConverter
erialization.format=NUMERIC`via connector configurations or
> worker configs?
>
> Anyway, great job so far!
>
> Best regards,
>
> Randall
>
> On Thu, Aug 15, 2019 at 12:00 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > T
io is okay as the upgraded ~producer~ consumer will be able to read
> binary as today" (again according to my suggestion above, it could be as
> the upgraded source converter ...)
>
> - "consumers cannot consumer NUMERIC data. " -> "consumers c
Good catches! Fixed :)
On Thu, Aug 8, 2019 at 10:36 PM Arjun Satish wrote:
> Cool!
>
> Couple of nits:
>
> - In public interfaces, typo: *json.decimal.serialization.fromat*
> - In public interfaces, you use the term "HEX" instead of "BASE64".
>
>
&g
EDIT: everywhere I've been using "HEX" I meant to be using "BASE64". I will
update the KIP to reflect this.
On Wed, Aug 7, 2019 at 9:44 AM Almog Gavra wrote:
> Thanks for the feedback Arjun! I'm happy changing the default config to
> HEX instead of BINARY
config value “HEX” instead of “BINARY”?
>
> Should we call out the fact that JS systems might be susceptible to double
> precision round offs with the new numeric format? here are some people
> discussing a similar problem
> https://github.com/EventStore/EventStore/issues/1541
>
>
Hello Everyone,
After discussions on
https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
I've opened this KIP up for formal voting.
KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect
ersion of the KIP to a
vote.
Almog
On Mon, Jul 29, 2019 at 9:29 AM Almog Gavra wrote:
> I'm mostly happy with your current suggestion (two configs, one for
> serialization and one for deserialization) and your implementation
> suggestion. One thing to note:
>
> > We should
or backwards compatibility, but I agree that discussions / plans around
> switching the defaults should not block this KIP.
>
> Andy
>
>
> On Thu, 25 Jul 2019 at 18:26, Almog Gavra wrote:
>
> > Thanks for the replies Andy and Andrew (2x Andy?)!
> >
> > &g
IMHO, we should then plan to switch the default of decimal serialization
> to
> > numeric, and text serialization to base 10 in the next major release.
> > (With upgrade notes to match). Though I know this is more contentious, I
> > think it moves us forward in a much more st
Hi Everyone!
Kicking off discussion for a new KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
For those who are interested, I have a prototype implementation that helped
guide my design: https://github.com/agavra/kafka/pull/1
Almog Gavra created KAFKA-8595:
--
Summary: Support SerDe of Decimals in JSON that are not HEX encoded
Key: KAFKA-8595
URL: https://issues.apache.org/jira/browse/KAFKA-8595
Project: Kafka
Issue
Hello everyone - the PR is out and ready to review!
https://github.com/apache/kafka/pull/6728/
On Fri, May 10, 2019 at 11:57 AM Almog Gavra wrote:
> Thanks everyone for the comments and discussion! Closing the voting out
> for this KIP:
>
> * 4 Binding (Randall, Manikumar, Colin,
at 09:00, Ryanne Dolan wrote:
> > > > +1 (non-binding) for the core feature, but I could take or leave the
> > > > builder.
> > > >
> > > > Ryanne
> > > >
> > > > On Fri, May 10, 2019 at 10:43 AM Almog Gavra
> >
rate KIP?
>
> best,
> Colin
>
>
> On Fri, May 10, 2019, at 09:00, Ryanne Dolan wrote:
> > +1 (non-binding) for the core feature, but I could take or leave the
> > builder.
> >
> > Ryanne
> >
> > On Fri, May 10, 2019 at 10:43 AM Almog Gavr
prefix? It would be good to
> check existing builders in `clients` if any exist for what they're doing.
>
> If we didn't add a builder, another option would be a single new
> constructor:
>
> public NewTopic(String name, Optional numPartitions,
> Optional replicationFact
ntion time vs size?
>
> I am also wondering, what the cut-off line for important configs is,
> that get their own method, vs all other that are set via `config()`.
>
> It seems to be an "random" selection atm.
>
>
> -Matthias
>
> On 5/9/19 6:56 PM, Almog Ga
; imminent KIP deadline, I'd keep it simple and just have the constructor
> > > with just the name parameter.
> > >
> > > Ismael
> > >
> > > On Thu, May 2, 2019 at 1:58 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >
than relying upon the generic
> properties. But I'm fine if others think they should be removed. I'm also
> fine with not deprecating the Connect version of the builder.
>
> On Fri, May 3, 2019 at 11:27 AM Almog Gavra wrote:
>
> > Ack. KIP updated :) Perhaps instead
ubclass,
> should we ever want to do that (e.g, in Connect). If we make it protected
> or public, then subclassing is easier.
>
> On Thu, May 2, 2019 at 9:44 AM Almog Gavra wrote:
>
> > Thanks for the input Randall. I'm happy adding it natively to NewTopic
> > inst
NewTopic myTopic =
>
> NewTopic.build(name).compacted().partitions(1).replicationFactor(3).build();
>
> This is a bit shorter, a bit easier to read (no "new New..."), and more
> discoverable since anyone looking at the NewTopic source or JavaDoc will
> maybe n
> > else being set via builder methods.
> > >
> > > This would result in a better long term api as the number of options
> > > increases.
> > >
> > > Sent from my iPhone
> > >
> > > > On 30 Apr 2019, at 16:28, Almog Gavra wrote
nt from my iPhone
>
> > On 30 Apr 2019, at 16:28, Almog Gavra wrote:
> >
> > Hello Everyone,
> >
> > I'd like to start a discussion on KIP-464:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-464%3A+Default+Replication+Factor+for+AdminClient%2
Hello Everyone!
Kicking off the voting for
https://cwiki.apache.org/confluence/display/KAFKA/KIP-464%3A+Defaults+for+AdminClient%23createTopic
You can see discussion thread here (please respond with suggestions on that
thread):
https://lists.apache.org/thread.html/c0adfd2457e5984be7471fe6ade8a94
e to think - it gets
> them to pick a random number... (not sure there is any configuration that
> can get anyone to think, unfortunately).
>
> On Tue, Apr 30, 2019 at 10:22 AM Almog Gavra wrote:
>
> > I have a preference toward requiring specifying partitions per topic, but
> >
e partition count broker default to be used (the one used for auto
> topic creation).
>
> Ismael
>
> On Tue, Apr 30, 2019, 8:39 AM Almog Gavra wrote:
>
> > Hello Everyone,
> >
> > I'd like to start a discussion on KIP-464:
> >
> >
> h
Hello Everyone,
I'd like to start a discussion on KIP-464:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-464%3A+Default+Replication+Factor+for+AdminClient%23createTopic
It's about allowing users of the AdminClient to supply only a partition
count and to use the default replication factor
Hello Kafka Devs,
I'm following the instructions on
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals,
which first requires me to request edit access on the wiki.
My JIRA Id is: *agavra*
Cheers,
Almog
Almog Gavra created KAFKA-8305:
--
Summary: AdminClient should support creating topics with
`default.replication.factor`
Key: KAFKA-8305
URL: https://issues.apache.org/jira/browse/KAFKA-8305
Project
84 matches
Mail list logo
