+1 to generic interface for sorted state and Binary***State.
Very happy to be able to go one step further and thank you for your discussion.
Best,
Jingsong
On Fri, May 6, 2022 at 8:37 PM Nico Kruber wrote:
>
> While working a bit more on this, David A and I noticed a couple of things
> that wer
While working a bit more on this, David A and I noticed a couple of things
that were not matching each other in the proposed APIs:
a) the proposed BinarySortedMultiMapState class didn't actually have getters
that return multiple items per key, and
b) while having a single multi-map like implemen
Hi all,
Yun, David M, David A, and I had an offline discussion and talked through a
couple of details that emerged from the discussion here. We believe, we have
found a consensus on these points and would like to share our points for
further feedback:
Let me try to get through the points that w
l
> >> #tailMap
> >> > > like API. I just use NavigableMap's API to show examples.
> >> > >
> >> > > I think we can introduce new APIs like:
> >> > >
> >> > > SortedMapState extends State
> >> > >
record
>> the
>> > > > changes to any state, thus this should be included in the design
>> doc as
>> > >
>> > > we
>> > >
>> > > > need to introduce another kind of state, especially you might need
>> to
>> > > > consider how to store key bytes
7;s the same thing the
> > > Window
> > > API is already using - so there's nothing special here. The order in
> the
> > > log
> > > doesn't have to follow the natural order because this is only required
> > > inside
> > > the delegate
; > state. I
> >
> > > could give some examples of how to use sorted map state in min/max with
> > > retract functions.
> > >
> > > As you know, NavigableMap in java has several APIs like:
> > > Map.Entry firstEntry();
> >
zed byte arrays.
>
> Best
> Yun Tang
>
> --
> *From:* Nico Kruber
> *Sent:* Thursday, April 21, 2022 23:50
> *To:* dev
> *Cc:* David Morávek ; Yun Tang
> *Subject:* Re: [DISCUSS] FLIP-220: Temporal State
>
> Thanks Yun Tang for your
Hi David.
Thanks for the proposal of Temporal State. I'm working for the benchmark of
CEPOperator. I have question about how much performance improvement can
Temporal State bring to CEPOperator and whether there is any expected
performance?
Best,
Nicholas Jiang
On 2022/04/11 12:54:15 David An
Best
Yun Tang
From: Nico Kruber
Sent: Thursday, April 21, 2022 23:50
To: dev
Cc: David Morávek ; Yun Tang
Subject: Re: [DISCUSS] FLIP-220: Temporal State
Thanks Yun Tang for your clarifications.
Let me keep my original structure and reply in these points...
3. Should we generalis
state, especially you might need to
> consider how to store key bytes serialized by the new serializer (as we
> might not be able to write the length in the beginning of serialized bytes
> to make the order of bytes same as natural order).
> Best
> Yun Tang.
>
: Yun Tang ; David Morávek
Subject: Re: [DISCUSS] FLIP-220: Temporal State
Hi all,
I have read the discussion points from the last emails and would like to add
my two cents on what I believe are the remaining points to solve:
1. Do we need a TemporalValueState?
I guess, this all boils down to
> [1]
> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime
> /src/main/java/org/apache/flink/table/runtime/functions/aggregate/MinWithRet
> ractAggFunction.java
> Best,
> Yun Tang
>
> From: David Morávek
> Sent: Wednesday, April 13, 2022
13, 2022 19:50
To: dev
Subject: Re: [DISCUSS] FLIP-220: Temporal State
Here is a very naive implementation [1] from a prototype I did few months
back that uses list and insertion sort. Since the list is sorted we can use
binary search to create sub-list, that could leverage the same thing I'
;>
>> David
>>
>>
>> On Tue, Apr 12, 2022 at 9:21 AM Yun Tang wrote:
>>
>> > Hi David,
>> >
>> > Could you share some explanations why SortedMapState cannot work in
>> > details? I just cannot catch up what the statement below mean
was rejected as being overly difficult to implement in a way that
> > would cleanly leverage RocksDB’s iterators.
> >
> >
> > Best
> > Yun Tang
> > ____
> > From: Aitozi
> > Sent: Tuesday, April 12, 2022 15:00
> > To: dev@
icult to implement in a way that
> would cleanly leverage RocksDB’s iterators.
>
>
> Best
> Yun Tang
>
> From: Aitozi
> Sent: Tuesday, April 12, 2022 15:00
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] FLIP-220: Temporal Sta
Aitozi,
Our POC can be seen at [1].
My personal opinion is the namespace is an implementation detail that is
better not exposed directly. Flink state is already quite complex to
understand, and I fear that if we expose the namespaces the additional
flexibility this offers will be more confusing t
David, thanks for the feedback, much appreciated!
I'm hoping you can explain a bit more about how
Iterable> readRange(long minTimestamp, long
limitTimestamp);
would be used (and perhaps, implemented) in practice. I worry that this
might either prevent certain optimizations and/or be rather more
Hi David,
I really like the proposal. This has so much potential for various
optimizations, especially for temporal joins. My only concern is that the
interfaces seems unnecessarily complicated.
My feeling would be that we only need a single, simple interface that would
fit it all (the same way a
Hi David,
Thanks for driving.
I understand that state storage itself supports byte ordering, have we
considered exposing Binary**State? This way the upper layers can be
implemented on demand, Temporal is just one of them.
Best,
Jingsong
On Tue, Apr 12, 2022 at 3:01 PM Aitozi wrote:
>
> Hi Davi
From: Aitozi
Sent: Tuesday, April 12, 2022 15:00
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-220: Temporal State
Hi David
I have look through the doc, I think it will be a good improvement to
this pattern usage, I'm interested in it. Do you have some POC work to
Hi David
I have look through the doc, I think it will be a good improvement to
this pattern usage, I'm interested in it. Do you have some POC work to
share for a closer look.
Besides, I have one question that can we support expose the namespace in
the different state type not limited to `Tempo
Greetings, Flink developers.
I would like to open up a discussion of a proposal [1] to add a new kind of
state to Flink.
The goal here is to optimize a fairly common pattern, which is using
MapState>
to store lists of events associated with timestamps. This pattern is used
internally in quite a
24 matches
Mail list logo
