to check whether eviction of (inactive/quiet) state from
>>>>> memory is something that I should consider implementing, or whether Flink
>>>>> already had some built-in way of doing it.
>>>>>
>>>>> Along the same lines, I am also wondering
whether Flink already has
>>>> means of compacting the state of a window by applying an aggregation
>>>> function to the elements so-far (eg. every time window is triggered)? For
>>>> example, if you are only executing a sum on the contents of the window, the
>>
ndow,
>>> it only needs to store the sum. Aggregations other than "sum" might have
>>> that characteristic too. I don't know if Flink is already that intelligent
>>> or whether I should figure out how to aggregate window contents myself when
>>> p
executing a sum on the contents of the window, the window
>>> state doesn't need to store all the individual items in the window, it only
>>> needs to store the sum. Aggregations other than "sum" might have that
>>> characteristic too. I don't know if Fli
ossible with something like a window fold? Another poster (Aljoscha) was
>> talking about adding incremental snapshots, but it sounds like that would
>> only improve the write throughput not the memory usage.
>>
>> Thanks again!
>> Shannon Carey
>>
>>
>
gregate window contents myself when
> possible with something like a window fold? Another poster (Aljoscha) was
> talking about adding incremental snapshots, but it sounds like that would
> only improve the write throughput not the memory usage.
>
> Thanks again!
> Shannon Carey
roughput not
the memory usage.
Thanks again!
Shannon Carey
From: Stephan Ewen mailto:se...@apache.org>>
Date: Wednesday, April 6, 2016 at 10:37 PM
To: mailto:user@flink.apache.org>>
Subject: Re: State in external db (dynamodb)
Hi Shannon!
Welcome to the Flink community!
You are rig
Hi Shannon!
Welcome to the Flink community!
You are right, sinks need in general to be idempotent if you want
"exactly-once" semantics, because there can be a replay of elements that
were already written.
However, what you describe later, overwriting of a key with a new value (or
the same value
FYI Cassandra has a TTL on data:
https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_expire_t.html
On Wed, Apr 6, 2016 at 7:55 AM, Shannon Carey wrote:
> Hi, new Flink user here!
>
> I found a discussion on user@flink.apache.org about using DynamoDB as a
> sink. However, as noted, sinks have
Hi, new Flink user here!
I found a discussion on user@flink.apache.org about using DynamoDB as a sink.
However, as noted, sinks have an at-least-once guarantee so your operations
must idempotent.
However, another way to go about this (and correct me if I'm wrong) is to write
the state to the e
10 matches
Mail list logo
