Re: Storing large lists into state per key

Hi Jan, You could associate a key to each element of your Key's list (e.g., hashing the value), keep only the keys in heap (e.g., in a list) and the associated state key-value/s in an external store like RocksDB/Redis, but you will notice large overheads due to de/serializing - a huge penatly f

Re: KeyGroupRangeAssignment ?

rmann >> wrote: >>>> >>>>> Hi Ovidiu, >>>>> >>>>> at the moment it is not possible to plugin a user defined hash >>> function/key >>>>> group assignment function. If you like, then you can file a JIRA i

Re: KeyGroupRangeAssignment ?

Filled in https://issues.apache.org/jira/browse/FLINK-5873 <https://issues.apache.org/jira/browse/FLINK-5873> Best, Ovidiu > On 21 Feb 2017, at 12:00, Ovidiu-Cristian MARCU > wrote: > > Hi Till, > > I will look into filling a jira issue. > > Regarding the key

Re: KeyGroupRangeAssignment ?

the first case. What do the >> numbers mean then? >> >> Cheers, >> Till >> >> On Mon, Feb 20, 2017 at 3:45 PM, Ovidiu-Cristian MARCU < >> ovidiu-cristian.ma...@inria.fr> wrote: >> >>> Hi, >>> >>> Thank you for cla

Re: KeyGroupRangeAssignment ?

en? > > Cheers, > Till > > On Mon, Feb 20, 2017 at 3:45 PM, Ovidiu-Cristian MARCU < > ovidiu-cristian.ma...@inria.fr <mailto:ovidiu-cristian.ma...@inria.fr>> wrote: > >> Hi, >> >> Thank you for clarifications (I am working with KeyedStream so a

Re: KeyGroupRangeAssignment ?

ed to input.keyBy) could be easily tackled with some sort of local mapping but I am considering an use case with millions of keys. Best, Ovidiu > On 20 Feb 2017, at 15:45, Ovidiu-Cristian MARCU > wrote: > > Hi, > > Thank you for clarifications (I am working with

Re: KeyGroupRangeAssignment ?

)) % maxParallelism. > > Alternatively if you don’t need a keyed stream, you could try to use a custom > partitioner via DataStream.partitionCustom. > > Cheers, > Till > > > On Mon, Feb 20, 2017 at 11:46 AM, Ovidiu-Cristian MARCU > mailto:ovidiu-cristian.ma...@in

RE: [FLINK-3035] Redis as State Backend

using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU To: dev@flink.apache.org Sent

RE: TopSpeedWindowing - in error: Could not forward element to next operator

all elements and the window function always sees an empty iterable. I have a fix for this in a recent PR: https://github.com/apache/flink/pull/2656 Cheers, Aljoscha On Fri, 21 Oct 2016 at 16:57 Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.com> wrote: > Hi > > I have th

RE: [FLINK-3035] Redis as State Backend

only used as an out-of-core hash table. When checkpointing we write everything from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB instance from the data in HDFS. Cheers, Aljoscha On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.

Re: [FLINK-3035] Redis as State Backend

g we repopulate an empty local RocksDB > instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > ovidiu.cristian.ma...@huawei.com> wrote: > >> Hi >> >> I missed your reply, thank you for f

RE: TopSpeedWindowing - in error: Could not forward element to next operator

2016 at 6:38 PM, Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.com> wrote: > Could you check the following issue on master? > > When running this example org.apache.flink.streaming.examples.windowing. > TopSpeedWindowing > With default configuration I have no errors. &

RE: [FLINK-3035] Redis as State Backend

s pointed out on the issue. The second Idea is problematic because there is no easy way to read all state for a given Flink operator from a running Redis instance to store it in HDFS. That's what I was getting at in my comment. Cheers, Aljoscha On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Ma

TopSpeedWindowing - in error: Could not forward element to next operator

Could you check the following issue on master? When running this example org.apache.flink.streaming.examples.windowing. TopSpeedWindowing With default configuration I have no errors. When I change the state backend with RocksDB I receive this error: java.lang.RuntimeException: Could not forward

[FLINK-3035] Redis as State Backend

Hi Can you please expand the last comment: "I think, however, that for other reasons we will probably not be able to implement this well. The problem is that we have to somehow get at the state in redis for checkpointing. And if we use only one Redis instance for all states then this will be p

Re: Evaluating Apache Flink

Hi Kevin, I have orchestrated an evaluation of Spark and Flink for various batch and graph processing workloads (no streaming, no sql) (this work has been accepted as a paper at Cluster and I will publish soon a report, for more details please contact me directly). Both engines did well, stabl

Re: Side-effects of DataSet::count

Hi Stephan and all, Some reference to this may be https://issues.apache.org/jira/browse/FLINK-2250 ? I agree your priorities on streaming are very high, it will make a big +1 for the community to create a discussion/place for the design proposal