Re: Parallelizing DataStream operations on Array elements

2016-11-07 Thread Till Rohrmann
7; Arrays via > countWindowAll and using flatMap to emit 'row' Arrays. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Parallelizing- > DataStream-operations-on-Array-elements-tp9911p9917.html > Se

Re: Parallelizing DataStream operations on Array elements

2016-11-04 Thread danielsuo
I was able to resolve my issue by collecting all the 'column' Arrays via countWindowAll and using flatMap to emit 'row' Arrays. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Parallelizing-DataStream-operations-on-Array-

Re: Parallelizing DataStream operations on Array elements

2016-11-04 Thread danielsuo
th) Now I have a windowSize (representing time) by arrayLength (representing voxels) matrix. Flink lets me parallelize by time easily, but I'd like to parallelize by voxel. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Parallelizing-DataSt

Re: Parallelizing DataStream operations on Array elements

2016-11-04 Thread Till Rohrmann
Hi Daniel, I'm not sure whether I grasp the whole problem, but can't you split the vector up into the different rows, group by the row index and then apply some kind of continuous aggregation or window function? Maybe it helps if you can share some of your code with the community to discuss the i

Parallelizing DataStream operations on Array elements

2016-11-04 Thread Daniel Suo
Hello! I have a data source that emits Arrays that I collect into windows via countWindow. Rather than parallelize my subsequent operations by groups of these arrays, I'd like to parallelize my operations across the elements of the array (rows rather than columns, if you will) within each window.