Thank you!
On Wednesday, 29 June 2016, Aljoscha Krettek wrote:
> Hi,
> the result of splitting by key is that processing can easily be
> distributed among the workers because the windows for individual keys can
> be processed independently. This should improve cluster utilization.
>
> Cheers,
>
Hi,
the result of splitting by key is that processing can easily be distributed
among the workers because the windows for individual keys can be processed
independently. This should improve cluster utilization.
Cheers,
Aljoscha
On Tue, 28 Jun 2016 at 17:26 Vishnu Viswanath
wrote:
> Hi,
>
> Than
Hi,
Thank you for the responses.
I am not sure if I will be able to use Fold/Reduce function, but I will
keep that in mind.
I have one more question, so what is the implication of having a key that
splits the data into window of very small size(=> large number of small
windows) ?
Thanks and Rega
Hi,
one thing to add: if you use a ReduceFunction or a FoldFunction for your
window the state will not grow with bigger window sizes or larger numbers
of elements because the result is eagerly computed. In that case, state
size is only dependent on the number of individual keys.
Cheers,
Aljoscha
Hi Vishnu,
RocksDB allows for storing the window contents on disk when the state of a
window becomes too big.
BUT when you have to trigger and apply the computation of your window function
on that big window,
then all of its state is loaded in memory.
So although during the window formation ph
Hi Kostas,
Thank you.
Yes 2) was exactly what I wanted to know.
- So if I am using RocksDB as state backend, does that mean that I don't
have to worry much about the memory available per node since RocksDB will
use RAM and Disk to store the window state?
Regards,
Vishnu
On Mon, Jun 27, 2016 at
Hi Vishnu,
I hope the following will help answer your question:
1) Elements are first split by key (apart from global windows) and then are put
into windows. In other words, windows are keyed.
2) A window belonging to a certain key is handled by a single node. In other
words, no matter how big
