Re: Reduce one event under multiple keys

2019-02-21 Thread Stephen Connolly
Thanks! On Mon, 18 Feb 2019 at 12:36, Fabian Hueske wrote: > Hi Stephen, > > Sorry for the late response. > If you don't need to match open and close events, your approach of using a > flatMap to fan-out for the hierarchical folder structure and a window > operator (or two for open and close) fo

Re: Reduce one event under multiple keys

2019-02-18 Thread Fabian Hueske
Hi Stephen, Sorry for the late response. If you don't need to match open and close events, your approach of using a flatMap to fan-out for the hierarchical folder structure and a window operator (or two for open and close) for counting and aggregating should be a good design. Best, Fabian Am Mo.

Re: Reduce one event under multiple keys

2019-02-11 Thread Stephen Connolly
On Mon, 11 Feb 2019 at 09:42, Fabian Hueske wrote: > Hi Stephen, > > A window is created with the first record that is assigned to it. > If the windows are based on time and a key, than no window will be created > (and not space be occupied) if there is not a first record for a key and > time int

Re: Reduce one event under multiple keys

2019-02-11 Thread Fabian Hueske
Hi Stephen, A window is created with the first record that is assigned to it. If the windows are based on time and a key, than no window will be created (and not space be occupied) if there is not a first record for a key and time interval. Anyway, if tracking the number of open files & average o

Re: Reduce one event under multiple keys

2019-02-10 Thread Stephen Connolly
On Sun, 10 Feb 2019 at 09:09, Chesnay Schepler wrote: > This sounds reasonable to me. > > I'm a bit confused by this question: "*Additionally, I am (naïevely) > hoping that if a window has no events for a particular key, the > memory/storage costs are zero for that key.*" > > Are you asking wheth

Re: Reduce one event under multiple keys

2019-02-10 Thread Chesnay Schepler
This sounds reasonable to me. I'm a bit confused by this question: "/Additionally, I am (naïevely) hoping that if a window has no events for a particular key, the memory/storage costs are zero for that key./" Are you asking whether a key that was received in window X (as part of an event) is