Re: designing a streaming task for count and event time difference

2020-01-05 Thread John Roesler
Hey Chris, Yeah, I think what you’re really looking for is data-driven windowing, which we haven’t implemented yet. In lieu of that, you’ll want to build on top of session windows. What you can do is define an aggregate object similar to what Sachin proposed. After the aggregation, you can ju

Re: designing a streaming task for count and event time difference

2020-01-05 Thread Chris Madge
This is great, thank you very much. I've read more into session windowing and suppression and they seem to fit my needs perfectly. I'm struggling to find a method of triggering the window to close early when I receive the end event. Maybe I could assign a monotonically increasing identifier ea

Re: designing a streaming task for count and event time difference

2020-01-04 Thread Sachin Mittal
Try something like this: stream .groupBy( (key, value) -> value.userId ) .aggregate( () -> new Session(), (aggKey, newValue, aggValue) -> { aggValue.userId = newValue.userId if (newValue.start) { aggValue.start = newValue.start aggValue.duration = 0

designing a streaming task for count and event time difference

2020-01-04 Thread Chris Madge
Hi there, It’s my first voyage into stream processing - I’ve tried a few things but I think I’m struggling to think in the streams way. I wondered if I could be cheeky and ask if someone could give me some clues as to the correct design for my first task to get me started? I have application e

designing a streaming task for count and event time difference

2020-01-04 Thread Chris Madge
Hi there, It’s my first voyage into stream processing - I’ve tried a few things but I think I’m struggling to think in the streams way. I wondered if I could be cheeky and ask if someone could give me some clues as to the correct design for my first task to get me started? I have application e