Javier
sorry to jumping in, but I think your case is very similar to what I am
trying to achieve in the thread just next to yours (called "Watermarks as
"process completion" flags". I also need to process a stream which is
produced for some time, but then take an action after certain event. Also
w
Hi,
Thanks for the example. We have done it with windows before and it works.
We are using state because the data comes with a gap of several days and we
can't handle a window size of several days. That's why we decided to use
the state.
On 27 November 2015 at 11:09, Aljoscha Krettek wrote:
> H
Hi,
I’ll try to go into a bit more detail about the windows here. What you can do
is this:
DataStream> input = … // fields are (id, sum,
count), where count is initialized to 1, similar to word count
DataStream> counts = input
.keyBy(0)
.timeWindow(Time.minutes(10))
.reduce(new MyCounting
Hi!
In streaming, there is no "end" of the stream when you would emit the final
sum. That's why there are windows.
If you do not want the partial sums, but only the final sum, you need to
define what window in which the sum is computed. At the end of that window,
that value is emitted. The window
Hi, thanks for the answer. It worked but not in the way we expected. We
expect to have only one sum per ID and we are getting all the consecutive
sums, for example:
We expect this: (11,6) but we get this (11,1) (11,3) (11,6) (the initial
values are ID -> 11, values -> 1,2,3). Here is the code we a
Hi Javier,
Thanks for your question. I've corrected the documentation (will be
online soon).
Cheers,
Max
On Wed, Nov 25, 2015 at 5:19 PM, Stephan Ewen wrote:
> Hi Javier!
>
> You can solve this both using windows, or using manual state.
>
> What is better depends a bit on when you want to have
Hi Javier!
You can solve this both using windows, or using manual state.
What is better depends a bit on when you want to have the result (the sum).
Do you want a result emitted after each update (or do some other operation
with that value) or do you want only the final sum after a certain time?
Hi,
We are trying to do a test using States but we have not been able to
achieve our desired result. Basically we have a data stream with data as
[{"id":"11","value":123}] and we want to calculate the sum of all values
grouping by ID. We were able to achieve this using windows but not with
states
