Re: Calculate summary of bits in a sliding count window

2021-11-01 Thread Caizhi Weng
Hi! It seems that what you want is for *each* row compute the sum of the five previous rows (including the current row). This is the use case of over aggregation, not sliding window. I don't know if table API supports this, but you can see [1] for over aggregation in Flink SQL. But if you insist

Re: Calculate summary of bits in a sliding count window

2021-11-01 Thread Long Nguyễn
Hi. Thank you for the clarification. I updated my code as below and got the desired result. result = table.window(Slide.over( row_interval(WINDOW_SIZE)).every(row_interval(WINDOW_SLIDE)).on(col('proctime')).alias("w")) \ .group_by(col('w')) \ .select(call(read_raw_data, co

Re: Calculate summary of bits in a sliding count window

2021-11-01 Thread Caizhi Weng
Hi! You're not only grouping by the over window but also grouping by the value, thus only the records with the same value will be in the same group. I guess this is no intended. Long Nguyễn 于2021年11月2日周二 上午3:05写道: > I have set up a program that takes bits 0 and 1 from a Kafka topic and > then u

Calculate summary of bits in a sliding count window

2021-11-01 Thread Long Nguyễn
I have set up a program that takes bits 0 and 1 from a Kafka topic and then uses Flink to create a sliding count window of size 5. In that window, I'd like to output 1 if there are 3 or more of the bit 1, otherwise, output 0. Currently, I follow the way of calculating the sum of bits in the window.