Hello everyone,
I'm trying to understand how I can use the Incremental Aggregation + Window Functions, as I've been unsuccessfully trying for a while now. The use-case is one where I have a stream of objects, I want to count the number of objects within a sliding window, and then within the window compare the count against a Redis threshold value, then emit a new type of object with the count, the window meta-data, and a boolean of whether the threshold was reached. The individual parts I understand, but it is getting the fold and window function correct that is giving me trouble. The stripped down code for example is here: https://gist.github.com/dbciar/36e4cf7df6eae2c214efb72d30385fd8 I think one issue is that what I want to do may not be possible, as when I look at the apply(accumulator, fold, window) definition here: https://github.com/apache/flink/blob/09e28109b3707f23230ea50526b7db8aa166df15/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala ..I think that the apply function needs to output a DataStream of the same type that the fold function uses as an aggregator, is this correct? At the moment I get the error that apply does not have a matching signature, even when I changed the window function to return an Int. I'm using version 1.1.0 artefacts from https://repository.apache.org/content/repositories/orgapacheflink-1098/. Any help or links would be great, I've tried searching the mailing list but I wasn't able to find anything that's close to this situation that I could use, Thanks David ________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ________________________________
