Re: Sort streams in windows

like "sorted map state" will probably > be needed in order to efficiently query the state for minimal timestamp. > A less efficient implementation might work with ListState as well. > > Jan > > [1] > > https://docs.google.com/document/d/1ObLVUFsf1NcG8ZuIZE4aVy2RYKx2Ff

Sort streams in windows

Hi folks I want to sort stream based on event time field derived from events. To do this I can use one of the existing windows like TimeWindow to collect events in a window of a particular size, or SlidingWindow to run sort logic more often (and sort within slide). Ideally, I want to sort events a

Watermark alignment during unit tests

Hi devs During the work on https://issues.apache.org/jira/browse/FLINK-10050 I've found unstable behaviour of unit tests for unioned streams (which are used in CoGroupedStream/JoinedStream under the hood). Let's assume we have late elements in one of the stream. The thing is we have no guarantees

Kinesis connector shard discovery mechanism

Hi there Flink Kinesis consumer checks shards id for a particular pattern: "^shardId-\\d{12}" https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/model/StreamShardHandle.java#L132 While this inlines with

Allowed lateness for coGroup/join operators

Hi folks I'm surprised coGroup and join doesn't support allowedLateness feature (while underlying WindowedStream does) in the current release (1.5/1.6). I've opened a jira ticket with more details and ready to make appropriate changes in API: https://issues.apache.org/jira/browse/FLINK-10050 I'm