Re: [flink 1.8.1]window closed unexpectedly and data drop

2019-07-13 Thread Hequn Cheng
Hi Ever, The window only fires when the watermark passes the end of a window. > now the fourth data came with timestamp:03:17:55, at that time, a new window should be open, and the previous window should closed The previous window may not close if the watermark hasn't passed the end of the wind

Apache Flink - Side output time semantics for DataStream

2019-07-13 Thread M Singh
Hi: I wanted to find out what is the timestamp associated with the elements of a stream side output with different stream time characteristics. Thanks ManĀ 

How to create Row with RowTypeInfo

2019-07-13 Thread Soheil Pourbafrani
Hi Creating a new DataSet of type Row, how can I the RowTypeInfo of the row? For example when I create a new dataset like the following: Row row = Row.of(1, new Timestamp(1), new Date(1)); System.out.println(env.fromElements(row).getType()); it results in: Row(f0: Integer, f1: Timestamp, f2: Da

Re: Checkpoint failure

2019-07-13 Thread Navneeth Krishnan
Hi All, Any pointers on the below checkpoint failure scenario. Appreciate all the help. Thanks Thanks On Sun, Jul 7, 2019 at 9:23 PM Navneeth Krishnan wrote: > Hi All, > > Occasionally I run into failed checkpoints error where 2 or 3 consecutive > checkpoints fails after running for a minute a

Union of streams performance issue (10x)

2019-07-13 Thread Peter Zende
Hi all We have a pipeline (runs on YARN, Flink v1.7.1) which consumes a union of Kafka and HDFS sources. We remarked that the throughput is 10 times higher if only one of these sources is consumed. While trying to identify the problem I implemented a no-op source which was unioned with one of the

[flink 1.8.1]window closed unexpectedly and data drop

2019-07-13 Thread Ever
I have a streaming job based on Event time, which has a 60 seconds window and 10 seconds sliding window. Data will come in batches every 10 second. Here's the code. ``` val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) env