Using Event Timestamp sink get's back with machine timezone

2020-08-11 Thread Faye Pressly
Hello, I am having an issue with Event time stamp and timezone with Flink 1.8 (1.8 because I need it to work on AWS Kinesis) I have a very simple pipeline that read events from a stream, transform to a Table does a small window (Tumblin 1 min) aggregation and groupby, transforms back to a stre

Proper way to do Integration Testing ?

2020-08-10 Thread Faye Pressly
Hello! I have built a Flink pipeline that involve 1) Reading events from a Kinesis Stream 2) Create two DataStream (using .filter) with events of type 'A' going in stream1 and event of type 'B' going in stream2 3) Transform stream1 into a Table and use Table API to do a simple window tumble a

Re: GroupBy with count on a joint table only let met write using toRetractStream

2020-08-07 Thread Faye Pressly
ertId, impVariationName, impMinute") .select("impAdvertId, impVariationName, clickAdvertId.count as clickCount, impMinute") .where("clickAdvertId != null"); ____ From: Faye Pressly Sent: Friday, August 7, 2020 9:28 PM To: user@flink.apache.o

GroupBy with count on a joint table only let met write using toRetractStream

2020-08-07 Thread Faye Pressly
Hello, I have a steam of events (coming from a Kinesis Stream) of this form: impressionId | advertid | variationName | eventType | eventTime The end goal is to output back on a Kinesis Stream the count of event of type 'impression' and the count of events of type 'click' however, I need to dro