Re: Architecture question

2018-02-19 Thread Fabian Hueske
Hi, What you are looking for is a BucketingSink that works on event time (the timestamp is encoded in your data). AFAIK, Flink's BucketingSink has been designed to work in processing time, but you can implement a Bucketer that creates buckets based on a timestamp in the data. You might need to pla

Architecture question

2018-02-14 Thread robert
I need to grab avro data from a kafka topic and write to the local file system Inside the avro record there is a date time field. From that field I need to name the file accordingly. (20180103) as an example I was thinking of using flink to read, unpack this generic record then put to a sink tha