Watermark on keyed stream

2018-10-10 Thread Nick Triller
Hi everyone, it seems Flink only supports global watermarks currently which is a problem for my use case. Many sensors send data which might be buffered for days in upstream systems before arriving at the Flink job. The job keys the stream by sensor. If other sensors send values in the meantime

Get stream of rejected data from Elasticsearch6 sink

2018-08-31 Thread Nick Triller
Hi all, is it possible to further process data that could not be persisted by the Elasticsearch6 sink without breaking checkpointing? As I understand, the onFailure callback can't be used to forward rejected data into a separate stream. I would like to extend the sink if this use case is not co