Hi Jungtaek, Flink 1.5.0 features a TableSource for Kafka and JSON [1], incl. timestamp & watemark generation [2]. It would be great if you could let us know, if that addresses your use case and if not what's missing or not working.
So far Table API / SQL does not have support for late-data side outputs. However, that's on the road map. The idea is to filter streams during ingestion for late events and passing them to a side output. Currently, operators just drop late events. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/sourceSinks.html#kafkajsontablesource [2] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/sourceSinks.html#configuring-a-rowtime-attribute 2018-07-04 10:39 GMT+02:00 Chesnay Schepler <ches...@apache.org>: > The watermark display in the UI is bugged in 1.5.0. > > It is fixed on master and the release-1.5 branch, and will be included in > 1.5.1 that is slated to be released next week. > > > On 04.07.2018 10:22, Jungtaek Lim wrote: > > Sorry I forgot to mention the version: Flink 1.5.0, and I ran the app in > IntelliJ, not tried from cluster. > > 2018년 7월 4일 (수) 오후 5:15, Jungtaek Lim <kabh...@gmail.com>님이 작성: > >> Hi Flink users, >> >> I'm new to Flink and trying to evaluate couple of streaming frameworks >> via implementing same apps. >> >> While implementing apps with both Table API and SQL, I found there's 'no >> watermark' presented in Flink UI, whereas I had been struggling to apply >> row time attribute. >> >> For example, below is one of TableSource implementation which wraps >> DataStream reading from Kafka. >> >> https://github.com/HeartSaVioR/iot-trucking-app- >> flink/blob/master/src/main/scala/net/heartsavior/flink/ >> datasource/TruckSpeedSource.scala >> >> (Actually I ended up implementing TableSource to address adding rowtime >> attribute as well as reading and parsing JSON. I'd be really happy if >> someone can guide a way to get rid of needed of custom implementation of >> TableSource.) >> >> and below is one of app I implemented: >> >> https://github.com/HeartSaVioR/iot-trucking-app- >> flink/blob/master/src/main/scala/net/heartsavior/flink/app/sql/ >> IotTruckingAppMovingAggregationsOnSpeedSql.scala >> >> Btw, I'm about to experiment side-output with late events, but is it >> possible to leverage side-output with Table API / SQL? Looks like >> DataStream exposes late events only when it's converted to >> AllWindowedStream. >> >> Thanks in advance! >> >> Best Regards, >> Jungtaek Lim (HeartSaVioR) >> > >
