Re: Using Kafka 0.10.x timestamps as a record value in Flink Streaming

2017-05-17 Thread Jia Teoh
a bit easier to use than implementing your own custom operator, > which is quite low-level. > > Cheers, > Gordon > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.2/dev/stream/process_function.html > > On 17 May 2017 at 1:45:38 PM, Jia Teoh (jiat...@gmai

Re: Kafka 0.10.x event time with windowing

2017-05-16 Thread Jia Teoh
> > Cheers, > Gordon > > > On 17 May 2017 at 1:34:31 PM, Jia Teoh (jiat...@gmail.com) wrote: > > Hi Gordon, > > Thanks for confirming my understanding that the extractor should not have > to be defined for 0.10. However, I'm still experiencing the case wher

Re: Using Kafka 0.10.x timestamps as a record value in Flink Streaming

2017-05-16 Thread Jia Teoh
gt; functions. > > At the operator level, you’ll have access to a `StreamRecord` in the > `processElement` function which wraps the record value (which you get when > implementing functions) as well as the internal timestamp that comes with > the record. > > Cheers, > Gor

Re: Kafka 0.10.x event time with windowing

2017-05-16 Thread Jia Teoh
directly to produce watermarks for event > time. > > One quick clarification: did you also check whether the timestamps that > come with the Kafka 0.10 records are sound and reasonable? > > Cheers, > Gordon > > > On 17 May 2017 at 4:45:19 AM, Jia Teoh (jiat...@gmail.com)

Kafka 0.10.x event time with windowing

2017-05-16 Thread Jia Teoh
iguration I am missing? I have verified that the Kafka messages have timestamps. The docs mention that there is no need for a timestamp extractor, but using one to explicitly assign the current time does result in windows being triggered. Thanks, Jia Teoh

Re: Using Kafka 0.10.x timestamps as a record value in Flink Streaming

2017-05-16 Thread Jia Teoh
those functions provide arguments that can be used to extract the embedded timestamp. Thanks, Jia Teoh On Fri, May 12, 2017 at 9:25 AM, Robert Metzger wrote: > Hi Jia, > > The Kafka 0.10 connector internally relies on the Kafka09 fetcher, but it > is extensible / pluggable so that als

Using Kafka 0.10.x timestamps as a record value in Flink Streaming

2017-05-11 Thread Jia Teoh
ector (1.2.1). Thanks, Jia Teoh