Thanks for clearing things up, Stephan and Kostas On Mon, Dec 5, 2016 at 8:08 AM, Kostas Kloudas <k.klou...@data-artisans.com> wrote:
> Hi Jeff, > > As Stephan said the interpretation of the timestamps is up to the logic of > your job. > And as for the documentation, thanks for reporting this. > We should update it. > > Cheers, > Kostas > > On Dec 5, 2016, at 1:56 PM, Stephan Ewen <se...@apache.org> wrote: > > @Jeff - good point about the docs. > > I think Kostas is right though - the event timestamps are up to the user's > interpretation. > > The built-in window assigners interpret them as "Unix Epoch Millis", but > you can define your own window assigners that interpret the timestamps > differently. > The system interprets them as also as Unix Epoch Millis when mixing event > time and processing time (because processing time comes from > System.currentTimeMillis()) > > So, you can "re-interpret" them by using custom window assigners and not > using processing time. > If you want to use a processing time like component, I'd suggest to > incorporate that in your watermark generator. > > Stephan > > > > On Mon, Dec 5, 2016 at 1:05 PM, jeff jacobson <jeff.harold.jacobson@gmail. > com> wrote: > >> Thanks Kostas. So if we're comfortable treating timestamps as longs (and >> doing conversions to human readable time at our application level), we can >> use CEP, ML lib etc. in addition to all basic Flink functions? That's great >> news? >> >> To Matthias's point, *why then does the following not read "**Both >> timestamps and watermarks are specified as longs"?* Before I go headlong >> into developing on Flink, I just want to be sure I'm covered here. Again, >> thanks. (The Youtube videos from FlinkForward are also great, btw. >> Incredibly impressed with Data Artisans.) >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.1/ >> apis/streaming/event_timestamps_watermarks.html#assigning-timestamps >> >> "Both timestamps and watermarks are specified as milliseconds since the Java >> epoch of 1970-01-01T00:00:00Z." >> >> >> >> On Mon, Dec 5, 2016 at 4:57 AM, Kostas Kloudas < >> k.klou...@data-artisans.com> wrote: >> >>> Hi Jeff, >>> >>> Actually in Flink timestamps are simple longs. >>> This means that you can assign anything you want as a timestamp, as long >>> as it fits in a long. >>> >>> Hope this helps and if not, we can discuss to see if we can find a >>> solution that >>> fits your needs together. >>> >>> Cheers, >>> Kostas >>> >>> On Dec 4, 2016, at 11:39 PM, jeff jacobson < >>> jeff.harold.jacob...@gmail.com> wrote: >>> >>> Wow. Really? Is there a way to do micros? A hack? A Jira story? Most >>> (all?) U.S. equity and European futures, options, and stock markets >>> timestamp in microseconds. This makes Flink unusable for a massive industry >>> vertical. To the extent lower-frequency time-series data is being used >>> (e.g. end of data prices), stream processing is kind of overkill. Love >>> everything I've read about Flink...there's got to be a way to make this >>> work, no? >>> >>> On Sun, Dec 4, 2016 at 5:27 PM, Matthias J. Sax <mj...@apache.org> >>> wrote: >>> >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA512 >>>> >>>> Oh. My bad... Did not read your question carefully enough. >>>> >>>> Than the answer is no, it does not support microseconds (only >>>> milliseconds). >>>> >>>> - -Matthias >>>> >>>> >>>> On 12/4/16 2:22 PM, jeff jacobson wrote: >>>> > Sorry if I'm missing something. That link mentions milliseconds, >>>> > no? My question is whether or not I can specify microseconds where >>>> > 1000microseconds = 1millisecond. Thanks! >>>> > >>>> > On Sun, Dec 4, 2016 at 5:05 PM, Matthias J. Sax <mj...@apache.org >>>> > <mailto:mj...@apache.org>> wrote: >>>> > >>>> > Yes. It does. >>>> > >>>> > See: >>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.1/ >>>> apis/strea >>>> ming/event_timestamps_watermarks.html#assigning-timestamps >>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps> >>>> > >>>> > >>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.1 >>>> /apis/stream >>>> ing/event_timestamps_watermarks.html#assigning-timestamps >>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps> >>>> > >>>> > >>>> > "Both timestamps and watermarks are specified as millliseconds >>>> > since the Java epoch of 1970-01-01T00:00:00Z." >>>> > >>>> > >>>> > >>>> > -Matthias >>>> > >>>> > >>>> > On 12/04/2016 10:57 AM, jeff jacobson wrote: >>>> >> I've sourced stackoverflow, the docs, and the web but I can't >>>> >> figure out: does flink support microsecond timestamp resolution? >>>> >> Thanks! >>>> > >>>> > >>>> -----BEGIN PGP SIGNATURE----- >>>> Comment: GPGTools - https://gpgtools.org >>>> >>>> iQIYBAEBCgAGBQJYRJhUAAoJELz8Z8hxAGOiNKoP32ChGeNd7N8Zco2q6lsu+Hxd >>>> JZq62ey3wTrIUS+3oRlILwnu81cViQHtMMVBly3+YnqB85gNiaEUxEQTQCdKPl8G >>>> AqxoFIkMcrKGzwGXigKnCAoVIiyuPeNuhY1d1yv4rWrkt7qb0lCC02Xoq1C0hoS6 >>>> Stwk62GXmNRXPYpyjnSq/iAIMbjWaU+ZU0t4V3J8loroNuJ5QcUsJLfRXeo3/5ho >>>> f42L+IANyB5K7vnTxNZYyf5ShNVbTY9/iFaviluxrCNztqGTo7CxMpcyWyMS3wcF >>>> ycXcq/daB+guEJpW0sm4JtMPSsQ/kN99c/ig3t0HX1kDV7xrDDSF2qPvbYOWF38n >>>> omTr7RY3YRFi5LOKvBGa96Aw5UYjMddjcqozWId6xgdXfvz6RUeJCWa9RW8I6ptg >>>> 8TaJpM2WgDJMgMuzdl8dDv65l78DkLlNlNo53O66b/9Pt78P75KNjj8naD5kkj4C >>>> i9amwnUNNEnZucA2/1vhzr6cVSzrzBLL7juVj0VmABZo4itUZjjR0UkN7MB+ioWU >>>> trNhaXgE6EP/160n6D0/NUu02prm3jq8mK6gu9lZFWGbAeCUcch+CbvWSaiXAw3H >>>> BOieCsgZD1wfXQJ3wEmnqj/YP94uDlx1IjynskDevjk6OIyIysbBSIqgsUK6fvQ8 >>>> ztXO6ls7ARMOBmA= >>>> =/O+Q >>>> -----END PGP SIGNATURE----- >>>> >>> >>> >>> >> > >
