Hi Jeff, As Stephan said the interpretation of the timestamps is up to the logic of your job. And as for the documentation, thanks for reporting this. We should update it.
Cheers, Kostas > On Dec 5, 2016, at 1:56 PM, Stephan Ewen <se...@apache.org> wrote: > > @Jeff - good point about the docs. > > I think Kostas is right though - the event timestamps are up to the user's > interpretation. > > The built-in window assigners interpret them as "Unix Epoch Millis", but you > can define your own window assigners that interpret the timestamps > differently. > The system interprets them as also as Unix Epoch Millis when mixing event > time and processing time (because processing time comes from > System.currentTimeMillis()) > > So, you can "re-interpret" them by using custom window assigners and not > using processing time. > If you want to use a processing time like component, I'd suggest to > incorporate that in your watermark generator. > > Stephan > > > > On Mon, Dec 5, 2016 at 1:05 PM, jeff jacobson <jeff.harold.jacob...@gmail.com > <mailto:jeff.harold.jacob...@gmail.com>> wrote: > Thanks Kostas. So if we're comfortable treating timestamps as longs (and > doing conversions to human readable time at our application level), we can > use CEP, ML lib etc. in addition to all basic Flink functions? That's great > news? > > To Matthias's point, why then does the following not read "Both timestamps > and watermarks are specified as longs"? Before I go headlong into developing > on Flink, I just want to be sure I'm covered here. Again, thanks. (The > Youtube videos from FlinkForward are also great, btw. Incredibly impressed > with Data Artisans.) > > > https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps > > <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps> > > "Both timestamps and watermarks are specified as milliseconds since the Java > epoch of 1970-01-01T00:00:00Z." > > > > On Mon, Dec 5, 2016 at 4:57 AM, Kostas Kloudas <k.klou...@data-artisans.com > <mailto:k.klou...@data-artisans.com>> wrote: > Hi Jeff, > > Actually in Flink timestamps are simple longs. > This means that you can assign anything you want as a timestamp, as long as > it fits in a long. > > Hope this helps and if not, we can discuss to see if we can find a solution > that > fits your needs together. > > Cheers, > Kostas > >> On Dec 4, 2016, at 11:39 PM, jeff jacobson <jeff.harold.jacob...@gmail.com >> <mailto:jeff.harold.jacob...@gmail.com>> wrote: >> >> Wow. Really? Is there a way to do micros? A hack? A Jira story? Most (all?) >> U.S. equity and European futures, options, and stock markets timestamp in >> microseconds. This makes Flink unusable for a massive industry vertical. To >> the extent lower-frequency time-series data is being used (e.g. end of data >> prices), stream processing is kind of overkill. Love everything I've read >> about Flink...there's got to be a way to make this work, no? >> >> On Sun, Dec 4, 2016 at 5:27 PM, Matthias J. Sax <mj...@apache.org >> <mailto:mj...@apache.org>> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA512 >> >> Oh. My bad... Did not read your question carefully enough. >> >> Than the answer is no, it does not support microseconds (only >> milliseconds). >> >> - -Matthias >> >> >> On 12/4/16 2:22 PM, jeff jacobson wrote: >> > Sorry if I'm missing something. That link mentions milliseconds, >> > no? My question is whether or not I can specify microseconds where >> > 1000microseconds = 1millisecond. Thanks! >> > >> > On Sun, Dec 4, 2016 at 5:05 PM, Matthias J. Sax <mj...@apache.org >> > <mailto:mj...@apache.org> >> > <mailto:mj...@apache.org <mailto:mj...@apache.org>>> wrote: >> > >> > Yes. It does. >> > >> > See: >> > https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/strea >> ming/event_timestamps_watermarks.html#assigning-timestamps >> <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps> >> > >> > >> <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/stream >> ing/event_timestamps_watermarks.html#assigning-timestamps >> <https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/event_timestamps_watermarks.html#assigning-timestamps>> >> > >> > "Both timestamps and watermarks are specified as millliseconds >> > since the Java epoch of 1970-01-01T00:00:00Z." >> > >> > >> > >> > -Matthias >> > >> > >> > On 12/04/2016 10:57 AM, jeff jacobson wrote: >> >> I've sourced stackoverflow, the docs, and the web but I can't >> >> figure out: does flink support microsecond timestamp resolution? >> >> Thanks! >> > >> > >> -----BEGIN PGP SIGNATURE----- >> Comment: GPGTools - https://gpgtools.org <https://gpgtools.org/> >> >> iQIYBAEBCgAGBQJYRJhUAAoJELz8Z8hxAGOiNKoP32ChGeNd7N8Zco2q6lsu+Hxd >> JZq62ey3wTrIUS+3oRlILwnu81cViQHtMMVBly3+YnqB85gNiaEUxEQTQCdKPl8G >> AqxoFIkMcrKGzwGXigKnCAoVIiyuPeNuhY1d1yv4rWrkt7qb0lCC02Xoq1C0hoS6 >> Stwk62GXmNRXPYpyjnSq/iAIMbjWaU+ZU0t4V3J8loroNuJ5QcUsJLfRXeo3/5ho >> f42L+IANyB5K7vnTxNZYyf5ShNVbTY9/iFaviluxrCNztqGTo7CxMpcyWyMS3wcF >> ycXcq/daB+guEJpW0sm4JtMPSsQ/kN99c/ig3t0HX1kDV7xrDDSF2qPvbYOWF38n >> omTr7RY3YRFi5LOKvBGa96Aw5UYjMddjcqozWId6xgdXfvz6RUeJCWa9RW8I6ptg >> 8TaJpM2WgDJMgMuzdl8dDv65l78DkLlNlNo53O66b/9Pt78P75KNjj8naD5kkj4C >> i9amwnUNNEnZucA2/1vhzr6cVSzrzBLL7juVj0VmABZo4itUZjjR0UkN7MB+ioWU >> trNhaXgE6EP/160n6D0/NUu02prm3jq8mK6gu9lZFWGbAeCUcch+CbvWSaiXAw3H >> BOieCsgZD1wfXQJ3wEmnqj/YP94uDlx1IjynskDevjk6OIyIysbBSIqgsUK6fvQ8 >> ztXO6ls7ARMOBmA= >> =/O+Q >> -----END PGP SIGNATURE----- >> > > >
