We collect driving data from thousands of users, each vehicle is associated with a IMEI (unique code). The device installed in these vehicles emits GPS points in 5 second intervals. My requirement is to assemble all the GPS points that belong to a single trip and construct a Trip object, for a given IMEI.
I am using event time and Session windows to detect the end of a trip (10 minutes of non reception of GPS coordinates), and another 15 minutes of allowed lateness to wait for late events. The watermark then gets advanced to the latest received event time. Let's say this is for IMEI=100. Now, if I receive data for IMEI's 1 through 99 that have event time behind this watermark, all that data will be deemed late and won't be processed. In other words, if one vehicle's data advances in event time, then the data from all other vehicles will be considered late, because Watermarks are global. Given my problem, is there a way I can implement different watermarks for different streams? If not directly possible, is there some way I can simulate it to suit my application? Any help would be greatly appreciated! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
