Hi Sunitha, First and foremost, the DataSet API will be deprecated soon [1] so I would suggest trying to migrate to the DataStream API. When using the DataStream API it doesn't mean that you can not work with bounded inputs - you can. Flink SQL (Blink planner) is in fact using DataStream API to execute both streaming and batch queries. Maybe this path would be easier?
And about answering your question using the DataSet API - sorry, I don't know it :( I will try to ping someone who could help here. Piotrek [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 pon., 28 wrz 2020 o 15:14 s_penakalap...@yahoo.com <s_penakalap...@yahoo.com> napisał(a): > Hi All, > > Need your help in Flink Batch processing: scenario described below: > > we have multiple vehicles, we get data from each vehicle at a very high > speed, 1 record per minute. > thresholds can be set by the owner for each vehicle. > > Say: we have 3 vehicles, threshold is set for 2 vehicles. > Vehicle 1, threshold 20 hours, allowedPetrolConsumption=15 > vehicle 2, threshold 35 hours, allowedPetrolConsumption=28 > vehicle 3 no threshold set by owner. > > All the vehicle data is stored in HBase tables. We have a scheduled Batch > Job every day at 12 pm to check the status of vehicle movement and Petrol > consumption against threshold and raise an alert (vehicle1 did not move for > past 20 hours, vehicle 2 consumed more petrol. ) > > Since it is a Batch Job, I loaded all threshold data in one DataSet and > HBase Data in another Dataset using HbaseInputFormat. > > What I am failing to figure out is: > 1> vehicle 1 is having threshold of 20 hours where as vehicle 2 has > threshold of 35 hours, I need to fetch data from Hbase for different > scenario. Is there any better approach to get all data using one Hbase > connection. > 2> how to apply alert on Dataset. CEP pattern/ Match_recognize is allowed > only on DataStream. Please help me with a simple example. (alert can be > raised if count is zero or like petrol consumption is too high) > > > I could not get any example for Dataset on google where an alert is > raised. Kindly guide me if there is any better approach > > Regards, > Sunitha. >
