Hello I've done my first steps with Flink and i am very impressed of its capabilities. Thank you for that :) I want to use it for a project we are currently working on. After reading some documentation i am not sure if it's the right tool for the job. We have an IoT application in which we are monitoring machines in production plants. The machines have sensors attached and they are sending their data to a broker ( Kafka, Azure Iot Hub ) currently on a per minute basis.
Following requirements must be fulfilled - Lateness We have to allow lateness for 7 days because machines can have down time due network issues, maintenance or something else. If thats the case buffering of data happens localy on the machine and once they are online again all data will be sent to the broker. This can result in some relly heavy burst. - Out of order Events come out of order due this lateness issues - Last write wins Machines are not stateful and can not guarantee exactly once sending of their data. It can happen that sometimes events are sent twice. In that case the last event wins and should override the previous one. Events are unique due a sensor_id and a timestamp - Computations per minute We can not wait until the windows ends and have to do computations on a per minute basis. For example aggregating data per sensor and writing it to a db My biggest concern in that case is the huge lateness. Keeping data for 7 days would result in 10080 data points for just one sensor! Multiplying that by 10.000 sensors would result in 100800000 datapoints which Flink would have to handle in its state. The number of sensors are constantly growing so will the number of data points So my questions are - Is Flink the right tool for the Job ? - Is that lateness an issue ? - How can i implement the Last write wins ? - How to tune flink to handle that growing load of sensors and data points ? - Hardware requirements, storage and memory size ? I don't want to maintain two code base for batch and streaming because the operations are all equal. The only difference is the time range! Thats the reason i wanted to do all this with Flink Streaming. Hope you can guide me in the right direction Thx
