Hi, I have been using Flink 1.1.1 with Kafka 0.9 to process real time streams. We have written our own Window Function and are processing data with Sliding Windows. We are using Event Time and use a custom watermark generator.
We select a particular window out of multiple sliding windows and process all items in the window in an iterative loop to increment counts of the items selected. After this we call the sink method to log the result in a database. This is working fine most of the times, i.e. it produces the expected result most of the times. However there are situations when certain windows are not processed (under same test conditions), this means the results are less than expected i.e. the counts are less than expected. Sometimes, items instead of certain windows not getting processed, certain items do not get processed in the window. This is unpredictable. I wish to know what could be the cause of this inconsistent behaviour. How do we resolve it. We have integrated 1.2 and Kafka 0.10 now, the problem persists. Please could you suggest about what the problem could be and how to resolve this. Many thanks. *Sujit Sakre* -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1908 264500 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NN. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
