Hi, I have a sink that writes my records into HBase.
The data stream is attached to measurements from an internal testing instance of the website. As a consequence there are periods of really high load (someone is doing a load test) and really low load (only a hand full of people are testing stuff). I read the records from Kafka and I want to write the records into HBase. Because under high load it is more efficient to buffer the writes between the client and the server and as indicated by HBase I use a BufferedMutator. This BufferedMutator works with a 'fixed size' buffer and under high load setting it to a few MiB improves the performance writing to HBase greatly. However under low load you have to wait until the buffer is full and that can be a LONG time (hours) when the load is really low. I want to fire a periodic event into my sink to ensure I get a flush of the buffers atleast every few seconds. Simply implement a standard Java TimerTask and fire that using a Timer? Or is there a better way of doing that in Flink? -- Best regards / Met vriendelijke groeten, Niels Basjes
