I expect Flink expert to answer your question. bq. I get a flush of the buffers atleast every few seconds
>From hbase point of view, during low traffic period, the above may result in many small hfiles, leading to more work for the compaction. FYI On Sat, Apr 29, 2017 at 7:32 AM, Niels Basjes <ni...@basjes.nl> wrote: > Hi, > > I have a sink that writes my records into HBase. > > The data stream is attached to measurements from an internal testing > instance of the website. > As a consequence there are periods of really high load (someone is doing a > load test) and really low load (only a hand full of people are testing > stuff). > > I read the records from Kafka and I want to write the records into HBase. > Because under high load it is more efficient to buffer the writes between > the client and the server and as indicated by HBase I use a BufferedMutator. > > This BufferedMutator works with a 'fixed size' buffer and under high load > setting it to a few MiB improves the performance writing to HBase greatly. > However under low load you have to wait until the buffer is full and that > can be a LONG time (hours) when the load is really low. > > I want to fire a periodic event into my sink to ensure I get a flush of > the buffers atleast every few seconds. > > Simply implement a standard Java TimerTask and fire that using a Timer? > Or is there a better way of doing that in Flink? > > > -- > Best regards / Met vriendelijke groeten, > > Niels Basjes >
