Performance Issue in the Flume HTTP Source to KafkaChannel

2017-12-08 Thread wenxing zheng
Dear all, We have deployed the flume 1.7.0 on our production box, and now we have at least 60,000 active connections to Flume HTTP Source running on 3 boxes. With so “high” QPS, the HTTP Source instances will report "java.io.IOException: Too many open files" error. Except to scale out the boxes,

Re: Performance Issue in the Flume HTTP Source to KafkaChannel

2017-12-08 Thread Matt Sicker
Can you change that setting in your kernel? Or is that not an option either? More connections means more files. On 8 December 2017 at 05:43, wenxing zheng wrote: > Dear all, > > We have deployed the flume 1.7.0 on our production box, and now we have at > least 60,000 active connections to Flume