We're seeing problems when we try to live-reload our Flume agents rather than
restart them. They seem to maintain their incoming Syslog connections from the
clients, but they stop sending out data to ElasticSearch (and probably the HDFS
plugin as well). I see these errors during the reload, and I'm wondering if
they're related. The process to reproduce for us is to make any change to the
flume.conf file, and wait until Flume detects the file change. When this
happens, everything basically breaks.
08 May 2013 19:07:32,413 ERROR [lifecycleSupervisor-1-6]
(org.apache.flume.instrumentation.MonitoredCounterGroup.register:92) - Failed
to register monitored counter group for type: CHANNEL, name: fc1
javax.management.InstanceAlreadyExistsException:
org.apache.flume.channel:type=fc1
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
at
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:87)
at
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:67)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:323)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
08 May 2013 19:07:32,418 INFO [lifecycleSupervisor-1-8]
(org.apache.flume.sink.elasticsearch.ElasticSearchSink.start:319) -
ElasticSearch sink {} started
08 May 2013 19:07:32,418 ERROR [lifecycleSupervisor-1-8]
(org.apache.flume.instrumentation.MonitoredCounterGroup.register:92) - Failed
to register monitored counter group for type: SINK, name: elasticsearch
javax.management.InstanceAlreadyExistsException:
org.apache.flume.sink:type=elasticsearch
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
at
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:87)
at
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:67)
at
org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:320)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
08 May 2013 19:07:32,426 INFO [lifecycleSupervisor-1-8]
(org.apache.flume.instrumentation.MonitoredCounterGroup.start:73) - Component
type: SINK, name: elasticsearch started
