Snehal, The Sppoling Dir source requires that the files in the directory it is monitoring to consume are not be written/appended to. Basically once the files are completed they need to be moved into the directory monitored by SpoolingDir source. I suspect your setup is appending to the files that are being consumed by this source. -roshan
On Mon, Oct 28, 2013 at 12:05 PM, Snehal Nagmote <[email protected]>wrote: > Hi All, > > I am using spooling directory source With Flume 1.4.0 (HDP) . Nginx is set > up in a way that it rotates logs into spool directory every 15 minutes. > > I am seeing this exception quite often with flume agents > > (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:173) > - Uncaught exception in Runnable > java.lang.IllegalStateException: Serializer has been closed > at > org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124) > at > org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88) > at > org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221) > at > org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > and it just get stuck and needs Flume restart . I found these two tickets > describing same issue, > > https://issues.apache.org/jira/browse/FLUME-2160 > > https://issues.apache.org/jira/browse/FLUME-1887 > > Has anyone faced similar issues and is there any workaround to avoid this ? > > > > Thanks, > Snehal > > > > > > > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
