Hi Roshan , Thank you for your reply. We just found more than two log rotate scripts running on the same spooling directory source, that may be causing an issue , Turned off other script , would test it . That should solve an issue, I guess.
Thanks, Snehal On 28 October 2013 13:34, Roshan Naik <[email protected]> wrote: > Snehal, > The Sppoling Dir source requires that the files in the directory it is > monitoring to consume are not be written/appended to. Basically once the > files are completed they need to be moved into the directory monitored by > SpoolingDir source. I suspect your setup is appending to the files that are > being consumed by this source. > -roshan > > > On Mon, Oct 28, 2013 at 12:05 PM, Snehal Nagmote <[email protected] > > wrote: > >> Hi All, >> >> I am using spooling directory source With Flume 1.4.0 (HDP) . Nginx is >> set up in a way that it rotates logs into spool directory every 15 minutes. >> >> I am seeing this exception quite often with flume agents >> >> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:173) >> - Uncaught exception in Runnable >> java.lang.IllegalStateException: Serializer has been closed >> at >> org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124) >> at >> org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88) >> at >> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221) >> at >> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> >> and it just get stuck and needs Flume restart . I found these two tickets >> describing same issue, >> >> https://issues.apache.org/jira/browse/FLUME-2160 >> >> https://issues.apache.org/jira/browse/FLUME-1887 >> >> Has anyone faced similar issues and is there any workaround to avoid this >> ? >> >> >> >> Thanks, >> Snehal >> >> >> >> >> >> >> >> >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
