Raymond, If you mean custom components that you wrote for Flume-1.2.0, it will work without any issues in Flume-1.3.0 also. If there are some custom patches you applied, then you might need to rework the patches based on where the changes went into.
I believe this issue you are hitting should be reason enough (we improved FileChannel a *lot* from Flume-1.2.0 to Flume-1.3.0) to upgrade. Thanks, Hari -- Hari Shreedharan On Friday, October 19, 2012 at 8:15 AM, Brock Noland wrote: > The problem is that 1.2 writes first the marker for the event and then the > event. So it happens that sometimes the marker is written without the event. > The patch is small and in that jira Hari mentioned. > > On Fri, Oct 19, 2012 at 4:19 AM, Raymond Ng <[email protected] > (mailto:[email protected])> wrote: > > thanks for the replies > > > > I'm using Flume 1.2, and I'll look into getting 1.3 after assessing how > > much rework is needed to port the local customised changes from 1.2 to 1.3 > > > > also does the problem with the trailing 7F always happen when flume is > > killed ungracefully? > > > > Ray > > > > > > On Thu, Oct 18, 2012 at 5:54 PM, Brock Noland <[email protected] > > (mailto:[email protected])> wrote: > > > Hari is correct, but that won't fix that log file. To do that, you > > > need to truncate the log file. If you do a hexdump, there should be a > > > trailing 7F. That is a 7F with the rest of the file being 80. > > > > > > somedata....7F808080... > > > > > > That trailing 7F is the problem. Be sure to back up the log file > > > before truncating it. > > > > > > Brock > > > > > > On Thu, Oct 18, 2012 at 11:48 AM, Hari Shreedharan > > > <[email protected] (mailto:[email protected])> wrote: > > > > Raymond, > > > > > > > > This was an issue which we fixed (FLUME-1380) and will be part of > > > > Flume-1.3.0. If you need it immediately, you could clone > > > > trunk/flume-1.3.0 > > > > branch and build it. You should not see this error once you do that. Or > > > > you > > > > can wait for the next release - Flume-1.3.0, which should be released > > > > in the > > > > next few weeks or so. > > > > > > > > > > > > Thanks > > > > Hari > > > > > > > > -- > > > > Hari Shreedharan > > > > > > > > On Thursday, October 18, 2012 at 9:32 AM, Raymond Ng wrote: > > > > > > > > Hi > > > > > > > > I'm getting the following exception when restarting flume after it was > > > > killed manually, please advice > > > > > > > > > > > > 2012-10-18 17:04:33,889 INFO [conf-file-poller-0] > > > > DefaultLogicalNodeManager.java - Starting Channel fileChannel4 > > > > 2012-10-18 17:04:33,914 INFO [lifecycleSupervisor-1-4] > > > > DirectMemoryUtils.java - Unable to get maxDirectMemory from VM: > > > > NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) > > > > 2012-10-18 17:04:33,916 INFO [lifecycleSupervisor-1-4] > > > > DirectMemoryUtils.java - Direct Memory Allocation: Allocation = > > > > 1048576, > > > > Allocated = 0, MaxDirectMemorySize = 1908932608, Remaining = 1908932608 > > > > 2012-10-18 17:04:34,058 INFO [lifecycleSupervisor-1-0] > > > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel1 > > > > started > > > > 2012-10-18 17:04:34,058 INFO [lifecycleSupervisor-1-2] > > > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel2 > > > > started > > > > 2012-10-18 17:04:40,835 ERROR [lifecycleSupervisor-1-1] Log.java - > > > > Failed to > > > > initialize Log > > > > java.io.IOException: Header 80808080 not expected value: deadbeef > > > > at > > > > org.apache.flume.channel.file.TransactionEventRecord.fromDataInput(TransactionEventRecord.java:136) > > > > at > > > > org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:378) > > > > at > > > > org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:88) > > > > at org.apache.flume.channel.file.Log.replay(Log.java:251) > > > > at > > > > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:228) > > > > at > > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > > > > at > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > > at > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 2012-10-18 17:04:40,836 ERROR [lifecycleSupervisor-1-1] > > > > FileChannel.java - > > > > Failed to start the file channel > > > > java.io.IOException: Header 80808080 not expected value: deadbeef > > > > at > > > > org.apache.flume.channel.file.TransactionEventRecord.fromDataInput(TransactionEventRecord.java:136) > > > > at > > > > org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:378) > > > > at > > > > org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:88) > > > > at org.apache.flume.channel.file.Log.replay(Log.java:251) > > > > at > > > > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:228) > > > > at > > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > > > > at > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > > at > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > at > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 2012-10-18 17:04:53,740 INFO [lifecycleSupervisor-1-4] > > > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel4 > > > > started > > > > 2012-10-18 17:04:53,740 INFO [conf-file-poller-0] > > > > DefaultLogicalNodeManager.java - Starting Sink hdfsSink4 > > > > 2012-10-18 17:04:53,741 INFO [conf-file-poller-0] > > > > DefaultLogicalNodeManager.java - Starting Sink hdfsSink2 > > > > > > > > ..... > > > > ..... > > > > 2012-10-18 17:10:36,789 ERROR > > > > [SinkRunner-PollingRunner-DefaultSinkProcessor] SinkRunner.java - > > > > Unable to > > > > deliver event. Exception follows. > > > > java.lang.IllegalStateException: Channel closed [channel=fileChannel3] > > > > at > > > > com.google.common.base.Preconditions.checkState(Preconditions.java:145) > > > > at > > > > org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:266) > > > > at > > > > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) > > > > at com.test.CustomHDFSSink.process(CustomHDFSSink.java:428) > > > > at > > > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > > > at > > > > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 2012-10-18 17:10:41,790 ERROR > > > > [SinkRunner-PollingRunner-DefaultSinkProcessor] SinkRunner.java - > > > > Unable to > > > > deliver event. Exception follows. > > > > java.lang.IllegalStateException: Channel closed [channel=fileChannel3] > > > > at > > > > com.google.common.base.Preconditions.checkState(Preconditions.java:145) > > > > at > > > > org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:266) > > > > at > > > > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) > > > > at com.test.CustomHDFSSink.process(CustomHDFSSink.java:428) > > > > at > > > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > > > at > > > > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > > > > > > > > -- > > > > Rgds > > > > Ray > > > > > > > > > > > > > > > > > > > > -- > > > Apache MRUnit - Unit testing MapReduce - > > > http://incubator.apache.org/mrunit/ > > > > > > > > -- > > Rgds > > Ray > > > > -- > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
