thanks for the replies I'm using Flume 1.2, and I'll look into getting 1.3 after assessing how much rework is needed to port the local customised changes from 1.2 to 1.3
also does the problem with the trailing 7F always happen when flume is killed ungracefully? Ray On Thu, Oct 18, 2012 at 5:54 PM, Brock Noland <[email protected]> wrote: > Hari is correct, but that won't fix that log file. To do that, you > need to truncate the log file. If you do a hexdump, there should be a > trailing 7F. That is a 7F with the rest of the file being 80. > > somedata....7F808080... > > That trailing 7F is the problem. Be sure to back up the log file > before truncating it. > > Brock > > On Thu, Oct 18, 2012 at 11:48 AM, Hari Shreedharan > <[email protected]> wrote: > > Raymond, > > > > This was an issue which we fixed (FLUME-1380) and will be part of > > Flume-1.3.0. If you need it immediately, you could clone > trunk/flume-1.3.0 > > branch and build it. You should not see this error once you do that. Or > you > > can wait for the next release - Flume-1.3.0, which should be released in > the > > next few weeks or so. > > > > > > Thanks > > Hari > > > > -- > > Hari Shreedharan > > > > On Thursday, October 18, 2012 at 9:32 AM, Raymond Ng wrote: > > > > Hi > > > > I'm getting the following exception when restarting flume after it was > > killed manually, please advice > > > > > > 2012-10-18 17:04:33,889 INFO [conf-file-poller-0] > > DefaultLogicalNodeManager.java - Starting Channel fileChannel4 > > 2012-10-18 17:04:33,914 INFO [lifecycleSupervisor-1-4] > > DirectMemoryUtils.java - Unable to get maxDirectMemory from VM: > > NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) > > 2012-10-18 17:04:33,916 INFO [lifecycleSupervisor-1-4] > > DirectMemoryUtils.java - Direct Memory Allocation: Allocation = 1048576, > > Allocated = 0, MaxDirectMemorySize = 1908932608, Remaining = 1908932608 > > 2012-10-18 17:04:34,058 INFO [lifecycleSupervisor-1-0] > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel1 > > started > > 2012-10-18 17:04:34,058 INFO [lifecycleSupervisor-1-2] > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel2 > > started > > 2012-10-18 17:04:40,835 ERROR [lifecycleSupervisor-1-1] Log.java - > Failed to > > initialize Log > > java.io.IOException: Header 80808080 not expected value: deadbeef > > at > > > org.apache.flume.channel.file.TransactionEventRecord.fromDataInput(TransactionEventRecord.java:136) > > at > > > org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:378) > > at > > > org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:88) > > at org.apache.flume.channel.file.Log.replay(Log.java:251) > > at > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:228) > > at > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > 2012-10-18 17:04:40,836 ERROR [lifecycleSupervisor-1-1] FileChannel.java > - > > Failed to start the file channel > > java.io.IOException: Header 80808080 not expected value: deadbeef > > at > > > org.apache.flume.channel.file.TransactionEventRecord.fromDataInput(TransactionEventRecord.java:136) > > at > > > org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:378) > > at > > > org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:88) > > at org.apache.flume.channel.file.Log.replay(Log.java:251) > > at > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:228) > > at > > > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > 2012-10-18 17:04:53,740 INFO [lifecycleSupervisor-1-4] > > MonitoredCounterGroup.java - Component type: CHANNEL, name: fileChannel4 > > started > > 2012-10-18 17:04:53,740 INFO [conf-file-poller-0] > > DefaultLogicalNodeManager.java - Starting Sink hdfsSink4 > > 2012-10-18 17:04:53,741 INFO [conf-file-poller-0] > > DefaultLogicalNodeManager.java - Starting Sink hdfsSink2 > > > > ..... > > ..... > > 2012-10-18 17:10:36,789 ERROR > > [SinkRunner-PollingRunner-DefaultSinkProcessor] SinkRunner.java - Unable > to > > deliver event. Exception follows. > > java.lang.IllegalStateException: Channel closed [channel=fileChannel3] > > at > > com.google.common.base.Preconditions.checkState(Preconditions.java:145) > > at > > > org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:266) > > at > > > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) > > at com.test.CustomHDFSSink.process(CustomHDFSSink.java:428) > > at > > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > 2012-10-18 17:10:41,790 ERROR > > [SinkRunner-PollingRunner-DefaultSinkProcessor] SinkRunner.java - Unable > to > > deliver event. Exception follows. > > java.lang.IllegalStateException: Channel closed [channel=fileChannel3] > > at > > com.google.common.base.Preconditions.checkState(Preconditions.java:145) > > at > > > org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:266) > > at > > > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118) > > at com.test.CustomHDFSSink.process(CustomHDFSSink.java:428) > > at > > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > -- > > Rgds > > Ray > > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ > -- Rgds Ray
