Hello- We are using Flume 1.4 with File Channel configured to use a very large capacity. We keep the checkpoint and backup checkpoint on separate disks.
Normally the file channel is mostly empty (<<1% of capacity). For the checkpoint the disk I/O seems to be very reasonable due to the usage of a MappedByteBuffer. On the other hand, the backup checkpoint seems to be written to disk in its entirety over and over again, resulting in very high disk utilization. I noticed that, because the checkpoint file is mostly empty, it is very compressible. I was able to GZIP our checkpoint from 381M to 386K. I was wondering if it would be possible to always compress the backup checkpoint before writing it to disk. I would be happy to work on a patch to implement this functionality if there is interest. Thanks in Advance, -- Abraham Fine | Software Engineer (516) 567-2535 BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com
