On Thu, 20 Oct 2011 20:46:22 +0200 Miko??aj Izdebski <[email protected]> wrote: > There is a simple way of solving this issue. Changes that need to be made: [...] > If anyone is interested I can prepare an appropriate patch. Just let me know.
Miko??aj, I am interested in a patch. I looked over lbzip2 code enough to realize I couldn't think of a simple patch for this issue. I was thinking it would be simple to limit the number of memory buffers between worker and muxer. Then looking at how blocks are split into input for the workers, started to worry that this would get into trouble with deadlocks on blocks which are split between two input blocks. So I filed a bug with no patch. ;) My guess of the easiest way to "fix" this is to change the behavior of the splitter. Instead of sending random blocks of input to the worker threads for them to split, the splitter actually reads and scans for the block markers at the same time and passes along single blocks into a queue of worker threads. Then a worker who is picking up a new block to decompress can check if he is "too far ahead" of the muxer and then block/wait for the muxer to catch up. I think this approach would mean that blocks always being decompression in correct order and it's very easy to control the size of worker to muxer queue. A downside is the splitter would be both I/O and CPU bound scanning for blocks ... which could be "fixed" by having two threads a "reader" and a different "splitter" (aka "scanner). On Thu, 20 Oct 2011 20:23:05 +0200 (CEST) "Ersek, Laszlo" <[email protected]> wrote: > I'll tell you this: you might want to check out pbzip2; I believe > pbzip2 should have a working memory throttle staring with v1.1.2. Laszlo, Yes, thanks for the suggestion. I do fallback on pbzip2 when lbzip2 fails. I used to use pbzip2, back then decompression required it to read the input file twice, first pass was single threaded and scanned for block markers, the second read was feed to worker threads to decompress... for large input streams (anything larger than fit in ram) that implementation sucked as it was input I/O bound for many minutes before generating any output. This double read also prevented it from decompressing pipes. At that time, I think due to the double I/O, lbzip2 was wall clock faster for any given input stream. I see now that the pbzip2 I/O layer has changed and it appears to support pipes and scans input only once. I should re-compare them. -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected]
