Hi all,
I have a large production application that uses MPI + threads. The i/o pattern is either pure parallel-hdf5 or one file/rank (in a similar way to the VFD interface but done by hand). I make heavy use of compression filters. As my application is MPI + threads, only one thread per MPI rank actually does all the i/o work. In an uncompressed scenario, that is fine as I can then saturate the Infiniband cards with my 2 (or 4) MPI ranks per node. However, when compressing, this gets a lot slower. Only one thread does the compression and this now the clear bottleneck of the whole operation. It seems a bit silly as I have many idling threads which could all be drafted in for parallel compression. Would it hence be possible to see the use of `pigz` over plain `gzip` in future releases? Note that I'd like to avoid creating my own custom filter as I'd then have to distribute it collaborators using simulation results. Too high a barrier for access for many unfortunately. --- [Visit Topic](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656/1) or reply to this email to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://forum.hdfgroup.org/email/unsubscribe/77de6ecb3ced6322a56818c689e2b0c1477571543328f3f06ad4447dd88c8bce).
