It does appear as though it's the "update" chunk that is zero-sized. Is there any way to know that before decompressing, and to skip the update higher up in the stack (perhaps in H5D__link_chunk_filtered_collective_io())?
On Thu, Nov 9, 2017 at 11:18 AM, Michael K. Edwards <m.k.edwa...@gmail.com> wrote: > Would it be better for that read-decompress-update-recompress-write > operation to skip zero-sized chunks? I imagine it's a bit tricky if > the lowest-indexed rank's contribution to the chunk is zero-sized; but > can that happen? Doesn't ownership move to the rank that has the > largest contribution to the chunk that's being written? > > On Thu, Nov 9, 2017 at 10:26 AM, Jordan Henderson > <jhender...@hdfgroup.org> wrote: >> Since Parallel Compression operates by applying the filter on a >> per-chunk-basis, this should be consistent with what you're seeing. However, >> zero-sized chunks is a case I had not actually considered yet, and I could >> reasonably see blosc failing due to a zero-sized allocation. >> >> >> Since reading in the parallel case with filters doesn't affect the metadata, >> the H5D__construct_filtered_io_info_list() function will simply cause each >> rank to construct a local list of all the chunks they have selected in the >> read operation, read their respective chunks into locally-allocated buffers, >> and decompress the data on a chunk-by-chunk basis, scattering it to the read >> buffer along the way. Writing works the same way in that each rank works on >> their own local list of chunks, with the exception that some of the chunks >> may get shifted around before the actual write operation of "pull data from >> the read buffer, decompress the chunk, update the chunk, re-compress the >> chunk and write it" happens. In general, it shouldn't cause an issue that >> you're reading the Dataset with a different number of MPI ranks than it was >> written with. _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
