Re: ZStandard compression crashes

A similar bug was reported: HADOOP-17096 On Mon, May 11, 2020 at 3:48 PM Eric Yang wrote: > If I recall this problem correctly, the root cause is the default zstd > compression block size is 256kb, and Hadoop Zstd compression will attempt > to

Re: ZStandard compression crashes

If I recall this problem correctly, the root cause is the default zstd compression block size is 256kb, and Hadoop Zstd compression will attempt to use the OS platform default compression size, if it is available. The recommended output size is slightly bigger than input size to account for header

Re: ZStandard compression crashes

Thanks for the pointer, it does look similar. However we are roughly on the latest of branch-3.1 and this fix is in our branch. I'm pretty sure we have all the zstd fixes. I believe the libzstd version used is 1.4.4 but need to confirm. I suspected it's a library version issue because we've been u

Re: ZStandard compression crashes

Hi Wei Chiu, What is the Hadoop version being used? Give a check if HADOOP-15822 is there, it had something similar error. -Ayush > On 11-May-2020, at 10:11 PM, Wei-Chiu Chuang wrote: > > Hadoop devs, > > A colleague of mine recently hit a strange issue where zstd compression > codec crashes.

ZStandard compression crashes

Hadoop devs, A colleague of mine recently hit a strange issue where zstd compression codec crashes. Caused by: java.lang.InternalError: Error (generic) at org.apache.hadoop.io.compress.zstd.ZStandardCompressor.deflateBytesDirect(Native Method) at org.apache.hadoop.io.compress.zstd.ZStandardCompre