Re: Issue with BulkWriter

2019-10-22 Thread Ravi Bhushan Ratnakar
Hi, If possible, kindly share one output file to inspect, in the meanwhile you could also give a try with "org.apache.hadoop.io.compress.GzipCodec" Regards, Ravi On Tue, Oct 22, 2019 at 7:25 PM amran dean wrote: > > Hello, > These changes result in the following error: > $ lzop -d part-1-0 > l

Re: Issue with BulkWriter

2019-10-22 Thread amran dean
Hello, These changes result in the following error: $ lzop -d part-1-0 lzop: part-1-0: not a lzop file public class BulkRecordLZOSerializer implements BulkWriter { private final CompressionOutputStream compressedStream; public BulkRecordLZOSerializer(OutputStream stream) { Compr

Re: Issue with BulkWriter

2019-10-21 Thread Ravi Bhushan Ratnakar
Hi, Seems like that you want to use "com.hadoop.compression.lzo.LzoCodec" instead of "com.hadoop.compression.lzo.*LzopCodec*" in the below line. compressedStream = factory.getCodecByClassName("com.hadoop.compression.lzo.LzopCodec").createOutputStream(stream); Regarding "lzop: unexpected end of

Issue with BulkWriter

2019-10-21 Thread amran dean
Hello, I'm using BulkWriter to write newline-delimited, LZO-compressed files. The logic is very straightforward (See code below). I am experiencing an issue decompressing the created files created in this manner, consistently getting "lzop: unexpected end of file". Is this an issue with caller of