Thanks! Is this compressing everyline or in blocks? Is it possible to set it to compress per block?
On Sun, Apr 29, 2012 at 1:12 PM, Prashant Kommireddi <prash1...@gmail.com>wrote: > The ones you mentioned are for map output compression, not job output. > > On Apr 29, 2012, at 1:07 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > > > I tried these and didn't work with STORE? Is this different than the one > > you mentioned? > > > > SET mapred.compress.map.output true; > > > > SET mapred.output.compression org.apache.hadoop.io.compress.SnappyCodec; > > > > > > On Sun, Apr 29, 2012 at 11:57 AM, Prashant Kommireddi > > <prash1...@gmail.com>wrote: > > > >> Have you tried setting output compression to Snappy for Store? > >> > >> grunt> set output.compression.enabled true; > >> grunt> set output.compression.codec > >> org.apache.hadoop.io.compress.SnappyCodec; > >> > >> You should be able to read and write Snappy compressed files with > >> PigStorage which uses Hadoop TextInputFormat internally. > >> > >> Thanks, > >> Prashant > >> > >> > >> On Thu, Apr 26, 2012 at 12:40 PM, Mohit Anchlia <mohitanch...@gmail.com > >>> wrote: > >> > >>> I think I need to write both store and load functions. It appears that > >> only > >>> intermediate output that is stored on temp location can be compressed > >>> using: > >>> > >>> SET mapred.compress.map.output true; > >>> > >>> SET mapred.output.compression > org.apache.hadoop.io.compress.SnappyCodec; > >>> > >>> > >>> > >>> Any pointers as to how I can store and load using snappy would be > >> helpful. > >>> On Thu, Apr 26, 2012 at 12:32 PM, Mohit Anchlia < > mohitanch...@gmail.com > >>>> wrote: > >>> > >>>> I am able to write with Snappy compression. But I don't think pig > >>>> provides anything to read such records. Can someone suggest or point > me > >>> to > >>>> relevant code that might help me write LoadFunc for it? > >>> > >> >
