Hi Tarandeep, I think for that you would need to set a codec factory on the DataFileWriter. Sadly we don't expose that method to the user.
If you want, you can contribute this change to Flink. Otherwise, I can quickly fix it. Regards, Robert On Mon, Apr 18, 2016 at 2:36 PM, Ufuk Celebi <u...@apache.org> wrote: > Hey Tarandeep, > > regarding the network part: not possible at the moment. It's pretty > straight forward to add support for it, but no one ever got around to > actually implementing it. If you would like to contribute, I am happy > to give some hints about which parts of the system would need to be > modified. > > – Ufuk > > > On Mon, Apr 18, 2016 at 12:56 PM, Tarandeep Singh <tarand...@gmail.com> > wrote: > > Hi, > > > > How can I set compression for AvroOutputFormat when writing files on > HDFS? > > Also, can we set compression for intermediate data that is sent over > network > > (from map to reduce phase) ? > > > > Thanks, > > Tarandeep >
