Hi Fengyun,

Have you tried to use saveAsHadoopFile() (or
saveAsNewAPIHadoopFile())? You should be able to do something with
that API by using AvroKeyValueOutputFormat.

The API is defined here:
http://spark.apache.org/docs/1.0.0/api/scala/#org.apache.spark.rdd.PairRDDFunctions

Lots of RDD types include that functionality already.


On Wed, Jul 30, 2014 at 2:14 AM, Fengyun RAO <[email protected]> wrote:
> We used mapreduce for ETL and storing results in Avro files, which are
> loaded to hive/impala for query.
>
> Now we are trying to migrate to spark, but didn't find a way to write
> resulting RDD to Avro files.
>
> I wonder if there is a way to make it, or if not, why spark doesn't support
> Avro as well as mapreduce? Are there any plans?
>
> Or what's the recommended way to output spark results with schema? I don't
> think plain text is a good choice.



-- 
Marcelo

Reply via email to