Recommended way to serialize Hadoop Writables' in Spark

Hi, Is there any recommended way of serializing Hadoop Writables' in Spark? Here is my problem. Question1: I have a pair RDD which is created by reading a SEQ[LongWritable, BytesWritable]: RDD[(LongWritable, BytesWritable)] I have these two settings set in spark conf. spark.serializer=org.apach

Recommended way to serialize Hadoop Writables' in Spark

Hi, Is there any recommended way of serializing Hadoop Writables' in Spark? Here is my problem. Question1: I have a pair RDD which is created by reading a SEQ[LongWritable, BytesWritable]: RDD[(LongWritable, BytesWritable)] I have these two settings set in spark conf. spark.serializer=org.apach

[Spark-SQL] - Creating Hive Metastore Parquet table from Avro schema

Hello Everyone, I have my parquet files stored on HDFS. I am trying to create a table in Hive Metastore from Spark SQL. I have an Avro schema file from which I generated the parquet files. I am doing the following to create the table. 1) Firstly create an Avro dummy table from the schema file