This is my write API. how do i integrate it here.
protected def writeOutputRecords(detailRecords:
RDD[(AvroKey[DetailOutputRecord], NullWritable)], outputDir: String) {
val writeJob = new Job()
val schema = SchemaUtil.outputSchema(_detail)
AvroJob.setOutputKeySchema(writeJob, schema)
val outputRecords = detailRecords.coalesce(100)
outputRecords.saveAsNewAPIHadoopFile(outputDir,
classOf[AvroKey[GenericRecord]],
classOf[org.apache.hadoop.io.NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
writeJob.getConfiguration)
}
On Wed, Jul 1, 2015 at 8:11 AM, Koert Kuipers <[email protected]> wrote:
> rdd.persist(StorageLevel.MEMORY_AND_DISK_SER)
>
> On Wed, Jul 1, 2015 at 11:01 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]>
> wrote:
>
>> How do i persist an RDD using StorageLevel.MEMORY_AND_DISK_SER ?
>>
>>
>> --
>> Deepak
>>
>>
>
--
Deepak
