val hadoopConf = new Configuration(sc.hadoopConfiguration)
hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
"67108864")
sc.hadoopConfiguration(hadoopConf)
or
sc.hadoopConfiguration = hadoopConf
threw error.
On Sun, Jun 28, 2015 at 9:32 PM, Ted Yu <[email protected]> wrote:
> sequenceFile() calls hadoopFile() where:
> val confBroadcast = broadcast(new
> SerializableConfiguration(hadoopConfiguration))
>
> You can set the parameter in sc.hadoopConfiguration before calling
> sc.sequenceFile().
>
> Cheers
>
> On Sun, Jun 28, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]>
> wrote:
>
>> I can do this
>>
>> val hadoopConf = new Configuration(sc.hadoopConfiguration)
>>
>> * hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
>> "67108864")*
>>
>> sc.newAPIHadoopFile(
>>
>> path + "/*.avro",
>>
>> classOf[AvroKeyInputFormat[GenericRecord]],
>>
>> classOf[AvroKey[GenericRecord]],
>>
>> classOf[NullWritable],
>>
>> hadoopConf)
>>
>>
>> But i cant do the same with
>>
>> sc.sequenceFile("path", classOf[Text], classOf[Text])
>> How can i achieve the same with sequenceFile
>> --
>> Deepak
>>
>>
>
--
Deepak
