I tried but I'm getting the same error (task not serializable)
> On 25 בספט׳ 2015, at 20:10, Ted Yu <yuzhih...@gmail.com> wrote:
>
> Is the Schema.parse() call expensive ?
>
> Can you call it in the closure ?
>
>> On Fri, Sep 25, 2015 at 10:06 AM, Daniel Haviv
>> <daniel.ha...@veracity-group.com> wrote:
>> Hi,
>> I'm getting a NotSerializableException even though I'm creating all the my
>> objects from within the closure:
>> import org.apache.avro.generic.GenericDatumReader
>> import java.io.File
>> import org.apache.avro._
>>
>> val orig_schema = Schema.parse(new File("/home/wasabi/schema"))
>> val READER = new GenericDatumReader[GenericRecord](schema)
>>
>> val bd = sc.broadcast(orig_schema.toString)
>>
>>
>> val rdd=sc.binaryFiles("/daniel").map(zibi => {
>> val schema_obj = new Schema.Parser
>> val schema2 = schema_obj.parse(bd.value)
>> })
>>
>> I think that the problem is that Schema itself is an abstract class with
>> static methods (such as Parser).
>>
>> Am I correct?
>> How can I overcome it ?
>>
>> Thank you.
>> Daniel
>
