Re: when use broadcast variable and run on bigdata display this error please help

2015-08-29 Thread hagersaleh
where are any ways for use broadcast variable with bigdata -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/when-use-broadcast-variable-and-run-on-bigdata-display-this-error-please-help-tp2455p2566.html Sent from the Apache Flink User Mailing

Re: Kafka avro recordschema serializing / deserializing

2015-08-29 Thread Stephan Ewen
The functions/sources have an open() method that is exactly intended for this type of initialization (constructing the Avro Schema). You can try and subclass the kafka source and override the open() method to initialize the schema there. Make sure you call super.open(). Greetings, Stephan On S

Re: Kafka avro recordschema serializing / deserializing

2015-08-29 Thread Robert Metzger
Hi, yes, the Avro Schema is not serializable. Can you make the "_schema" field "transient" and then lazily initialize the field when serialize()/deserialize() is called? That way, you initialize the schema on the cluster, so there is no need to transfer it over the network. I think Flink's own s

Kafka avro recordschema serializing / deserializing

2015-08-29 Thread Ferenc Turi
Hi, I tried to read avro (RecordSchema) data from Kafka using the flink-kafka connector but I have problems: Exception says at program startup: Caused by: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java