Re: KafkaProducer with generic (Avro) serialization schema

2018-05-02 Thread Wouter Zorgdrager
Hi, Thanks for the suggestions. Unfortunately I cannot make FromRecord/ForRecord/SchemaFor serializable, since those classes are out of my control. I use those from the avro4s library (https://github.com/sksamuel/avro4s). The problem here, especially with the deserializer is that I need to conver

Re: KafkaProducer with generic (Avro) serialization schema

2018-05-02 Thread Fabian Hueske
Hi Wouter, you can try to make the SerializationSchema serializable by overriding Java's serialization methods writeObject() and readObject() similar as Flink's AvroRowSerializationSchema [1] does. Best, Fabian [1] https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/jav

Re: KafkaProducer with generic (Avro) serialization schema

2018-05-02 Thread Aljoscha Krettek
Hi, Piotr is right, the SerializationSchema has to be serializable, which means that the implicit values passed on for SchemaFor[IN], FromRecord[IN], and ToRecord[IN] need to be serializable. Is there no way of making those serializable? As a workaround you could think about having a factory fo

Re: KafkaProducer with generic (Avro) serialization schema

2018-05-02 Thread Piotr Nowojski
Hi, My Scala knowledge is very limited (and my Scala's serialization knowledge is non existent), but one way or another you have to make your SerializationSchema serialisable. If indeed this is the problem, maybe a better place to ask this question is on Stack Overflow or some scala specific ma

Re: KafkaProducer with generic (Avro) serialization schema

2018-05-01 Thread Wouter Zorgdrager
So, I'm still struggling with this issue. I dived a bit more into the problem and I'm pretty sure that the problem is that I have to (implicitly) pass the SchemaFor and RecordTo classes to my serialization schema (otherwise I can't make it generic). However those class aren't serializable, but of c

Re: KafkaProducer with generic (Avro) serialization schema

2018-04-26 Thread Wouter Zorgdrager
Hi Bill, Thanks for your answer. However this proposal isn't going to solve my issue, since the problem here is that the context bounds I need to give in order to serialize it to Avro (SchemaFor, ToRecord and FromRecord) aren't serializable classes. This results in Flink not being able to serializ

RE: KafkaProducer with generic (Avro) serialization schema

2018-04-25 Thread Nortman, Bill
The things I would try would first in you are you class Person and Address have getters and setters and a no argument constructor. From: Wouter Zorgdrager [mailto:zorgdrag...@gmail.com] Sent: Wednesday, April 25, 2018 7:17 AM To: user@flink.apache.org Subject: KafkaProducer with generic (Avro) se