Re: Flink+avro integration

2015-10-22 Thread Till Rohrmann
In the Java API, we only support the `max` operation for tuple types where you reference the fields via indices. Cheers, Till On Thu, Oct 22, 2015 at 4:04 PM, aawhitaker wrote: > Stephan Ewen wrote > > This is actually not a bug, or a POJO or Avro problem. It is simply a > > limitation in the f

Re: Flink+avro integration

2015-10-22 Thread aawhitaker
Stephan Ewen wrote > This is actually not a bug, or a POJO or Avro problem. It is simply a > limitation in the functionality, as the exception message says: > "Specifying > fields by name is only supported on Case Classes (for now)." > > Try this with a regular reduce function that selects the max

Re: Flink+avro integration

2015-10-21 Thread Stephan Ewen
This is actually not a bug, or a POJO or Avro problem. It is simply a limitation in the functionality, as the exception message says: "Specifying fields by name is only supported on Case Classes (for now)." Try this with a regular reduce function that selects the max and it should work fine... Gr

Re: Flink+avro integration

2015-10-21 Thread aawhitaker
Till Rohrmann wrote > What was your problem with using Java POJOs with the Scala API? Here's a quick example that demonstrates some of the problems I'm having. I used `max` in the example, but actually I get an exception for most of

Re: Flink+avro integration

2015-10-21 Thread Stephan Ewen
@Andrew Flink should work with Scala classes that follow the POJO style (public fields), so you should be able to use the Java Avro Library just like that. If that does not work in your case, please file a bug report! On Wed, Oct 21, 2015 at 9:41 AM, Till Rohrmann wrote: > What was your proble

Re: Flink+avro integration

2015-10-21 Thread Till Rohrmann
What was your problem with using Java POJOs with the Scala API? According to https://issues.apache.org/jira/browse/AVRO-1105, the progress on adding a Scala API to Avro is kind of stalling. Cheers, Till On Tue, Oct 20, 2015 at 9:06 PM, aawhitaker wrote: > One more follow up: > > There doesn't a

Re: Flink+avro integration

2015-10-20 Thread aawhitaker
One more follow up: There doesn't appear to be an official avro library for Scala. How would you recommend integrating Avro in a Scala project? The most straightforward option seems to be to just use the Java library, but Scala Flink operations don't work on Java POJOs, as far as I could tell. Doe

Re: Flink+avro integration

2015-10-20 Thread aawhitaker
Stephan, Thanks for the quick reply and the example. That does help! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-avro-integration-tp3162p3185.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabbl

Re: Flink+avro integration

2015-10-20 Thread Stephan Ewen
Hi Andrew! TL;DR There is no out of the box (de)serializer for Flink with Kafka, but it should be not very hard to add. Here is a gist that basically does it. Let me know if that works for you, I'll add it to the Flink source then: https://gist.github.com/StephanEwen/d515e10dd1c609f70bed Greeti

Re: Flink+avro integration

2015-10-19 Thread Márton Balassi
Hi Andrew, 1a, In general Flink can read and write Avro data through the AvroInputFormat and AvroOutputtFormat in both the batch and the streaming API. In general you can write the following: StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream dataSt