Re: Problem with SQL-API and nested objects in case class

2017-11-23 Thread Lothium
Hey Timo, when I try to access the array in the case class via the SQL syntax, I get back an error that the syntax is invalid. Here is an example of the case class structure: case class Envelope(name: String, entity: Product) case class Product(name: String, items: List[Item]) case class Item(attr

Re: Problem with SQL-API and nested objects in case class

2017-11-21 Thread Timo Walther
Actually, your use case should be doable with Flink's Table & SQL API with some additional UDFs. The API can handle JSON objects if they are valid composite types and you can access arrays as well. The splitting might be a bit tricky in SQL, you could model it simply as a where() clause or mayb

Re: Problem with SQL-API and nested objects in case class

2017-11-20 Thread Lothium
Hey Timo, thanks for your warm welcome and for creating a ticket to fix this! My scenario is the following: I receive different JSON entities from an AMQP queue. I have a source to collect the events, after that I parse them into the different internal case classes and split the stream via the spl

Re: Problem with SQL-API and nested objects in case class

2017-11-20 Thread Timo Walther
Hi, thanks for writing on the mailling list. I could reproduce your error and opened an issue for it (https://issues.apache.org/jira/browse/FLINK-8107). UNNEST currently only supports unnesting and joining an array of the same relation. However joining of two relations will be supported soon

Problem with SQL-API and nested objects in case class

2017-11-19 Thread Lothium
Hey, I need some help regarding the sql and table api. I'm using Apache Flink 1.3.2 with scala 2.11.11 (also tried 2.11.8) and I created a DataStream (based on a scala case class) and I registered this as a table. The case class includes some Lists, because the underlying JSON has some Arrays in t