Hi Timo, Thanks for the clarification. It's very helpful.
For the classpath, I suppose it can also support Python later if it's called in Python table api? Do we want to indicate if it's Java classpath or Python class? Or we support a list of classpath which can consist both Python, Java or other languages later? Thanks, Hao On Thu, Apr 24, 2025 at 5:34 AM Arvid Heise <ar...@apache.org> wrote: > Hi Timo, > > thank you very much for responding. I see that this is just the first step > to get consistency between SQL and Table API and more work is to come. > > I still think that there is some redundancy between STRUCT and ROW but tbh > I have more issues with ROW than with STRUCT. (What is even the meaning of > a nested ROW?) > > So +1 with your proposal and maybe we can deprecate ROW at some later point > in time. > > Best, > > Arvid > > On Thu, Apr 24, 2025 at 11:57 AM Timo Walther <twal...@apache.org> wrote: > > > Hi Arvid, Hi Hao, > > > > thanks for this valuable feedback. Let me clarify a few things before I > > go into the details. > > > > Just to avoid any confusion: the FLIP does not propose introducing the > > StructuredType. Structured types backed by classes already exist in > > Flink for years and are already supported in UDFs, Table.collect(), > > StreamTableEnvironment.toDataStream, and connectors. Structured types > > have been introduced for a better programmatic story in Table API. They > > avoid the need for manually defining the full schema at the edges. > > Manual schema work is annoying and with structured types it is possible > > to use classes whereever a type is expected. > > > > The goal of this FLIP only to bring Table API and SQL closer together. > > In general, this is only the first step of my larger vision of > > structured data handling. There are basically 3 kinds of structured > types: > > > > 1) a typed, fixed field struct like STRUCTURED<'Money', i INT, s STRING> > > 2) an untyped, fixed field struct like STRUCTURED<i INT, s STRING> > > (similar to Snowflake OBJECT(i INT, s STRING)) > > 3) an untyped struct for semi-structured data like STRUCTURED (similar > > to Snowflake OBJECT) > > > > RowType represents 2), StructuredType represents 1) and a future > > semi-structured type can represent 3) (but out of scope for this FLIP). > > > > If we don't support a typed struct, Money(i INT) and User(i INT) are not > > distinct in SQL. For table.collect() or eval(Row row) in UDFs, it would > > mean that those need the full schema declaration in order to map to a > > target type. Structured types avoid all of that and make Table API very > > powerful. > > > > Usually both the UDF and the collect()/toDataStream() are defined in the > > same Table API program. Thus, the class is usually present in the same > > classpath and this becomes less of an issue in production. Casting > > structured types to ROW is also supported. > > > > The implementation effort of this FLIP is very low. It's mostly intended > > to fill missing gaps, no major overhaul of the type system. Also to > > avoid any backwards compatibility issues. > > > > Let me know what you think. > > > > Cheers, > > Timo > > > > On 23.04.25 21:27, Hao Li wrote: > > > I think Arvid has a good point. Why not define Object type without > class > > > and when you get it in table api, try to cast it to some class? I found > > > > > > https://docs.oracle.com/javase/1.5.0/docs/guide/jdbc/getstart/mapping.html > > . > > > Under `JAVA_OBJECT` type section. They have: > > > > > > ``` > > > > > > ResultSet rs = stmt.executeQuery("SELECT ENGINEERS FROM PERSONNEL"); > > > while (rs.next()) { > > > Engineer eng = (Engineer)rs.getObject("ENGINEERS"); > > > System.out.println(eng.lastName + ", " + eng.firstName); > > > } > > > > > > ``` > > > > > > For us, how about add `getFieldAs(int post, Class class)` method in Row > > > type? Your example: > > > > > > ``` > > > > > > TableEnvironment env = ... > > > > > > Table t = env.sqlQuery("SELECT OBJECT_OF('com.example.User', 'name', > > 'Bob', > > > 'age', 42)"); > > > > > > // Tries to resolve `com.example.User` in the classpath, if not present > > > returns `Row` > > > t.execute().collect(); > > > ``` > > > > > > Will be > > > ``` > > > TableEnvironment env = ... > > > > > > Table t = env.sqlQuery("SELECT OBJECT_OF('name', 'Bob', 'age', 42)"); > > > > > > // Tries to resolve `com.example.User` in the classpath, if not present > > > returns `Row` > > > For (Row row : t.execute().collect()) { > > > User user = row.getFieldAs(0, User.class); > > > } > > > ``` > > > > > > For Arvid's question: "However, at that point, why do we actually need > > > anything beyond ROW?" > > > > > > Maybe the difference is Row type shouldn't support to be casted as user > > > defined class but `StructuredType` can be. > > > > > > Thanks, > > > Hao > > > > > > On Wed, Apr 23, 2025 at 2:04 AM Arvid Heise > <ahe...@confluent.io.invalid > > > > > > wrote: > > > > > >> Hi Timo, > > >> > > >> thanks for addressing my points. I'm not set on using STRUCT et al. > but > > >> wanted to point out the alternatives. > > >> > > >> Regarding the attached class name, I have similar confusion to Hao. I > > >> wonder if Structures types shouldn't be anonymous by default in the > > sense > > >> that initially we don't attach a class name to it. As you pointed out, > > it > > >> has no real semantics in SQL and we can't validate it. > > >> Another thing to consider is that if one user creates a table through > > some > > >> means and another user wants to consume it, the second user may not > have > > >> access to the class as is. But the user could easily create a > compatible > > >> class on its own. > > >> > > >> Consequently, I'm thinking about getting rid of the type at all. Only > on > > >> the edges, we can use conversion to the user types when users actually > > >> access the ROW: > > >> * Any table API access that wants to collect results (in your last > > example > > >> what is t.execute().collect(); returning? How does that work in the > > >> multi-user setup sketched above? Wouldn't it be easier that the > consumer > > >> explicitly gives us the POJO type that it expects?) > > >> * Any DataStream conversion > > >> * Any UDF > > >> > > >> However, at that point, why do we actually need anything beyond ROW? > > >> > > >> Best, > > >> > > >> Arvid > > >> > > >> On Wed, Apr 23, 2025 at 8:52 AM Timo Walther <twal...@apache.org> > > wrote: > > >> > > >>> Hi Hao, > > >>> > > >>> 1. Can `StructuredType` be nested? > > >>> > > >>> Yes this is supported. > > >>> > > >>> 2. What's the main reason the class won't be enforced in SQL? > > >>> > > >>> SQL should not care about classes. Within the SQL ecosystem, the SQL > > >>> engine controls the data serialization and protocols. The SQL engine > > >>> will not load the class. Classes are a concept of a JVM or Python API > > >>> endpoint. This also the reason why a SQL ARRAY<BIGINT> can be > > >>> represented as List<Long>, long[], Long[]. The latter are only > concepts > > >>> in the target programming language and might look different in > Python. > > >>> > > >>> Regard, > > >>> Timo > > >>> > > >>> > > >>> On 22.04.25 23:54, Hao Li wrote: > > >>>> Hi Timo, > > >>>> > > >>>> Thanks for the FLIP. +1 with a few questions: > > >>>> > > >>>> 1. Can `StructuredType` be nested? e.g. > > `STRUCTURED<'com.example.User', > > >>>> name STRING, age INT NOT NULL, address > > >> STRUCTURED<'com.example.address', > > >>>> street STRING, zip STRING>>` > > >>>> > > >>>> 2. What's the main reason the class won't be enforced in SQL? Since > > >>> tables > > >>>> created in SQL can also be used in Table API, will it come as a > > >> surprise > > >>> if > > >>>> it's working in SQL and then failing in Table API? What if > > >>>> `com.example.User` was not validated in SQL when creating table, > then > > >> the > > >>>> class was created for something else with different fields and then > in > > >>>> Table api, it's not compatible. > > >>>> > > >>>> Hao > > >>>> > > >>>> On Tue, Apr 22, 2025 at 9:39 AM Timo Walther <twal...@apache.org> > > >> wrote: > > >>>> > > >>>>> Hi Arvid, Hi Sergey, > > >>>>> > > >>>>> thanks for your feedback. I updated the FLIP accordingly but let me > > >>>>> answer your questions > > >>>>> here as well: > > >>>>> > > >>>>> > Are we going to enforce that the name is a valid class name? > > What > > >> is > > >>>>> > happening if it's not a correct name? > > >>>>> > What are the implications of using a class that is not in the > > >>>>> > classpath in Table API? It looks to me that the name is > > >>> metadata-only > > >>>>> > until we try to access the objects directly in > Table/DataStream > > >> API. > > >>>>> > > >>>>> Names are not enforced or validated. They are pure metadata as > > >> mentioned > > >>>>> in Section 2.1. We fallback to Row as the conversion class if the > > name > > >>>>> cannot be resolved in the current classpath. So when staying in the > > >> SQL > > >>>>> ecosystem (i.e. not switching to Table API, DataStream API, or > UDFs), > > >>>>> the class must not be present. > > >>>>> > > >>>>> > Should Expressions.objectOf(String, Object... kv); also have > an > > >>>>> > overload where you can put in the StructuredType in case where > > >>>>> > the class is not in the CP? > > >>>>> > > >>>>> That makes a lot of sense. I added a DataTypes.STRUCTURED(String, > > >>>>> Field...) method and a Expressions.objectOf(String, Object...). > > >>>>> > > >>>>> > What is the expected outcome of supplying fewer keys than > > defined > > >>>>> > in the structured type? Are we going to make use of > nullability > > >>> here? > > >>>>> > If so, *_INSERT and *_REMOVE may have some use. > > >>>>> > > >>>>> Currently, we go with the most conservative approach, which means > > that > > >>>>> all keys need to be present. Maybe we can reserve this feature to > > >> future > > >>>>> work and make the logic more lenient. > > >>>>> > > >>>>> > Talking about nullability: Is there some option to make the > > >> declared > > >>>>> > fields NOT NULL? If so, could you amend one example to show > > that? > > >>>>> > (Grammar? implies that it's not possible) > > >>>>> > > >>>>> NOT NULL is supported similar to ROW<i INT NOT NULL>. I adjusted > one > > >> of > > >>>>> the examples. > > >>>>> > > >>>>> > One bigger concern is around the naming. For me, OBJECT is > used > > >> for > > >>>>> > semi-structured types that are open. Your FLIP implies a > closed > > >>> design > > >>>>> > and that you want to add an open OBJECT later. I asked ChatGPT > > >> about > > >>>>> > other DB implementations and it seems like STRUCT is used more > > >> often > > >>>>> > (see below). So, I'd propose to call it STRUCT<...>, > STRUCT_OF, > > > > > >>>>> > structOf, UPDATE_STRUCT, and updateStruct respectively. > > >>>>> > > >>>>> Naming is hard. I was also torn between STRUCT, STRUCTURED, or > > OBJECT. > > >>>>> In Flink, the ROW type is rather our STRUCT type, because it works > > >> fully > > >>>>> position based. Structured types might be name-based in the future > > for > > >>>>> better schema evolution, so they rather model an OBJECT type. This > > was > > >>>>> my reason for choosing OBJECT_OF (typed to class name and fixed > > >> fields) > > >>>>> vs. OBJECT (semi-structured without fixed fields). Snowflake also > > uses > > >>>>> OBJECT(i INT) (for structured types) and OBJECT (for semi > structured > > >>>>> types). > > >>>>> > > >>>>> Also, both structured and semi-structured types can then share > > >> functions > > >>>>> such as UPDATE_OBJECT(). > > >>>>> > > >>>>> What do others think? > > >>>>> > > >>>>> Thanks, > > >>>>> Timo > > >>>>> > > >>>>> On 22.04.25 12:08, Sergey Nuyanzin wrote: > > >>>>>> Thanks for driving this Timo > > >>>>>> > > >>>>>> The FLIP seems reasonable to me > > >>>>>> > > >>>>>> I have one minor question/clarification > > >>>>>> do I understand it correct that after this FLIP we can execute of > > >>>>>> `typeof` against result of `OBJECT_OF` > > >>>>>> for instance > > >>>>>> SELECT typeof(OBJECT_OF( > > >>>>>> 'com.example.User', > > >>>>>> 'name', 'Bob', > > >>>>>> 'age', 42 > > >>>>>> )); > > >>>>>> > > >>>>>> should return `STRUCTURED<'com.example.User', name STRING, age > INT>` > > >>>>>> ? > > >>>>>> > > >>>>>> On Tue, Apr 22, 2025 at 10:57 AM Timo Walther <twal...@apache.org > > > > >>>>> wrote: > > >>>>>>> > > >>>>>>> Hi everyone, > > >>>>>>> > > >>>>>>> I would like to ask again for feedback on this FLIP. It is a > rather > > >>>>>>> small change but with big impact on usability for structured > data. > > >>>>>>> > > >>>>>>> Are there any objections? Otherwise I would like to continue with > > >>> voting > > >>>>>>> soon. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Timo > > >>>>>>> > > >>>>>>> On 10.04.25 07:54, Timo Walther wrote: > > >>>>>>>> Hi everyone, > > >>>>>>>> > > >>>>>>>> I would like to start a discussion about FLIP-520: Simplify > > >>>>>>>> StructuredType handling [1]. > > >>>>>>>> > > >>>>>>>> Flink SQL already supports structured types in the engine, > > >>> serializers, > > >>>>>>>> UDFs, and connector interfaces. However, currently only Table > API > > >> was > > >>>>>>>> able to make use of them. While UDFs can take objects as input > and > > >>>>>>>> return types, it is actually quite inconvenient to use them in > > >>>>>>>> transformations. > > >>>>>>>> > > >>>>>>>> This FLIP fixes some immediate blockers in the use of structured > > >>> types. > > >>>>>>>> > > >>>>>>>> Looking forward to feedback. > > >>>>>>>> > > >>>>>>>> Cheers, > > >>>>>>>> Timo > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/display/FLINK/ > > >>>>>>>> FLIP-520%3A+Simplify+StructuredType+handling > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >> > > > > > > > >
