Thanks Mehul for the feedback. I will address these questions; I think some of these aspects are covered in the POC but I need to document them.
Should I wait for these questions to be resolved before starting the FIP or if the POC looks okay, can I write the FIP and address all these questions within it? Also, unfortunately, I won't be able to join the community call this time due to a conflict. Sagar. On Mon, Aug 24, 2026 at 11:23 PM Mehul Batra <[email protected]> wrote: > Thank you Sagar for putting this all together. I went through the proposal > and the code, and I have a few questions from my side that I think are > important to clarify before we move forward: > > 1. > > *Zero-copy:* I think zero-copy is a good direction, especially for the > Fluss → Arrow → Lance path. Could we clarify the scope of this claim? > Through Flink SQL, the data may still be materialized into Flink's own > representation, so it would be good to distinguish where we can actually > guarantee zero-copy. > 2. > > *Dimension as part of the type:* I prefer keeping the dimension in the > type itself, e.g. VECTOR(1536), rather than as a table property. This > makes dimensionality an explicit schema guarantee and avoids downstream > systems having to infer it from the data. > 3. > > *Dimension validation:* What happens when a VECTOR(1536) column receives > a vector with 1535 or 1537 elements? We should ideally have a clear > validation error at the write boundary rather than a failure later > during > Arrow/Lance conversion. > 4. > > *Null semantics:* Could we clarify how nulls are handled? In particular, > do we support a completely NULL vector, and are individual elements > within a vector allowed to be NULL? These are two different cases and > may have different implications for Arrow and Lance. I noticed the POC > has > testNullVectorTiering, so it would be good to understand exactly what > case is covered. > 5. > > *Compatibility:* Since this introduces a new type ID, how do older > clients behave when they encounter a VECTOR column? We should make sure > this is a clear and predictable compatibility failure rather than an > opaque > error. > 6. > > *Type ID:* The proposal currently mentions something like "e.g. 16". I > think we should formally reserve the type ID before merging so there is > no > possibility of conflicts with other changes. > 7. > > *Nesting/matrices:* Since matrices are explicitly out of scope, can we > confirm whether ARRAY<VECTOR(1536)> is actually supported in v1? > Otherwise, the suggested workaround of wrapping vectors in an array > would > not be valid. > 8. > > *Other lakehouse integrations:* It would be good to clarify how VECTOR > is expected to behave across the different lakehouse paths, particularly > where the hot tier has a fixed-size vector but the cold tier may only > support a list representation. We should make sure the type semantics > remain consistent across tiering and union reads. > > Also, I think this would be a great topic to bring to the community call. > There are several cross-component implications here, and getting more eyes > from the community would help us validate the design and catch any > compatibility or integration concerns early. > > Best Regards, > Mehul Batra >
