Hi Mehul, PFA my responses to the questions:
*Zero-Copy Scope:* Zero-copy is guaranteed for the native columnar tiering pipeline (Fluss Arrow Log-> Arrow FixedSizeListVector-> Lance Dataset). Row-oriented writes and Flink SQL query execution follow standard row/array serialization (InternalArray<-> Flink ArrayData). *Dimension as part of Datatype:* The POC already implements dimension as a first class parameter of VectorType, i.e you specify it with Vector(n). *Dimension Validation at Write boundary:* Validation is enforced at the client write boundary (AppendWriter, UpsertWriter, BinaryWriter, and Flink Sink serializer) as well as the Arrow batch writer (ArrowVectorWriter). When writing to a VECTOR(n) column, the writer verifies array.size() == n before serializing or sending data. Failure mode: If an incoming vector has n − 1 or n + 1 elements (e.g., 1535 vs. 1536), an explicit IllegalArgumentException / ValidationException is thrown immediately: "VECTOR dimension mismatch: expected %d elements but got %d." *NULL Semantics* The Vector type itself is nullable, but. it can't contain NULL elements. This means that represents dense embedding values. At the Flink SQL level, it is represented as ARRAY<FLOAT NOT NULL>. *Backwards compatibility with Older clients* Well, right now it would throw an IllegalArgumentException. How is this usually handled in Fluss? Via feature flags or is there another mechanism? Also, now that 1.0 is about to be released maybe older clients can fail in this case given that this would anyways be released in 1.0+ version? We still need to solve this within the 1.0 version range though. Also, at the flink-SQL layer, I don't expect any issues because those would still represent the vector as ARRAY<FLOAT> (not sure about NOT NULL today). *Formal TYPEID Assignment* Thanks for the suggestion. Would call it out in the FIP and maybe send an initial PR just adding this. *Nested Types and Matrices* ARRAY<VECTOR(n)> is syntactically valid and supported across the Fluss type system, DataTypeParser, JSON SerDe, row formats (CompactedRow, IndexedRow), and Arrow schema (List<FixedSizeList<Float32>(n)>) which means one can store, serialize, and read nested vectors / matrices without issue. However, Vector similarity search UDFs (e.g., cosine distance, L2 distance) and lakehouse vector indexing (ANN/KNN) strictly target top-level 1D VECTOR(n) columns. Matrix math, tensor operations, and multidimensional vector indexing over nested ARRAY<VECTOR(n)> are left for future versions. *Lakehouse Integrations* For lakehouse formats that support a native vector type (e.g. Lance and possibly Apache Paimon), Fluss VECTOR(n) will map directly to the engine's native vector / fixed-size array type to support cold-tier vector indexing. For lake formats without native vector support (e.g., Iceberg / standard Parquet), VECTOR(n) will fall back to ARRAY<FLOAT NOT NULL>. *Open Question / Discussion Point:* How should hybrid/union reads behave when combining hot Fluss data (VECTOR(n)) with cold tiers that only support a generic ARRAY<FLOAT>? Should the reader runtime automatically wrap and validate dimensions on the fly, or should we expose the schema as ARRAY<FLOAT> during hybrid scans to keep the reader lightweight? Let me know if these make sense. Happy to address more questions! And once we have these things sorted, I can send a FIP across. Sagar. On Fri, Aug 28, 2026 at 12:44 PM Sagar <[email protected]> wrote: > Thanks Mehul for the feedback. I will address these questions; I think > some of these aspects are covered in the POC but I need to document them. > > Should I wait for these questions to be resolved before starting the FIP > or if the POC looks okay, can I write the FIP and address all these > questions within it? > > Also, unfortunately, I won't be able to join the community call this time > due to a conflict. > > Sagar. > > > On Mon, Aug 24, 2026 at 11:23 PM Mehul Batra <[email protected]> > wrote: > >> Thank you Sagar for putting this all together. I went through the proposal >> and the code, and I have a few questions from my side that I think are >> important to clarify before we move forward: >> >> 1. >> >> *Zero-copy:* I think zero-copy is a good direction, especially for the >> Fluss → Arrow → Lance path. Could we clarify the scope of this claim? >> Through Flink SQL, the data may still be materialized into Flink's own >> representation, so it would be good to distinguish where we can >> actually >> guarantee zero-copy. >> 2. >> >> *Dimension as part of the type:* I prefer keeping the dimension in the >> type itself, e.g. VECTOR(1536), rather than as a table property. This >> makes dimensionality an explicit schema guarantee and avoids downstream >> systems having to infer it from the data. >> 3. >> >> *Dimension validation:* What happens when a VECTOR(1536) column >> receives >> a vector with 1535 or 1537 elements? We should ideally have a clear >> validation error at the write boundary rather than a failure later >> during >> Arrow/Lance conversion. >> 4. >> >> *Null semantics:* Could we clarify how nulls are handled? In >> particular, >> do we support a completely NULL vector, and are individual elements >> within a vector allowed to be NULL? These are two different cases and >> may have different implications for Arrow and Lance. I noticed the POC >> has >> testNullVectorTiering, so it would be good to understand exactly what >> case is covered. >> 5. >> >> *Compatibility:* Since this introduces a new type ID, how do older >> clients behave when they encounter a VECTOR column? We should make sure >> this is a clear and predictable compatibility failure rather than an >> opaque >> error. >> 6. >> >> *Type ID:* The proposal currently mentions something like "e.g. 16". I >> think we should formally reserve the type ID before merging so there >> is no >> possibility of conflicts with other changes. >> 7. >> >> *Nesting/matrices:* Since matrices are explicitly out of scope, can we >> confirm whether ARRAY<VECTOR(1536)> is actually supported in v1? >> Otherwise, the suggested workaround of wrapping vectors in an array >> would >> not be valid. >> 8. >> >> *Other lakehouse integrations:* It would be good to clarify how VECTOR >> is expected to behave across the different lakehouse paths, >> particularly >> where the hot tier has a fixed-size vector but the cold tier may only >> support a list representation. We should make sure the type semantics >> remain consistent across tiering and union reads. >> >> Also, I think this would be a great topic to bring to the community call. >> There are several cross-component implications here, and getting more eyes >> from the community would help us validate the design and catch any >> compatibility or integration concerns early. >> >> Best Regards, >> Mehul Batra >> >
