Talked with a bunch of people on ApacheCon, and I’d still very much like to see int8 and float16 vector support in Iceberg.
I’ve created a draft PR to add these types: https://github.com/apache/iceberg/pull/18485. Parquet already provides a logical type for int8 and a physical type for float16, so adding support in Iceberg seems like a reasonable effort. I’d be happy to spearhead this work so we can get these types into V4, if the community agrees. Thanks, Peter Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, 2:33): > Thank you everyone for the discussion today! The meeting notes can be > found here: > https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he > and below is a TLDR version: > - Focus on a 1D numeric vector type with fixed length and non-null, finite > coordinates; +0 and -0 remain valid. > - Keep the initial spec simple and build on Parquet’s emerging vector > representation. > - Initially support int, long, float, and double; defer narrower/special > types such as int8 as additional Iceberg primitive types (while allowing > such parquet representation for vector). > - Keep higher level properties such as distance functions out of scope, > and revisit them with the index design. Sparse vectors are also out of > scope for now: alternative options exist and waiting for concrete use cases. > - For formats without native vector support, use a list representation > with writer enforcement. Will explore an Avro vector logical type with Avro > community; ORC support remains open. > > I'll update the spec PR in the next few days (also appreciate the quick > feedback from everyone!) > > Thanks, > Yan > > On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: > >> Thank you Steven for helping secure the time slot for the vector type >> sync! And thanks everyone for your feedback on the design doc. >> >> In preparation for tomorrow's discussion, I've created a draft PR >> https://github.com/apache/iceberg/pull/18416 for the spec changes based >> on the progress in the parquet community on the Vector type proposal [1 >> <https://github.com/apache/parquet-format/pull/624>] [2 >> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >> , and will continue to iterate on both the spec PR and the design doc based >> on the discussion feedback. >> >> Thanks, >> Yan >> >> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> wrote: >> >>> Hi, >>> >>> As some of you are aware, there is an ongoing discussion about a new >>> *Vector* type in both the Parquet and Iceberg communities. >>> >>> Iceberg Vector type design doc (from Yan Yan): >>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>> >>> To accelerate the Vector type discussion, we are reusing/extending >>> Thursday's biweekly series to cover both File and Vector types (*starting >>> on Oct 8*). Since the file type discussion is reaching consensus, more >>> time will likely be spent on the Vector type discussion. The corresponding >>> dev calendar event title has been updated. >>> >>> Thanks, >>> Steven >>> >>
