gabotechs commented on code in PR #19707: URL: https://github.com/apache/datafusion/pull/19707#discussion_r2676833472
########## datafusion/substrait/src/variation_const.rs: ########## @@ -55,6 +55,13 @@ pub const TIME_64_TYPE_VARIATION_REF: u32 = 1; pub const DEFAULT_CONTAINER_TYPE_VARIATION_REF: u32 = 0; pub const LARGE_CONTAINER_TYPE_VARIATION_REF: u32 = 1; pub const VIEW_CONTAINER_TYPE_VARIATION_REF: u32 = 2; +/// Used for the arrow type [`DataType::FixedSizeList`]. +/// +/// The size of the fixed list is encoded by adding it to this base value. +/// For example, `FixedSizeList(10)` would use variation reference `3 + 10 = 13`. +/// +/// [`DataType::FixedSizeList`]: datafusion::arrow::datatypes::DataType::FixedSizeList +pub const FIXED_SIZE_LIST_TYPE_VARIATION_REF: u32 = 3; Review Comment: This is a smart way of representing the size of a list! However, I imagine this can easily backfire in the future if different type variation refs are needed for lists, as this approach eliminates the option to add new ones. I see two possible paths forward: 1. Convince the Substrait community that adding a new FixedSizeList type to the Substrait specification is worth it. There's an open issue about this https://github.com/substrait-io/substrait/issues/873 2. Implement this as a user defined type, same as what we did for Float16: https://github.com/apache/datafusion/blob/main/datafusion/substrait/src/logical_plan/producer/types.rs#L111-L120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
