Iceberg View Spec Improvements

2024-10-07 Thread Walaa Eldin Moustafa
Hi Everyone, As part of our discussions on the Materialized View (MV) spec, the topic of "SQL table identifiers" has been a constant source of complexity. After several iterations, the community has agreed not to use SQL table identifiers in the table-side representation of MVs. However, that stil

Re: Clarification on DayTransform Result Type

2024-10-07 Thread Kevin Liu
Thanks for confirming! To close the loop on this issue, we have added more documentation about the `result_type` function in PyIceberg. This clarifies the physical and display representations of partition transforms. For DayTransform, the physical representation is `int`, while the display represe

Re: Clarification on DayTransform Result Type

2024-10-07 Thread rdb...@gmail.com
Yes. When we return the Spark type, it shows up as date and Spark correctly displays the value. On Mon, Sep 30, 2024 at 9:56 AM Kevin Liu wrote: > Thank you both for the insights and context. > > As Russell pointed out, the "day partition transform" result is true of > int type. The Types.DateTy

Discuss the property write.metadata.metrics.max-inferred-column-defaults

2024-10-07 Thread Joshua Kolash
Hi, I would like to discuss the property write.metadata.metrics.max-inferred-column-defaults issue created here https://github.com/apache/iceberg/issues/11253 The fundamental issue is that this doesn't work for all kinds of wide schemas. I need to support many arbitrary schemas so having per sche