[DISCUSS] Define calendar used in specification?

2024-09-11 Thread Micah Kornfield
At the moment, the specification is ambiguous on which calendar is used for temporal conversion/writing [1]. Reading the java code it appears it is using Java's OffsetDateTime which conforms to ISO8601 [2]. ISO8601 appears to explicitly disallow the Julian calendar (but only says proleptic gregori

Re: Time-based partitioning on long column type

2024-09-11 Thread Micah Kornfield
> > Maybe we could update the time-based partition functions to be applied to > a long column directly. It would treat that column like a timestamp in > milliseconds. Would that work? I need to think more about the implications > of doing that, but I don't think that we currently have an issue with

Re: [DISCUSS] Improving Position Deletes in V3

2024-09-11 Thread Russell Spitzer
+1 I am on board, all of my doubts are resolved On Thu, Sep 5, 2024 at 3:42 AM Jean-Baptiste Onofré wrote: > Hi Anton > > Sorry for the late reply on this proposal. > I like it ! It looks good to me (I have a few minor comments). > > It would be great to include this spec update in V3. > Please

Re: [DISCUSS] September board report

2024-09-11 Thread Yufei Gu
LGTM. Thanks Ryan! Yufei On Wed, Sep 11, 2024 at 8:30 AM Xuanwo wrote: > Thank you, this report looks good to me. Happy to see iceberg-rust been > mentioned. > > On Wed, Sep 11, 2024, at 23:02, Jean-Baptiste Onofré wrote: > > Hi Ryan, > > > > It looks good to me. Thanks ! > > > > Regards > > J

Re: [Discuss] test logging is broken and Avro 1.12.0 upgraded slf4j-api dep to 2.x

2024-09-11 Thread Steven Wu
Following up on the discussion from the community sync meeting. Right now, Iceberg test code is in the 3rd row in the table pasted below. With the recent Avro 1.12 upgrade (and slf4j 2.x), the main code is also affected. That means downstream applications (Spark, Trino, Flink, ...) may run into the

Re: [DISCUSS] Additional language implementations for Iceberg Puffin reader/writer

2024-09-11 Thread Anton Okolnychyi
If C++ engines prefer not to depend on Iceberg Rust, I actually don't see a problem with having a separate C++ project even if it only contains Puffin readers/writers. The important part is to avoid multiple C++ writer/reader implementations in different engines. There were concerns with having on

Re: [DISCUSS] September board report

2024-09-11 Thread Xuanwo
Thank you, this report looks good to me. Happy to see iceberg-rust been mentioned. On Wed, Sep 11, 2024, at 23:02, Jean-Baptiste Onofré wrote: > Hi Ryan, > > It looks good to me. Thanks ! > > Regards > JB > > On Tue, Sep 10, 2024 at 11:43 PM rdb...@gmail.com wrote: >> >> Hi everyone, >> >> It’s

Re: [DISCUSS] September board report

2024-09-11 Thread rdb...@gmail.com
Thanks for the updates! I'll add those. On Wed, Sep 11, 2024 at 8:02 AM Jean-Baptiste Onofré wrote: > Hi Ryan, > > It looks good to me. Thanks ! > > Regards > JB > > On Tue, Sep 10, 2024 at 11:43 PM rdb...@gmail.com > wrote: > > > > Hi everyone, > > > > It’s time for another ASF board report! H

Re: [DISCUSS] September board report

2024-09-11 Thread Jean-Baptiste Onofré
Hi Ryan, It looks good to me. Thanks ! Regards JB On Tue, Sep 10, 2024 at 11:43 PM rdb...@gmail.com wrote: > > Hi everyone, > > It’s time for another ASF board report! Here’s my current draft. Please reply > if you think there is something that I should add or change. Thanks! > > Ryan > > Desc

Re: [DISCUSS] September board report

2024-09-11 Thread Russell Spitzer
This looks good to me On Wed, Sep 11, 2024 at 12:35 AM Steven Wu wrote: > > Flink Range distribution for Sinks > > It is already included in Ryan's draft > > > Flink Source V2 improvements and V1 deprecation to prepare for Flink 2.0 > > This is still ongoing. There is a blocking issue with FileI

Re: [DISCUSS] Iceberg Materialzied Views

2024-09-11 Thread Walaa Eldin Moustafa
I think this type of discussion is exactly what motivates a clarification in the view spec so that we can resolve MV lineage. Will create separate thread for view spec clarification. Following up on Jan’s point, yes I agree in order to support catalog name, it should be at the representation level

Re: [DISCUSS] Iceberg Materialzied Views

2024-09-11 Thread Jan Kaul
Hi Benny, I think that identifiers only being defined for a certain representation is exactly what we want. Each representation can define their own identifiers that then map to an UUID. This way the "catalog_name" of the identifier for a "Spark" dialect can be different then for a "Dremio" d