Re: Meeting Minutes 2024-10-02

Thanks Brian for sharing the notes. Some questions here. Target release date set for October 31st, 2023 > Should be 2024?😀 Proposal to create new Iceberg C++ library approved In this thread[1], Xuanwo and Renjie mentioned iceberg-rust implementation and c++ bindings. Do you have a strong opinion

Re: Spec changes for deletion vectors

Hi Ryan, Do you mean the doc Improve Position Deletes in V3 by Anton? I don't recall Anton used the term "deletion vector" in his proposal. On Sat, Oct 12, 2024 at 12:30 AM Micah Kornfield wrote: > I

Re: Iceberg View Spec Improvements

Hi Benny, > we don't need to list out such restrictions because they really depend on the setup I do not think this is correct. The restrictions do not depend on the setup. They rather dictate it. All restrictions discussed in this thread do that one way or the other. The single engine (Dremio)

Re: Iceberg View Spec Improvements

Hi Russell Yes, you listed out the requirements to make the two Spark engines case work. Basically, it allows each engine to dynamically resolve the table identifiers under the correct catalog name. Hello Walla IMO, we don't need to list out such restrictions because they really depend on the s

Re: Iceberg View Spec Improvements

Benny, "Iceberg View Spec Improvements" includes documenting what is supported and what is not. You listed a few restrictions. Many of them are not documented on the current spec. Documenting them is what this thread is about. We are trying to reach a consensus on the necessary constraints (so we a

Re: Spec changes for deletion vectors

I think it might be worth mentioning the current proposal makes some, mostly minor, design choices to try to be compatible with Delta Lake deletion vectors. I think there might be a general philosophical question on what compromises the community is willing to make for compatibility reasons. On T

Re: [DISCUSS] Iceberg Rust Sync Meeting

Thank you for starting this thread Xuanwo, I'm +1 for a Iceberg Rust meeting. Regarding the meeting time, I believe the Iceberg Catalog Community sync happens two consecutive weeks, at the same time as the Iceberg community sync, when there isn't the tri-weekly Iceberg meeting. For example, if

Re: Iceberg View Spec Improvements

The two Spark engines case is the only case I'm stuck on. I'm not sure how you can define a view that works regardless of configuration unless you require that the catalog holding the view is the default catalog (which is a config) and you also only produce catalog-less identifiers. On Fri, Oct 11

Re: Iceberg View Spec Improvements

Having spent some time testing Nessie views with multiple engines (Dremio + Spark) using different catalog names and different namespaces, I tend to agree with Dan and Amogh that the current view spec is fine. Unlike tables, I think when it comes to views, engines have to "work together" if they e

Re: [DISCUSS] Iceberg Rust Sync Meeting

> For reference, here's the doc we've been using for the pyiceberg sync > https://docs.google.com/document/d/1oMKodaZJrOJjPfc8PDVAoTdl02eGQKHlhwuggiw7s9U Thank you, Kevin. This document is excellent, and I'll use it as a template. > I think we have meeting records for catalog meetings and commun

Re: [DISCUSS] REST: Standardize vended credentials in Spec

Hi Eduard, The latest REST spec change PR LGTM overall. I think it does make sense to avoid putting vendor-specific credential properties into the REST spec itself. However, I still have a couple of comments. (reposting from GH) REST Servers have to provide concrete values for vendor-specific cr

Re: [DISCUSS] Iceberg Rust Sync Meeting

Thank Xuanwo for raising this. I'm also interested and excited to catch up and better participate in the community. As for the time, I think we don't need to use the same time as Iceberg Sync Meeting, and can choose a better time according to the Iceberg Rust developers? (Perhaps we can have a pol