Re: [DISCUSS] Iceberg Rust Sync Meeting

+1 on sync meeting for iceberg rust. I want to get involved and catch up on the recent developments. For reference, here's the doc we've been using for the pyiceberg sync https://docs.google.com/document/d/1oMKodaZJrOJjPfc8PDVAoTdl02eGQKHlhwuggiw7s9U Best, Kevin On Wed, Oct 9, 2024 at 5:30 AM Xua

Re: [VOTE] Table V3 Spec: Row Lineage

+1 Thanks for shepherding this, Russell! On Tue, Oct 8, 2024 at 7:07 PM Russell Spitzer wrote: > Hi Y'all! > > I think we are more or less in agreement on adding Row Lineage to the spec > apart from a few details which may change a bit during implementation. > Because of this, I'd like to call

Re: [Discuss] Iceberg community maintaining the docker images

I think it's important for a project to remain focused on its core purpose, and I've always advocated for Iceberg to remain a library that is easy to plug into other projects. I think that should be the guide here as well. Aren't projects like Spark and Trino responsible for producing easy to use D

Re: [Discuss] Replace Hadoop Catalog Examples with JDBC Catalog in Documentation

I support the idea of updating the docs to replace the Hadoop catalog example, but I'm wondering why not use a REST Catalog example instead? I saw Ajantha proposed adding Docker images for a REST Catalog adapter [1] so we could potentially use this with a JDBC Catalog backed by SQLite file as a co

[DISCUSS] Iceberg Rust Sync Meeting

Hi, I'm starting this thread to explore the idea of hosting an Iceberg Rust Sync Meeting. In this meeting, we will discuss recent major changes, pending PR reviews, and features in development. It will offer a space for Iceberg Rust contributors to connect and become familiar with each other, h

Re: [Discuss] Replace Hadoop Catalog Examples with JDBC Catalog in Documentation

I would also vote for jdbc catalog, ideally using sqlite as backend as it doesn't require setting up other databases. On Thu, Oct 10, 2024 at 8:42 AM Manu Zhang wrote: > I'd vote for JDBC catalog as it's simple for a quick-start guide. Setting > up a REST Service with docker image could be cumbe

Re: [DISCUSS] Iceberg Rust Sync Meeting

+1 for sync meeting for iceberg rust. These meetings will not be recorded. I think we have meeting records for catalog meetings and community sync, so we should also record this? For time, I would suggest moving it one hour ahead, e.g. 23:00 to 00:00 GTM+8, so that it's a little more friendly t

Re: [Discuss] Replace Hadoop Catalog Examples with JDBC Catalog in Documentation

I'd vote for JDBC catalog as it's simple for a quick-start guide. Setting up a REST Service with docker image could be cumbersome. We can have another page for REST Catalog. Regards, Manu On Thu, Oct 10, 2024 at 2:50 AM Marc Cenac wrote: > I support the idea of updating the docs to replace the

Re: Iceberg View Spec Improvements

Thanks Ryan and everyone who left feedback on the doc. Let me clarify a few things. "Improving the spec" also includes making the implicit assumptions explicitly stated in the spec. Explicitly stating the assumptions is discussed under the "Portable table identifiers" section in the doc. I am onb

Re: [DISCUSS] Defining a concept of "externally owned" tables in the REST spec

Summarizing discussion from today's Iceberg Catalog Community Sync, here were some of the key points: - General agreement on the need for some flavors of mechanisms for catalog federation in-line with this proposal - We should come up with a more fitting name for the endpoint other than

[PROPOSAL] Partially Loading Metadata - LoadTable V2

Hello Dev List, I want to bring this proposal to discussion: https://docs.google.com/document/d/1eXnT0ZiFvdm_Zvk6fLGT_UxVWO-HsiqVywqu1Uk8s7E/edit#heading=h.uad1lm906wz4 It proposes a new LoadTable API (branded LoadTableV2 at the moment) on REST spec that allows partially loading table metada

Re: Iceberg View Spec Improvements

+1 for Steven's comment. There is already an implicit assumption that the catalog names are consistent across engines. The best practice is to not reference identifiers across catalogs, but there isn't much we can do about the assumption here without rewriting SQL to fully qualify identifiers. On

[Discuss] Apache Iceberg 1.6.2 release because of Avro CVE ?

Hi everyone, Since 1.7.0 is still a few weeks away, how about releasing version 1.6.2 with just the Avro version update? The current Avro version in 1.6.1 (1.11.3) has a recently reported CVE: CVE-2024-47561 . [2] I'm happy to coordinate and be the

[DISCUSS] [PyIceberg] Use of asserts to "programming the negative space"

Hello Everyone, I would like to open a discussion about using "assert" in some functions to promote a more defensive programming approach, ensuring that certain assumptions in our code are always validated. The intention here is to propose a recommendation, not a strict rule. What are your tho