Re: Iceberg MV Refresh

Thanks Benny for bringing these issues up. I would agree with both of your propositions. Regarding the naming of the fields, we can go with the naming that you suggested. I just wanted to wait if some more people chime in with their opinions. Jan On 20.06.24 23:16, Benny Chow wrote: > So ba

Re: Summary of Iceberg Materialized View Meeting

Great idea, let's try to resolve all issues in the doc until 30th June and if there are still some open points we will have another meeting. Best wishes Jan On 20.06.24 19:10, Walaa Eldin Moustafa wrote: Jan, I think there are a couple of open questions on the doc. Let us discuss them on the

Re: [DISCUSS] Describing REST Server capabilities

> > The general idea behind a capability is that if e.g. a server supports > *views*, then that server must implement all endpoints grouped under that > capability. I haven't thought deeply about this, but is there a reason to be prescriptive about this by grouping endpoints in capabilities? Ano

Re: [DISCUSS] Describing REST Server capabilities

I think the capabilities proposal is intended to let people build in a different way than a versioning system would. It's probably valuable to think through the differences between the approaches. The capabilities that are proposed let catalogs declare sets of features that are supported, like sup

Re: Iceberg MV Refresh

> So basically this is just FYI and it is up to the consumer to assume what to do given the length of time between that timestamp and now? Yes.. some consumers will use it.. some won't. Only the engine producing the materialization will know when it started the refresh job. > Would the decision

Re: Iceberg MV Refresh

So basically this is just FYI and it is up to the consumer to assume what to do given the length of time between that timestamp and now? Would the decision be on the engine implementation side (which does not have the domain knowledge), or would it be on the user's side? In the latter case the user

Re: Iceberg MV Refresh

Piotr, thanks for the Trino pointers. I noticed that Trino stores the refresh start time as a snapshot summary property here . I thi

Re: Iceberg MV Refresh

Benny, is the suggestion to couple the "refresh-start-timestamp-ms" property with a grace period as well? Also, could you clarify which timestamp "refresh-start-timestamp-ms" refers to: (1) Timestamp when refresh is triggered (2) Timestamp when refresh is concluded and the snapshot is written. Als

Re: Summary of Iceberg Materialized View Meeting

Jan, I think there are a couple of open questions on the doc. Let us discuss them on the doc for a week, then meet again if they are still open? Thanks, Walaa. On Fri, Jun 7, 2024 at 12:27 PM Jan Kaul wrote: > No that's great, thank you. I'm thankful for the input. > > Jan > > Am 07.06.2024 17

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, That makes sense. Thanks. Maybe we can already anticipate a little and add a "catalog-level versioning" capability as it's a feature supported by Nessie catalog for instance ? We can also imagine a more generic capability like "version scope". Regards JB On Thu, Jun 20, 2024 at 3:47 

Re: [DISCUSS] Describing REST Server capabilities

Hey JB, If adding UDFs would require adding new endpoints, then you'd also add a *udf *capability when adding UDF support to the REST catalog. That way a client knows whether it's safe to call the UDF endpoints on a given server. Eduard On Thu, Jun 20, 2024 at 1:59 PM Jean-Baptiste Onofré wrote

Re: Iceberg MV Refresh

Hi Benny, on the staleness topic I'd recommend to check how Trino implements materialized views in Iceberg and how it defines staleness. In particular - a view can have defined grace period which defines how stale the data can be for the materialization to be considered useful (defaults to unlimi

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard It looks good to me. I have a question however :) Later, Imagine, we add UDF support in Iceberg. Does it mean that you will need to update REST Spec (ConfigResponse/capabilities) to add this capability ? For consistency, I think it makes sense as I don't think we often add new capabilit

Re: [DISCUSSION] Preparing the Apache iceberg-rust 0.3.0 release

Hi Fokko Thanks ! I'm on gradle/revapi right now (pretty close), I will be back on Avro just after and I will keep you posted :) Regards JB On Thu, Jun 20, 2024 at 10:09 AM Fokko Driesprong wrote: > > Hey JB, > > I agree that we're very close to the Avro release. I'm happy to help if you > nee

Re: [DISCUSSION] Preparing the Apache iceberg-rust 0.3.0 release

Hi, Yes, Avro release includes all modules/artifacts for now (we are discussing about individual releases per language). Regards JB On Thu, Jun 20, 2024 at 8:57 AM Xuanwo wrote: > > Hi, JB > > I'm not familiar with the Avro community. Do they release all packages > including rust together, or

[DISCUSS] Describing REST Server capabilities

Hey everyone, I'd like to bring up the discussion around describing REST server capabilities via the */config* endpoint. There is PR #9940 that describes the OpenAPI spec changes. Mainly we'd like to have a *capabilities* field in the *ConfigResponse*

Re: [DISCUSSION] Preparing the Apache iceberg-rust 0.3.0 release

Hey JB, I agree that we're very close to the Avro release. I'm happy to help if you need a couple of hands, but I'll let you take the lead :) I'm not familiar with the Avro community. Do they release all packages > including rust together, or do they release them separately? With Avro everythin