Re: [DISCUSS] Pre-Proposal: Improving Merge-On-Read Query Performance With Indexing

I am glad to see that folks are thinking about this problem. I am looking forward to a formal proposal/design doc to discuss details! Overall, this aligns with what we discussed in the community earlier w.r.t. the future of equality deletes and streaming upserts. If I were to summarize, we have th

Re: [VOTE] Add encryption key updates to REST spec

+1 (binding) On Thu, May 8, 2025 at 10:37 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > On Thu, May 8, 2025 at 9:29 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> +1 (binding) >> >> On Thu, May 8, 2025 at 5:23 PM Ryan Blue wrote: >> >>> Hi everyone, >>> >>> I’

Re: [DISCUSS] Pre-Proposal: Improving Merge-On-Read Query Performance With Indexing

Hi Zheng, Steven, Amogh and Gyula. Thank you all for the feedback! I agree with everyone, we need to narrow down the scope of this optimization. The primary issue I'm trying to address is the slow read performance caused by the growing number of equality delete files(streaming CDC scenarios). The

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

If a table referenced by the view SQL query is deleted and recreated, is the regular view still valid? If yes, it won't be correct to use resolved UUID to validate the name reference. On the materialized view side, we are aligned that UUID is necessary to detect this scenario and mark the refresh s

[VOTE] Add commit timestamp to CommitReport

Hi all, I'd like to start a vote to add commit timestamp `timestamp-millis` to CommitReport in PR 12990 . The timestamp info is valuable to schedule maintenance jobs, but currently we need to look it up from the snapshot metadata table. Please take a

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

Hi Dan, Thanks for the clarification! I agree that UUIDs should not be stored in the SQL definition of the view. Just to clarify: the proposal isn't about modifying the view definition itself, but rather about how table identifiers are represented in Iceberg view metadata, which naturally depends

Re: [DISCUSS] Table Identifiers in Iceberg View Spec

I don't think we want to include the resolved table UUIDs in the view definition, but rather in the storage table state. You can still resolve whether those drift at some point, but I don't feel like it's a good idea to capture data in the view that we may allow to drift if there isn't any require

Re: [VOTE] Add encryption key updates to REST spec

+1 (binding) On Thu, May 8, 2025 at 9:29 AM Eduard Tudenhöfner wrote: > +1 (binding) > > On Thu, May 8, 2025 at 5:23 PM Ryan Blue wrote: > >> Hi everyone, >> >> I’d like to raise a vote for committing PR 12987 >> that adds table updates >> for encr

Re: [VOTE] Add encryption key updates to REST spec

+1 (binding) On Thu, May 8, 2025 at 5:23 PM Ryan Blue wrote: > Hi everyone, > > I’d like to raise a vote for committing PR 12987 > that adds table updates > for encryption keys, AddEncryptionKey and RemoveEncryptionKey. These are > needed to maintai

[VOTE] Add encryption key updates to REST spec

Hi everyone, I’d like to raise a vote for committing PR 12987 that adds table updates for encryption keys, AddEncryptionKey and RemoveEncryptionKey. These are needed to maintain the encryption key list in v3 metadata. Please vote in the next 72 hours

Re: [DISCUSS] Pre-Proposal: Improving Merge-On-Read Query Performance With Indexing

Thank you for the proposal! I agree with what had been said above that we need to narrow down the scope here and what is the primary target for the optimization. As Amogh has also pointed out, CDC (streaming) read performance (with equality deletes) would be one of the biggest beneficiaries of th

Re: [DISCUSS] Pre-Proposal: Improving Merge-On-Read Query Performance With Indexing

Thank you for the proposal Xiaoxuan! I think I agree with Zheng and Steven's point that it'll probably be more helpful to start out with more specific "what" and "why" (known areas of improvement for Iceberg and driven by any use cases) before we get too deep into the "how". In my mind, the specif