Re: Implement position deletes in the Flink connector

2023-03-09 Thread Tamas Sule
that were upserted > and write deletes against those IDs for the new data files. That would > probably be better than retrying the entire checkpoint. From Iceberg's > perspective, recovery choices are left up to the engine. Flink might > recover as I described, but engines like Spar

Implement position deletes in the Flink connector

2023-03-08 Thread Tamas Sule
Hello, We are experimenting with implementing position deletes in the Flink connector to provide better support between components in our stack as equality deletes are not widely supported yet. I identified an issue which (potentially) makes this impossible to do correctly in upsert mode and I'd l

Re: Re: Issue when trying to read from table created using TestTables class

2023-02-24 Thread Tamas Sule
sily be that the state of the helper is > being cleared. I think you're probably better off trying to figure out why > the table is missing or was deleted, rather than switching. > > On Fri, Feb 24, 2023 at 2:55 AM Tamas Sule > wrote: > >> Hi Ryan, >> >> Tha

RE: Re: Issue when trying to read from table created using TestTables class

2023-02-24 Thread Tamas Sule
u trying to do? TestTables is intended to be an internal helper > class that manages some tables in memory and shouldn't actually be used > outside of Iceberg or the tests that currently rely on it. > > Ryan > > On Thu, Feb 23, 2023 at 8:55 AM Tamas Sule > wrote: > >

Issue when trying to read from table created using TestTables class

2023-02-23 Thread Tamas Sule
Hello, I’m having trouble working with the TestTables class: I’m writing a test which hits a code path that uses IcebergGenerics.read, which fails with the following error: Cannot load metadata: metadata file location is null. Looking at the TestTableOperations class, it doesn’t actually write out