I would probably try to just extend my expiration interval if that was
possible to fix the issue since it's basically functioning as a watermark
for state at the moment.
Is our underlying issue here that we cannot determine the lineage of a
Snapshot that has been expired? IE: We know all the files
Hi team,
We've recently bumped into an issue with a particular edge case that messes
with our implementation of leveraging the incremental read and the expire
snapshot features combined.
With incremental read we're relying on the client to preserve the snapshot
that was last used for reading data
I have come across a few similar issues while (mis)using the HiveCatalog.
My analysis was that the HiveCatalog owns the hive client connection pool
but shares it with the underlying TableOperations. Depending on the
application the HiveCatalog can be closed (or its finalize method run after
its GC-
