This isn't something that we can set to `true` because it is a
forward-incompatible change. That's why we added a flag.

However, we should make sure that this is the default behavior in v2
tables, since it is safe for v2 (where inheritance happens automatically).
If I remember correctly, we still rewrite by default in v2 even though it's
safe.

On Thu, Aug 31, 2023 at 9:40 AM Pucheng Yang <py...@pinterest.com.invalid>
wrote:

> Hi community,
>
> Table prop "compatibility.snapshot-id-inheritance.enabled" is introduced
> to avoid manifest rewrite if possible (PR:
> https://github.com/apache/iceberg/commit/c3dc9824b381e5e479e356be5e0f4fcf61a9fc37
> ).
>
> During my recent investigation on a super long snapshot table creation on
> a huge table, I found that the majority of time spent is on
> manifest rewrite during appendManifest operation (code link:
> https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L279)
> due to this table prop being default as False.
>
> Russell brought a point of considering setting this table prop to True and
> suggested I start a discussion on the dev list.
>
> Correct me if I am wrong, after looking at the code, my understanding of
> the implications are:
> 1. There will be manifests not having snapshot id in some cases. For
> example, during snapshot table creation, we append manifest files without
> snapshot id to a table.
> 2. The manifest file name will be the name specified during the "first
> write" (the "second write" is manifest copy during appendManifest
> operation). An example will be "stage-%d-task-%d-manifest-%s" which is the
> name used during snapshot creation, but since the last param is UUID, it
> should be fine.
>
> Would like to hear from you, thanks!
>


-- 
Ryan Blue
Tabular

Reply via email to