I think making the operation thread-safe and parallelizing is a good idea. It should be pretty easy.
And yes, versions of Iceberg older than the one where that config property was added would be the ones where it is unsafe. It's probably safe for most people, but we still can't change the default. On Thu, Aug 31, 2023 at 9:59 AM Pucheng Yang <py...@pinterest.com.invalid> wrote: > Thanks Ryan, what might you consider an "older" version of Iceberg? Is it > fair to say any version before > https://github.com/apache/iceberg/commit/c3dc9824b381e5e479e356be5e0f4fcf61a9fc37 > ? If that is the case, my organization controls the Iceberg reader so might > be a less concern for me. > > Another option is to make appendManifests operation thread-safe so that we > can parallel the execution when needed (also I think we should add comment > to indicate appendManifests API is not thread-safe, this part seems > missing, slack discussion: > https://apache-iceberg.slack.com/archives/C03LG1D563F/p1693410897758349?thread_ts=1693355747.700759&cid=C03LG1D563F > ) > >> -- Ryan Blue Tabular
