Thanks Rayn, then I think I have a path forward. And I will file a feature request on thread-safe appendManifest on github. Thanks again.
On Thu, Aug 31, 2023 at 10:04 AM Ryan Blue <b...@tabular.io> wrote: > I think making the operation thread-safe and parallelizing is a good idea. > It should be pretty easy. > > And yes, versions of Iceberg older than the one where that config property > was added would be the ones where it is unsafe. It's probably safe for most > people, but we still can't change the default. > > On Thu, Aug 31, 2023 at 9:59 AM Pucheng Yang <py...@pinterest.com.invalid> > wrote: > >> Thanks Ryan, what might you consider an "older" version of Iceberg? Is it >> fair to say any version before >> https://github.com/apache/iceberg/commit/c3dc9824b381e5e479e356be5e0f4fcf61a9fc37 >> ? If that is the case, my organization controls the Iceberg reader so might >> be a less concern for me. >> >> Another option is to make appendManifests operation thread-safe so that >> we can parallel the execution when needed (also I think we should add >> comment to indicate appendManifests API is not thread-safe, this part seems >> missing, slack discussion: >> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1693410897758349?thread_ts=1693355747.700759&cid=C03LG1D563F >> ) >> >>> > > -- > Ryan Blue > Tabular >
