Hi Ha, Thanks for the question! Typically, when concurrent writes happen to the same partition, the writer must retry the operation. This is because the state of the table (and the partition) has changed, and the current write cannot be safely applied to the updated state. For example, if the operation involves overwriting or deleting data, it needs to be applied to the new table state. In such cases, the client will need to regenerate both the data files and the metadata files before retrying the commit.
However, there's an optimization available for certain types of write operations that are independent of the new table state, such as appending new data to a partition. In these cases, retrying the commit will only require regenerating the metadata files, not the data files themselves. This is what the proposal Yufei mentioned is referring to. Clients can submit file-level changes, and the catalog server will generate the necessary metadata before committing the new files. Another option is to reduce concurrent write conflicts by creating partition-level isolation. Iceberg supports multi-level partitioning, so you can partition by multiple fields (e.g., partition by date and by cluster). This can help isolate concurrent writes to different partitions. However, this approach can introduce challenges such as the small-files problem. Hope this helps! Feel free to reach out if you'd like more detailed guidance on implementing any of these approaches, or if there are other specifics you'd like to discuss. Best, Kevin Liu On Tue, Dec 3, 2024 at 3:12 PM Yufei Gu <flyrain...@gmail.com> wrote: > If you’re looking for finer-grained isolation beyond the snapshot level, > the closest feature currently *WIP* is *Fine-Grained Commit* in the REST > catalog. You can find more details here: Fine-Grained Commit Design > Document > <https://docs.google.com/document/d/1n-cEE4-vFreTLnUTPgo7U8ih44MFo1ZHjyk4NCxxalc/edit?usp=sharing> > . > > Yufei > > > On Tue, Dec 3, 2024 at 2:41 PM Ha Cao <ha....@twosigma.com> wrote: > >> Hello, >> >> >> >> I have some concurrent writes to the same partition and they overlap in >> data files, and the isolation level is snapshot. Expectedly, I get this >> ValidationException thrown from this line >> <https://sourcegraph.com/github.com/linkedin/iceberg@78a31ff699dd019ed760ef70c8cbf8acbd74bed6/-/blob/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java?L266-271>. >> I can certainly retry the write from the beginning, rewrite the metadata >> and data files and commit again, but wonder if there is something internal >> in Iceberg that can help me with that instead. >> >> Thank you! >> >> Best, >> >> Ha >> >
