boneanxs commented on PR #8015:
URL: https://github.com/apache/hudi/pull/8015#issuecomment-1441146491
@stream2000 Hi, I'm quite confused with the behavior here, for
`insert_overwrite_table` with `bulk_insert` enabled, we'll directly drop the
data first, and then do the write operation, this breaks the ACID transactions,
and the weird thing is this only happens when we enable `bulk_insert`.
```scala
if (mode == SaveMode.Overwrite && tableExists && operation !=
WriteOperationType.INSERT_OVERWRITE_TABLE) {
// When user set operation as INSERT_OVERWRITE_TABLE,
// overwrite will use INSERT_OVERWRITE_TABLE operator in
doWriteOperation
log.warn(s"hoodie table at $tablePath already exists. Deleting
existing data & overwriting with new data.")
fs.delete(tablePath, true)
tableExists = false
}
```
while for `insert_overwrite`, we don't support `bulk_insert`, and will keep
the old files.
I'm thinking we should firstly don't allow for `bulk_insert` for
`insert_overwrite` and `insert_overwrite_table`, after we can fully support
`bulk_insert` for `insert_overwrite` and `insert_overwrite_table`(while should
also keep the overwritten data to keep the consistent behavior), then we can
consider to have tests to cover this.
cc @alexeykudinkin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
