stream2000 commented on PR #8015:
URL: https://github.com/apache/hudi/pull/8015#issuecomment-1441246202
> @stream2000 Hi, I'm quite confused with the behavior here, for
`insert_overwrite_table` with `bulk_insert` enabled, we'll directly drop the
data first, and then do the write operation, this breaks the ACID transactions,
and the weird thing is this only happens when we enable `bulk_insert`.
>
> ```scala
> if (mode == SaveMode.Overwrite && tableExists && operation !=
WriteOperationType.INSERT_OVERWRITE_TABLE) {
> // When user set operation as INSERT_OVERWRITE_TABLE,
> // overwrite will use INSERT_OVERWRITE_TABLE operator in
doWriteOperation
> log.warn(s"hoodie table at $tablePath already exists. Deleting
existing data & overwriting with new data.")
> fs.delete(tablePath, true)
> tableExists = false
> }
> ```
>
> while for `insert_overwrite`, we don't support `bulk_insert`, and will
keep the old files.
>
> I'm thinking we should firstly don't allow for `bulk_insert` for
`insert_overwrite` and `insert_overwrite_table`, after we can fully support
`bulk_insert` for `insert_overwrite` and `insert_overwrite_table`(which means
keep the overwritten data if using `bulk_insert` to keep the consistent
behavior), then we can consider to have tests to cover this.
>
> cc @alexeykudinkin
I agree that we should unify the semantics between insert overwrite
partition and insert overwrite table when bulk_insert is enabled. I will push a
commit later that directly throw exception when bulk_insert is enabled in
insert overwrite table/partittion.
cc @leesf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
