Ssv-21 opened a new issue, #11349:
URL: https://github.com/apache/hudi/issues/11349
**The problem:**
I am trying to perform a hard delete operation on a HUDI table but cannot
delete the data. My setup is pretty straightforward. I use a normal glue job
with python-spark script to create the hudi tables and use the Glue Data
Catalogue.
**To Reproduce**
the following configs are used to create/write into hudi table,
```
hudi_options: dict = {"hoodie.table.name": "test_table",
"hoodie.datasource.write.recordkey.field": "uid,
cust_id",
"hoodie.datasource.write.partitionpath.field": "date",
"hoodie.datasource.write.table.name": "test_table",
"hoodie.datasource.write.operation": "append",
"hoodie.datasource.write.precombine.field":
"insert_timestamp",
"hoodie.write.lock.provider":
"org.apache.hudi.client.transaction.lock.InProcessLockProvider",
"hoodie.datasource.write.reconcile.schema": "true",
"hoodie.datasource.write.storage.type": "COPY_ON_WRITE",
"hoodie.cleaner.policy.failed.writes": "LAZY",
"hoodie.write.concurrency.mode":
"OPTIMISTIC_CONCURRENCY_CONTROL",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.database": "test",
"hoodie.datasource.hive_sync.table": "test_table",
"hoodie.datasource.hive_sync.partition_fields": "date",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.enable.data.skipping": "true",
"hoodie.datasource.hive_sync.support_timestamp": "true",
"path": path,
}
```
not using any lock provider.
I tried the following approaches, I am not sure this will help in
reproducing the same.
1. EmptyHoodieRecordPayload:-
```
hudiOptions={
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.recordkey.field": "uid, cust_id",
"hoodie.datasource.write.precombine.field": "insert_timestamp",
"hoodie.datasource.write.partitionpath.field": "date",
"hoodie.table.name": "test_table"}
DF.write.format('org.apache.hudi') \
.option('hoodie.datasource.write.operation', delete') \
.option('hoodie.datasource.write.payload.class',
'org.apache.hudi.common.model.EmptyHoodieRecordPayload') \
.options(**hudiOptions) \
.mode('append') \
.save(path)
```
2. Normal delete write operation:
```
DF.write.format('org.apache.hudi') \
.option('hoodie.datasource.write.operation', 'delete') \
.options(**hudiOptions) \
.mode('append') \
.save(path)
```
3. Normal spark SQL:
```
spark.sql("set hoodie.schema.on.read.enable=true")
spark.sql("DELETE FROM hudi_cow_nonpcf_tbl WHERE uid = 1")
```
All three approaches weren't able to delete the data and also didn't throw
any errors.
**Expected behavior**
the data should be deleted from the table.
**Environment Description**
* Hudi version : 0.12.1
* Glue : 4.0
* Spark version : 3.3
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
**Additional context**
I am not adding any additional jars in my glue job apart from the following
configs,
```
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
--conf
'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
--conf
'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
--conf 'spark.sql.hive.convertMetastoreParquet=false'
```
**Stacktrace**
not getting any error and there is a commit initiated in the table.
But the data is not getting deleted.
**Expectation**
I need some guidance in setting up the configurations that allows DML
operations on the HUDI table. Kindly help me out for,
`native AWS glue + Glue Data Catalogue [not using any lock provider and no
extra jars]`
kind of set-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
