Re: ACID with Hive/Kylin

2023-12-12 Thread Nam Đỗ Duy
Thank you both of you for your valuable information. I will test and revert soon. Best regards On Tue, Dec 12, 2023 at 2:39 PM Xiaoxiang Yu wrote: > I don't know GDPR very well. Here is my understanding. > > For hive and hdfs, you can consider using these techniques which support > ACID in Spar

Re: ACID with Hive/Kylin

2023-12-11 Thread Xiaoxiang Yu
I don't know GDPR very well. Here is my understanding. For hive and hdfs, you can consider using these techniques which support ACID in Spark and Hive(I recommend first one): 1) Delta Lake, https://docs.databricks.com/en/security/privacy/gdpr-delta.html 2) Hive ACID table, here is a link, https://

Re: ACID with Hive/Kylin

2023-12-11 Thread ShaoFeng Shi
Hi Nam, As Kylin is used to store the aggregated data, there should be no PII information. (if you use Kylin to manage person level data, that is not a good case). If you do need to delete certain personal data, refresh the whole index or some partitions is what we can do. Best regards, Shaofen

ACID with Hive/Kylin

2023-12-11 Thread Nam Đỗ Duy
Dear Xiaoxiang, Sirs/Madams I face an issue with deleting data of user according to GPDR-like policy which means when user send request to delete their personal data, we need to delete it from all system, that means to delete data: 1- from Kylin index (cube) 2- from Hive 3- from HDFS Have you ha