Hello,

We have a security requirement to client side encrypt flink state for certain 
flink applications that process sensitive data. Currently, there is no feature 
that supports this out of the box on AWS S3 backend.

One way to do it is to use flink-s3-fs-hadoop compiled against hadoop 3.3.2 for 
checkpoints as hadoop 3.3.2 provides out of the box AWS client side encryption 
using AWS KMS.

I have filed a JIRA requesting the shaded Hadoop version upgrade in 
flink-filesystems module: https://issues.apache.org/jira/browse/FLINK-27333 
<https://issues.apache.org/jira/browse/FLINK-27333> 

We were able to experiment with an internal build on our flink jobs and saw no 
issues in checkpoints in flink 1.14.3 after the Hadoop version change. Wanted 
to check with the community if there are any concerns in moving ahead with this 
version upgrade. If there are no concerns, the JIRA can be assigned to me to 
try to take the changes to the main flink repo. 

Thanks,
Chinmay

Reply via email to