corwinjoy commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2742213463
So, to play the devil's advocate, here are some arguments for having encryption configurations encoded as plain strings: 1. Users may want to run datafusion using the CLI. I could see this being valuable for ETL tasks or in other settings, and in this case having the ability to use a set of environment variables to drive encryption or decryption could be quite valuable. 2. There are a number of contexts, such as Lambdas or ephemeral AWS boxes, where it may be a better security practice to have environment variables load dynamically with secrets like encryption keys. This may be a better fit and easier to maintain than a custom built KMS. 3. In a distributed setting, enabling keys via strings is much more doable than trying to distribute objects. So, even though it may be harder to configure encryption from strings, if we can enable this logic it would buy us a lot of flexibility in how end users can access this feature. It also may not restrict us that much. In practice most users want to use just a few KMS classes that connect to standard cloud API such as AWS or Azure and we could provide a few pre-built classes like is done in Java. A final idea could be to use this crate here: https://github.com/dtolnay/typetag We require that classes implementing the KMS trait be serializable. Then, we can serialize them to strings or distribute them across hosts as needed. This constraint might even help us in the end. @adamreeve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
