Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-06-28 Thread via GitHub
alamb closed issue #15216: Support integration with Parquet modular encryption URL: https://github.com/apache/datafusion/issues/15216 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-06-03 Thread via GitHub
adamreeve commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2933379249 I've created a draft PR with an example of what integration with a KMS could look like: https://github.com/apache/datafusion/pull/16237 Any feedback would be much apprec

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-05-05 Thread via GitHub
adamreeve commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2852780352 > I don't really understand the reason for using `Any` Actually I think I remember now that this would let us include structs in config types in `datafusion::common::con

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-05-05 Thread via GitHub
adamreeve commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2852529965 > Here is how spark does encryption configuration My understanding of how this works in Spark from reading this and looking at some of the code: * Spark requires spec

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-05-02 Thread via GitHub
alamb commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2847108736 Here is how spark does encryption configuration https://spark.apache.org/docs/latest/sql-data-sources-parquet.html -- This is an automated message from the Apache Git Serv

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-04-28 Thread via GitHub
alamb commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2836077703 > With the KMS API not being included in arrow-rs but being built as a third-party crate (https://github.com/apache/arrow-rs/pull/7387#issuecomment-2819908130), I would assume we

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-04-23 Thread via GitHub
adamreeve commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2826229093 With the KMS API not being included in arrow-rs but being built as a third-party crate (https://github.com/apache/arrow-rs/pull/7387#issuecomment-2819908130), I would assume

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-04-23 Thread via GitHub
corwinjoy commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2825947274 @alamb @adamreeve With the modular encryption essentially complete in arrow-rs, we are interested in beginning to move forward with adding support for this feature in datafus

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-04-05 Thread via GitHub
adamreeve commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2739092901 I had a go at seeing if I could use this callback based configuration approach to integrate with encryption without Datafusion needing to know anything about Parquet encryptio

Re: [I] Support integration with Parquet modular encryption [datafusion]

2025-04-04 Thread via GitHub
corwinjoy commented on issue #15216: URL: https://github.com/apache/datafusion/issues/15216#issuecomment-2742213463 So, to play the devil's advocate, here are some arguments for having encryption configurations encoded as plain strings: 1. Users may want to run datafusion using the CLI. I