wiedld opened a new pull request, #11444:
URL: https://github.com/apache/datafusion/pull/11444
## Which issue does this PR close?
Here's a proposed cleanup. **I'm not sure yet it this should be done**, so
it's a draft.
## Rationale for this change
* We have two session-level configurations for parquet writes:
* `TableParquetOptions` and `ParquetOptions`.
* We have two writer-level options for writer (a.k.a. props to be handed to
a writer, per a single write action):
* `WriterProperties` and ParquetWriterOptions (which wraps
`WriterProperties`).
At first, it looks like we should be able to get the writer props from
either session-level config options. (In fact, we made exactly [this
suggestion](https://github.com/influxdata/influxdb_iox/pull/11458#discussion_r1671123365)
ourselves.) But this is misleading; as the `ParquetOptions` has an incomplete
set of all the config needed for the arrow writer (it's missing the kv_metadata
& col-specific configurations).
## What changes are included in this PR?
Instead, I've done some cleanup and added documentation to hopefully make
the relationship more clear. Additionally, there's a new
`ParquetOptions::writer_props_from_global_opts()` that does fulfill our
suggestion -- while still making the difference clear.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
A new `ParquetOptions::writer_props_from_global_opts()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
