You can take a look at PartitionReaderFactory. It's created at the driver side, serialized and sent to the executor side. For the write side, there is a similar channel: DataWriterFactory
On Wed, May 19, 2021 at 4:37 AM Andrew Melo <andrew.m...@gmail.com> wrote: > Hello, > > When implementing a DSv2 datasource, where is an appropriate place to > store/transmit secrets from the driver to the executors? Is there > built-in spark functionality for that, or is my best bet to stash it > as a member variable in one of the classes that gets sent to the > executors? > > Thanks! > Andrew > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
