Potentially useful further context:
the current C++ FileSystem documentation at
https://arrow.apache.org/docs/cpp/io.html#filesystems
On Wed, Apr 9, 2025 at 9:35 AM Benjamin Kietzman
wrote:
> I have been working on modularizing the C++ library by extending
> FileSystem construction from URIs. I
I have been working on modularizing the C++ library by extending FileSystem
construction from URIs. I recently merged a PR which prompted some
discussion [1] of how the library should handle secrets.
Some FileSystems cannot be constructed without one or more secrets. For
example, an S3FileSystem m
I'm not all that familiar with the C++ filesystem abstraction, but for
ObjectStore, the closest equivalent abstraction in the Rust ecosystem,
we follow what fsspec [1] and Hadoop [2] do and allow providing a set of
key-value string pairs along with the URI [3]. This provides a great
deal of fle
Hi Ben, would you be able to elaborate on this part:
> Since URIs must be complete specifications of a filesystem, this necessitates
> inclusion of the secrets required by S3 in the URI. Since anyone with a URI
> has access to the filesystem to which it refers, these filesystem URIs are
> trans
I can't speak to why Hadoop or fsspec are designed that way, but the
following come to mind:
- Systems typically draw a separation between system config, such as
credentials, and the user-supplied URI, which may be provided as part of
a SQL string, for example
- It avoids needing to define a U
Thanks Raphael,
Do you have a reference which explains the rationale for that separation?
It's not obvious to me what the priorities are.
I can guess that a URI without secrets might be shared between multiple
users,
and their individual tokens etc inserted to grant distinct access. However
for t
Our next biweekly Arrow community meeting is today, Wednesday 9 April 17:00
UTC / 12:00 EDT / 9:00 PDT.
Zoom meeting URL:
https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09
Meeting ID: 876 4903 3008
Passcode: 958092
Meeting notes will be captured in this Google Doc:
https://docs.g
