Hi all,
I'll prepare a draft PR adding KV pairs to the interface.
Thanks,
Ben
On Wed, Apr 23, 2025 at 8:09 AM Antoine Pitrou wrote:
>
> Hi Ben and all,
>
> Sorry for chiming in lately. I do find the URI-and-kv-pairs interface
> attractive.
>
> That said, some filesystem options can't reasonabl
Hi Ben and all,
Sorry for chiming in lately. I do find the URI-and-kv-pairs interface
attractive.
That said, some filesystem options can't reasonably be expressed as
strings. For example, `S3Options` has a `std::shared_ptrKeyValueMetadata> default_metadata` and a
`std::shared_ptr`.
So, p
Got it, thanks. This reminds me of some of the confusion we saw in the
R bindings [1] with constructing filesystems from options versus a URI
and how you seem to have to pick one path or the other. I think the
type of approach ObjectStore uses where information gets merged is
more flexible and more
Hi Bryce,
I meant to say that since the C++ interface for constructing filesystems is
[1]
Result> FileSystemFromUri(const std::string
&uri);
it follows that the only argument (the uri) must contain all the
information required.
Certainly alternate interfaces such as the uri-and-kv-pairs whi
Potentially useful further context:
the current C++ FileSystem documentation at
https://arrow.apache.org/docs/cpp/io.html#filesystems
On Wed, Apr 9, 2025 at 9:35 AM Benjamin Kietzman
wrote:
> I have been working on modularizing the C++ library by extending
> FileSystem construction from URIs. I
Hi Ben, would you be able to elaborate on this part:
> Since URIs must be complete specifications of a filesystem, this necessitates
> inclusion of the secrets required by S3 in the URI. Since anyone with a URI
> has access to the filesystem to which it refers, these filesystem URIs are
> trans
I can't speak to why Hadoop or fsspec are designed that way, but the
following come to mind:
- Systems typically draw a separation between system config, such as
credentials, and the user-supplied URI, which may be provided as part of
a SQL string, for example
- It avoids needing to define a U
Thanks Raphael,
Do you have a reference which explains the rationale for that separation?
It's not obvious to me what the priorities are.
I can guess that a URI without secrets might be shared between multiple
users,
and their individual tokens etc inserted to grant distinct access. However
for t
I'm not all that familiar with the C++ filesystem abstraction, but for
ObjectStore, the closest equivalent abstraction in the Rust ecosystem,
we follow what fsspec [1] and Hadoop [2] do and allow providing a set of
key-value string pairs along with the URI [3]. This provides a great
deal of fle
I have been working on modularizing the C++ library by extending FileSystem
construction from URIs. I recently merged a PR which prompted some
discussion [1] of how the library should handle secrets.
Some FileSystems cannot be constructed without one or more secrets. For
example, an S3FileSystem m
10 matches
Mail list logo
