Re: Questions about S3 options

2020-08-26 Thread Joris Van den Bossche
Hi Weston, Sorry for the late reply. For using S3 in pyarrow, there are indeed 2 options: using the implementation provided by arrow (`pyarrow.fs.S3FileSystem`) or using s3fs which gets wrapped by pyarrow. Note that the wrapper is not actually DaskFileSystem: for the legacy filesystems we use s3fs

Questions about S3 options

2020-08-19 Thread Weston Pace
To use S3 it appears I can either use `pyarrow.fs.S3FileSystem` or I can use s3fs and it gets wrapped (I think) with `pyarrow.fs.DaskFileSystem`. However, I don't see any documentation for `pyarrow.fs.DaskFileSystem`. Is this option supported going forwards? I'm currently configuring an s3fs ins