Using a shared filesystem abstract API in Arrow Python libraries [was Re: file-system specification]

2018-07-06 Thread Wes McKinney
Thank you, Wes On Tue, May 15, 2018 at 7:47 AM, Antoine Pitrou wrote: > > Hi Martin, > > On Wed, 9 May 2018 11:28:15 -0400 > Martin Durant wrote: >> I have sketched out a possible start of a python-wide file-system >> specification >> https://github.com/martindura

Re: file-system specification

2018-05-15 Thread Antoine Pitrou
Hi Martin, On Wed, 9 May 2018 11:28:15 -0400 Martin Durant wrote: > I have sketched out a possible start of a python-wide file-system > specification > https://github.com/martindurant/filesystem_spec > > This came about from my work in some other (remote) file-systems > i

file-system specification

2018-05-09 Thread Martin Durant
I have sketched out a possible start of a python-wide file-system specification https://github.com/martindurant/filesystem_spec This came about from my work in some other (remote) file-systems implementations for python, particularly in the context of Dask. Since arrow also cares about both