Re: arrow:io:S3ReadableFile

2018-07-26 Thread Renato Marroquín Mogrovejo
Thanks Wes and Uwe for your suggestions! 2018-07-25 16:58 GMT+02:00 Uwe L. Korn : > Hello Renato, > > I don't think that the hdfs client will give you the necessary interface > to use Hadoop's S3 implementation. If it does, this might be a simple way > to support more filesystems just by using th

Re: arrow:io:S3ReadableFile

2018-07-25 Thread Uwe L. Korn
Hello Renato, I don't think that the hdfs client will give you the necessary interface to use Hadoop's S3 implementation. If it does, this might be a simple way to support more filesystems just by using their Hadoop implementation. In general, it would be preferred to have native (C/C++) implem

Re: arrow:io:S3ReadableFile

2018-07-25 Thread Wes McKinney
hey Renato, I would recommend following whatever TensorFlow has done. We can even reuse their code (Apache 2.0): https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/platform/s3 - Wes On Wed, Jul 25, 2018 at 6:49 AM, Renato Marroquín Mogrovejo wrote: > Hi Arrow experts, > > I a