Re: Arrow as a common open standard for machine learning data

2020-07-02 Thread Nicholas Poorman
Joaquin, > Do you know whether there any activity on supporting partial read/writes in arrow or fastparquet? I’m not entirely sure about the status of partial read/writes in Arrow’s Parquet implementation but https://github.com/xitongsys/parquet-go for example has this capability. > Even then, t

Re: Arrow as a common open standard for machine learning data

2020-06-30 Thread Nicholas Poorman
Joaquin, After reading your proposal I think there may be some things you may want to consider. It sounds like you are trying to come up with a one size fits all solution but it may be better to define your requirements based on your needs and environment. For starters, where do you plan to stor