Re: Petastorm: PyArrow based library for Tensorflow, PyTorch and others...

2018-10-07 Thread Yevgeni Litvin
port of Tensors there could help. Regards, - Yevgeni > -- Forwarded message -- > From: "Uwe L. Korn" > To: dev@arrow.apache.org > Cc: > Bcc: > Date: Fri, 05 Oct 2018 17:38:13 +0200 > Subject: Re: Petastorm: PyArrow based library for Tensorflow, PyT

Re: Petastorm: PyArrow based library for Tensorflow, PyTorch and others...

2018-10-05 Thread Uwe L. Korn
Hello Yevgeni, this looks interesting. Can you make a PR to https://github.com/apache/arrow so that Petastorm is listed on https://arrow.apache.org/powered_by/ ? I browsed a bit through your code. As far as I can see your approach is store to have a set of Parquet files in a directory with a

Petastorm: PyArrow based library for Tensorflow, PyTorch and others...

2018-09-26 Thread Yevgeni Litvin
Hi, My name is Yevgeni Litvin. I am working on ML infra with a small team within Uber ATG. Our team has recently open sourced Petastorm library. It heavily relies on Apache Arrow so I wanted to share it with the community. The goal of the project is to provide a convenient way for deep learning c