Re: Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-10-03 Thread Bipin Mathew
case, but I thought this may start to give insight on if there is an issue > and if so what it maybe. > > Regards, > > Bipin > > > On Fri, Sep 28, 2018 at 5:41 PM Wes McKinney wrote: > >> hi Abdul -- Plasma vs. a memory map on /dev/shm should have the same >>

Re: Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-09-28 Thread Bipin Mathew
2018 at 5:41 PM Wes McKinney wrote: > hi Abdul -- Plasma vs. a memory map on /dev/shm should have the same > semantics re: memory copying, so I don't believe using Plasma will > change the outcome > > - Wes > On Fri, Sep 28, 2018 at 5:38 PM Abdul Rahman > wrote: &g

Re: Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-09-28 Thread Wes McKinney
rying to do ? > > https://arrow.apache.org/docs/python/plasma.html#using-arrow-and-pandas-with-plasma > > > > From: Bipin Mathew > Sent: Friday, September 28, 2018 2:28:54 PM > To: dev@arrow.apache.org > Subject: Help with zero-copy conversio

Re: Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-09-28 Thread Wes McKinney
hi Bipin, There are narrow circumstances where zero-copy pandas deserialization is possible. Firstly, I noted that we are short of documentation for Table.to_pandas, so I opened https://issues.apache.org/jira/browse/ARROW-3356 It's possible there's a bug when zero_copy_only=True -- it is suppose

Re: Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-09-28 Thread Abdul Rahman
: Help with zero-copy conversion of pyarrow table to pandas dataframe. Hello Everyone, I am just getting my feet wet with apache arrow and I am running into a bug or, more likely, simply misunderstanding the pyarrow api. I wrote out a four column, million row apache arrow table to shared memory

Help with zero-copy conversion of pyarrow table to pandas dataframe.

2018-09-28 Thread Bipin Mathew
Hello Everyone, I am just getting my feet wet with apache arrow and I am running into a bug or, more likely, simply misunderstanding the pyarrow api. I wrote out a four column, million row apache arrow table to shared memory and I am attempting to read it into a python dataframe. It is advert