hi Peng, Here is a minimal reproduction of the issue you're having:
In [38]: arr = np.empty(2, dtype=object) In [39]: arr[0] = np.array([1, 2]) In [40]: arr[1] = np.array([2, 3]) In [41]: arr2 = np.empty(2, dtype=object) In [42]: arr2[0] = arr In [43]: arr2[1] = arr In [45]: pa.array(arr2) --------------------------------------------------------------------------- ArrowTypeError Traceback (most recent call last) <ipython-input-45-4dfe5219bac5> in <module> ----> 1 pa.array(arr2) ~/miniconda/envs/pyarrow-0.12.0-3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array() ~/miniconda/envs/pyarrow-0.12.0-3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_array() ~/miniconda/envs/pyarrow-0.12.0-3.7/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() ArrowTypeError: only size-1 arrays can be converted to Python scalars I suspect this is fixable but it will have to be done in https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/python_to_arrow.cc. Without digging in to the details I'm not sure where exactly the code is failing I'll post this information on the JIRA also. - Wes On Fri, Feb 22, 2019 at 11:17 AM peng yu <yupb...@gmail.com> wrote: > > Hey channel, > > I'm trying to fix issue 4350 > <https://issues.apache.org/jira/browse/ARROW-4350>, which is a result of > arrow.Table<-> pandas.df conversion is not symmetric > > basically we decide to use numpy array as the basis of list when converting > from arrow table to pandas. which makes me wondering why? since pandas df > doesn't have a very good multi-dimension array schema. and numpy array is > very strict with the compact memory layout. So it is very hard to support > any more than 1-D array/list data inside of pandas df cell using numpy 1D > array of 1D array. And it actually caused a lot of problem for us to use > the result of arrow in python land. > > Would it be possible to just use pure python list if the data is more than > 1D ? > > I can't think of any easy fix for that issue, if any of you have any > suggestions, please let me know :) > > THanks!
