Hello Simba,
it looks like you are running to
https://issues.apache.org/jira/browse/ARROW-1856.
To work around this issue, please "unset PARQUET_HOME" before you call the
setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export
PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By doing
this, you do the package discovery using pkg-config instead of the *_HOME
variables. Currently this is the only path on which we can auto-detect the
extension of the parquet shared library.
Nevertheless, I will take a shot at fixing the issues as it seems that multiple
users run into it.
Uwe
On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> Hi Wes,
>
> Apologies for the ambiguity there. To clarify, I used the conda
> instructions only to create a conda environment. So I did this
>
> conda create -y -q -n pyarrow-dev \
> python=2.7 numpy six setuptools cython pandas pytest \
> cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
> gflags brotli jemalloc lz4-c zstd -c conda-forge
>
>
> I followed the instructions closely and I've stumbled upon a different
> error from the one I initially had encountered. Now the issue seems to be
> that when I'm building the Arrow C++ i.e running the following steps:
>
> mkdir parquet-cpp/build
> pushd parquet-cpp/build
>
> cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> -DPARQUET_BUILD_BENCHMARKS=off \
> -DPARQUET_BUILD_EXECUTABLES=off \
> -DPARQUET_BUILD_TESTS=off \
> ..
>
> make -j4
> make install
> popd
>
>
> The make install step generates *libparquet.1.3.2.dylib* as one of the
> artefacts, as illustrated below:
>
> -- Install configuration: "RELEASE"-- Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> cppConfig.cmake--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> cmake/parquet-cppConfigVersion.cmake--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> 1.3.2.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> 1.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> libparquet.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_page.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_scanner.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> encoding.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> exception.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> file_reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> file_writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> metadata.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> printer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> properties.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> statistics.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> types.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> parquet_version.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/
> parquet.pc--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> io.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> buffer-builder.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> comparison.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> logging.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> macros.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> memory.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> stopwatch.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> visibility.h
>
>
> Subsequently when I want to build a standalone pyarrow wheel by running
> this step:
>
> python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
> --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel
>
>
>
> Then I get an error where one of the build steps in the *CMakelists.txt*
> expects to find *libparquet.1.0.0.dylib*. The error is illustrated below:
>
> running build_ext-- Runnning cmake for pyarrow
> cmake -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev/bin/python
> -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
> -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release
> /Users/simbarashenyatsanga/Projects/personal/oss/arrow/python
> INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
> INFOCompiler version: Apple LLVM version 8.0.0
> (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
> Thread model: posixInstalledDir: /Library/Developer/CommandLineTools/usr/bin
>
> INFOCompiler id: Clang
> Selected compiler clang 3.8.0svn
> Configured for RELEASE build (set with cmake
> -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: RELEASE-- Build
> output directory:
> /Users/simba/Projects/personal/oss/arrow/python/build/temp.macosx-10.9-
> x86_64-2.7/release/--
> Checking for module 'arrow'-- Found arrow, version 0.9.0-SNAPSHOT--
> Arrow ABI version: 0.0.0-- Arrow SO version: 0-- Found the Arrow core
> library: /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib--
> Found the Arrow Python library:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib
> Added shared library dependency arrow:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib
> Added shared library dependency arrow_python:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib--
> Found the Parquet library:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib
> CMake Error: File
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.0.0.dylib does
> not exist.
> CMake Error at CMakeLists.txt:213 (configure_file):
> configure_file Problem configuring file
> Call Stack (most recent call first):
> CMakeLists.txt:296 (bundle_arrow_lib)
>
>
> Added shared library dependency parquet:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib-- Checking
> for module 'plasma'-- Found plasma, version-- Plasma ABI version:
> 0.0.0-- Plasma SO version: 0-- Found the Plasma core library:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- Found
> Plasma executable:
> /Users/simba/anaconda/envs/pyarrow-dev/bin/plasma_store
> Added shared library dependency libplasma:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib--
> Configuring incomplete, errors occurred!
> See also "/Users/simba/Projects/personal/oss/arrow/python/build/
> temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeOutput.log".
> See also "/Users/simba/Projects/personal/oss/arrow/python/build/
> temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeError.log".error:
> command 'cmake' failed with exit status 1
>
>
> Also (might be) worth noting from above is that I'm picking up *arrow
> 0.9.0-SNAPSHOT.*
>
> From what I can see in the */Users/simba/anaconda/envs/pyarrow-dev/lib*
> folder the sym link is infact pointing to *libparquet.1.3.2.dylib *instead
> of the expected *libparquet.1.0.0.dylib*:
>
> > pwd/Users/simba/anaconda/envs/pyarrow-dev/lib> ll | grep
> > "libparquet"-rwxr-xr-x 1 simba staff 1.6M Jan 11 18:45
> > libparquet.1.3.2.dylib
> lrwxr-xr-x 1 simba staff 22B Jan 11 18:45 libparquet.1.dylib ->
> libparquet.1.3.2.dylib-rw-r--r-- 1 simba staff 3.0M Jan 11 18:45
> libparquet.a
> lrwxr-xr-x 1 simba staff 18B Jan 11 18:45 libparquet.dylib ->
> libparquet.1.dylib
>
>
>
> Just to clarify also, I'm attempting to build the wheel from within
> *arrow/python* folder where the *setup.py* file is.
>
> Thanks again for the help.
>
> Simba
>
>
>
> On Thu, 11 Jan 2018 at 09:09 simba nyatsanga <simnyatsa...@gmail.com> wrote:
>
> > Hi Wes,
> >
> > Thanks for the response. I was following the development instructions on
> > Github here:
> > https://github.com/apache/arrow/blob/master/python/doc/source/development.rst
> >
> > I took MacOS option and installed my virtual env via conda. I must've
> > missed an instruction when trying the 2.7 install, because I was able to
> > successfully install for 3.6.
> >
> > Although it looks like the instructions on Github are similar to the ones
> > you linked, I will give it another go with the later.
> >
> > Kind Regards
> > Simba
> >
> > On Thu, 11 Jan 2018 at 00:51 Wes McKinney <wesmck...@gmail.com> wrote:
> >
> >> hi Simba,
> >>
> >> Are you following development instructions in
> >>
> >> http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos
> >> or something else?
> >>
> >> - Wes
> >>
> >> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga
> >> <simnyatsa...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I've created a python 2.7 virtualenv in my attempt to build the pyarrow
> >> > project. But I'm having trouble running one of commands as specified in
> >> the
> >> > development docs on Github, specifically this command:
> >> >
> >> > cd arrow/python
> >> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
> >> > --with-parquet --with-plasma --inplace
> >> >
> >> > The error output looks like this:
> >> >
> >> > running build_ext-- Runnning cmake for pyarrow
> >> > cmake
> >> -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python
> >> > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
> >> > -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python
> >> > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
> >> > INFOCompiler version: Apple LLVM version 8.0.0
> >> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
> >> > Thread model: posixInstalledDir:
> >> /Library/Developer/CommandLineTools/usr/bin
> >> >
> >> > INFOCompiler id: Clang
> >> > Selected compiler clang 3.8.0svn
> >> > Configured for DEBUG build (set with cmake
> >> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build
> >> > output directory:
> >> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/--
> >> > Checking for module 'arrow'-- No package 'arrow' found-- Found the
> >> > Arrow core library:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib--
> >> > Found the Arrow Python library:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib
> >> > Added shared library dependency arrow:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib
> >> > Added shared library dependency arrow_python:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib--
> >> > Checking for module 'parquet'-- No package 'parquet' found-- Found
> >> > the Parquet library:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib
> >> > Added shared library dependency parquet:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib--
> >> > Checking for module 'plasma'-- No package 'plasma' found-- Found the
> >> > Plasma core library:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> >> > Found Plasma executable:
> >> > Added shared library dependency libplasma:
> >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> >> > Configuring done-- Generating done-- Build files have been written to:
> >> > /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for
> >> > pyarrow-- Running cmake --build for pyarrow
> >> > makemake: *** No targets specified and no makefile found. Stop.error:
> >> > command 'make' failed with exit status 2
> >> >
> >> >
> >> > It looks like there's a change dir happening at this line in the
> >> setup.py:
> >> > https://github.com/apache/arrow/blob/master/python/setup.py#L136
> >> > Which, in my case, is switching to the temp build which doesn't have the
> >> > required Makefile to run the make command.
> >> >
> >> > I could be missing something because I was able to build the project
> >> > successfully for python3. But I'd like to build it in python2.7 to
> >> attempt
> >> > a bug fix for this issue:
> >> https://issues.apache.org/jira/browse/ARROW-1976
> >> >
> >> > Thanks for help.
> >> >
> >> > Kind Regards
> >> > Simba
> >>
> >
