Amazing, thanks Uwe! I was able to build pyarrow successfully for python 2.7 using your workaround. I appreciate that you've got a possible solution for the too.
Besides the PR getting reviewed by more experienced maintainers, I'm thinking to pull your branch and try the building process from scratch. Otherwise I was wondering if it's valuable, in the meantime, to update the docs with your work around? Kind Regards Simba On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn <uw...@xhochy.com> wrote: > Hello Simba, > > it looks like you are running to > https://issues.apache.org/jira/browse/ARROW-1856. > > To work around this issue, please "unset PARQUET_HOME" before you call the > setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export > PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By > doing this, you do the package discovery using pkg-config instead of the > *_HOME variables. Currently this is the only path on which we can > auto-detect the extension of the parquet shared library. > > Nevertheless, I will take a shot at fixing the issues as it seems that > multiple users run into it. > > Uwe > > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote: > > Hi Wes, > > > > Apologies for the ambiguity there. To clarify, I used the conda > > instructions only to create a conda environment. So I did this > > > > conda create -y -q -n pyarrow-dev \ > > python=2.7 numpy six setuptools cython pandas pytest \ > > cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \ > > gflags brotli jemalloc lz4-c zstd -c conda-forge > > > > > > I followed the instructions closely and I've stumbled upon a different > > error from the one I initially had encountered. Now the issue seems to be > > that when I'm building the Arrow C++ i.e running the following steps: > > > > mkdir parquet-cpp/build > > pushd parquet-cpp/build > > > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ > > -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \ > > -DPARQUET_BUILD_BENCHMARKS=off \ > > -DPARQUET_BUILD_EXECUTABLES=off \ > > -DPARQUET_BUILD_TESTS=off \ > > .. > > > > make -j4 > > make install > > popd > > > > > > The make install step generates *libparquet.1.3.2.dylib* as one of the > > artefacts, as illustrated below: > > > > -- Install configuration: "RELEASE"-- Installing: > > /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet- > > cppConfig.cmake-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/ > > cmake/parquet-cppConfigVersion.cmake-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. > > 1.3.2.dylib-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. > > 1.dylib-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/ > > libparquet.dylib-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > column_reader.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > column_page.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > column_scanner.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > column_writer.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > encoding.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > exception.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > file_reader.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > file_writer.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > metadata.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > printer.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > properties.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > schema.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > statistics.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > types.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > parquet_version.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/ > > parquet.pc-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > io.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > reader.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > writer.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > schema.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > arrow/reader.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > arrow/schema.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > arrow/writer.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > buffer-builder.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > comparison.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > logging.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > macros.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > memory.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > stopwatch.h-- > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > visibility.h > > > > > > Subsequently when I want to build a standalone pyarrow wheel by running > > this step: > > > > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ > > --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel > > > > > > > > Then I get an error where one of the build steps in the *CMakelists.txt* > > expects to find *libparquet.1.0.0.dylib*. The error is illustrated > below: > > > > running build_ext-- Runnning cmake for pyarrow > > cmake > -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev/bin/python > > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on > > -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release > > /Users/simbarashenyatsanga/Projects/personal/oss/arrow/python > > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++ > > INFOCompiler version: Apple LLVM version 8.0.0 > > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 > > Thread model: posixInstalledDir: > /Library/Developer/CommandLineTools/usr/bin > > > > INFOCompiler id: Clang > > Selected compiler clang 3.8.0svn > > Configured for RELEASE build (set with cmake > > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: RELEASE-- Build > > output directory: > > /Users/simba/Projects/personal/oss/arrow/python/build/temp.macosx-10.9- > > x86_64-2.7/release/-- > > Checking for module 'arrow'-- Found arrow, version 0.9.0-SNAPSHOT-- > > Arrow ABI version: 0.0.0-- Arrow SO version: 0-- Found the Arrow core > > library: /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib-- > > Found the Arrow Python library: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib > > Added shared library dependency arrow: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib > > Added shared library dependency arrow_python: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib-- > > Found the Parquet library: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib > > CMake Error: File > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.0.0.dylib does > > not exist. > > CMake Error at CMakeLists.txt:213 (configure_file): > > configure_file Problem configuring file > > Call Stack (most recent call first): > > CMakeLists.txt:296 (bundle_arrow_lib) > > > > > > Added shared library dependency parquet: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib-- Checking > > for module 'plasma'-- Found plasma, version-- Plasma ABI version: > > 0.0.0-- Plasma SO version: 0-- Found the Plasma core library: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- Found > > Plasma executable: > > /Users/simba/anaconda/envs/pyarrow-dev/bin/plasma_store > > Added shared library dependency libplasma: > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- > > Configuring incomplete, errors occurred! > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeOutput.log". > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeError.log".error: > > command 'cmake' failed with exit status 1 > > > > > > Also (might be) worth noting from above is that I'm picking up *arrow > > 0.9.0-SNAPSHOT.* > > > > From what I can see in the */Users/simba/anaconda/envs/pyarrow-dev/lib* > > folder the sym link is infact pointing to *libparquet.1.3.2.dylib > *instead > > of the expected *libparquet.1.0.0.dylib*: > > > > > pwd/Users/simba/anaconda/envs/pyarrow-dev/lib> ll | grep > "libparquet"-rwxr-xr-x 1 simba staff 1.6M Jan 11 18:45 > libparquet.1.3.2.dylib > > lrwxr-xr-x 1 simba staff 22B Jan 11 18:45 libparquet.1.dylib -> > > libparquet.1.3.2.dylib-rw-r--r-- 1 simba staff 3.0M Jan 11 18:45 > > libparquet.a > > lrwxr-xr-x 1 simba staff 18B Jan 11 18:45 libparquet.dylib -> > > libparquet.1.dylib > > > > > > > > Just to clarify also, I'm attempting to build the wheel from within > > *arrow/python* folder where the *setup.py* file is. > > > > Thanks again for the help. > > > > Simba > > > > > > > > On Thu, 11 Jan 2018 at 09:09 simba nyatsanga <simnyatsa...@gmail.com> > wrote: > > > > > Hi Wes, > > > > > > Thanks for the response. I was following the development instructions > on > > > Github here: > > > > https://github.com/apache/arrow/blob/master/python/doc/source/development.rst > > > > > > I took MacOS option and installed my virtual env via conda. I must've > > > missed an instruction when trying the 2.7 install, because I was able > to > > > successfully install for 3.6. > > > > > > Although it looks like the instructions on Github are similar to the > ones > > > you linked, I will give it another go with the later. > > > > > > Kind Regards > > > Simba > > > > > > On Thu, 11 Jan 2018 at 00:51 Wes McKinney <wesmck...@gmail.com> wrote: > > > > > >> hi Simba, > > >> > > >> Are you following development instructions in > > >> > > >> > http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos > > >> or something else? > > >> > > >> - Wes > > >> > > >> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga > > >> <simnyatsa...@gmail.com> wrote: > > >> > Hi, > > >> > > > >> > I've created a python 2.7 virtualenv in my attempt to build the > pyarrow > > >> > project. But I'm having trouble running one of commands as > specified in > > >> the > > >> > development docs on Github, specifically this command: > > >> > > > >> > cd arrow/python > > >> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ > > >> > --with-parquet --with-plasma --inplace > > >> > > > >> > The error output looks like this: > > >> > > > >> > running build_ext-- Runnning cmake for pyarrow > > >> > cmake > > >> > -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python > > >> > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on > > >> > -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python > > >> > INFOCompiler command: > /Library/Developer/CommandLineTools/usr/bin/c++ > > >> > INFOCompiler version: Apple LLVM version 8.0.0 > > >> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 > > >> > Thread model: posixInstalledDir: > > >> /Library/Developer/CommandLineTools/usr/bin > > >> > > > >> > INFOCompiler id: Clang > > >> > Selected compiler clang 3.8.0svn > > >> > Configured for DEBUG build (set with cmake > > >> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build > > >> > output directory: > > >> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/-- > > >> > Checking for module 'arrow'-- No package 'arrow' found-- Found the > > >> > Arrow core library: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib-- > > >> > Found the Arrow Python library: > > >> > > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib > > >> > Added shared library dependency arrow: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib > > >> > Added shared library dependency arrow_python: > > >> > > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib-- > > >> > Checking for module 'parquet'-- No package 'parquet' found-- Found > > >> > the Parquet library: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib > > >> > Added shared library dependency parquet: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib-- > > >> > Checking for module 'plasma'-- No package 'plasma' found-- Found > the > > >> > Plasma core library: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- > > >> > Found Plasma executable: > > >> > Added shared library dependency libplasma: > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- > > >> > Configuring done-- Generating done-- Build files have been written > to: > > >> > /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for > > >> > pyarrow-- Running cmake --build for pyarrow > > >> > makemake: *** No targets specified and no makefile found. > Stop.error: > > >> > command 'make' failed with exit status 2 > > >> > > > >> > > > >> > It looks like there's a change dir happening at this line in the > > >> setup.py: > > >> > https://github.com/apache/arrow/blob/master/python/setup.py#L136 > > >> > Which, in my case, is switching to the temp build which doesn't > have the > > >> > required Makefile to run the make command. > > >> > > > >> > I could be missing something because I was able to build the project > > >> > successfully for python3. But I'd like to build it in python2.7 to > > >> attempt > > >> > a bug fix for this issue: > > >> https://issues.apache.org/jira/browse/ARROW-1976 > > >> > > > >> > Thanks for help. > > >> > > > >> > Kind Regards > > >> > Simba > > >> > > > >
