Nice to hear that it worked. Updating the docs should not be necessary, we should rather see that we soon get a 0.9.0 release out (but that will also take some more weeks)
Uwe On Sun, Jan 14, 2018, at 2:42 PM, simba nyatsanga wrote: > Amazing, thanks Uwe! > > I was able to build pyarrow successfully for python 2.7 using your > workaround. I appreciate that you've got a possible solution for the too. > > Besides the PR getting reviewed by more experienced maintainers, I'm > thinking to pull your branch and try the building process from scratch. > Otherwise I was wondering if it's valuable, in the meantime, to update the > docs with your work around? > > Kind Regards > Simba > > On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn <uw...@xhochy.com> wrote: > > > Hello Simba, > > > > it looks like you are running to > > https://issues.apache.org/jira/browse/ARROW-1856. > > > > To work around this issue, please "unset PARQUET_HOME" before you call the > > setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export > > PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By > > doing this, you do the package discovery using pkg-config instead of the > > *_HOME variables. Currently this is the only path on which we can > > auto-detect the extension of the parquet shared library. > > > > Nevertheless, I will take a shot at fixing the issues as it seems that > > multiple users run into it. > > > > Uwe > > > > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote: > > > Hi Wes, > > > > > > Apologies for the ambiguity there. To clarify, I used the conda > > > instructions only to create a conda environment. So I did this > > > > > > conda create -y -q -n pyarrow-dev \ > > > python=2.7 numpy six setuptools cython pandas pytest \ > > > cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \ > > > gflags brotli jemalloc lz4-c zstd -c conda-forge > > > > > > > > > I followed the instructions closely and I've stumbled upon a different > > > error from the one I initially had encountered. Now the issue seems to be > > > that when I'm building the Arrow C++ i.e running the following steps: > > > > > > mkdir parquet-cpp/build > > > pushd parquet-cpp/build > > > > > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ > > > -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \ > > > -DPARQUET_BUILD_BENCHMARKS=off \ > > > -DPARQUET_BUILD_EXECUTABLES=off \ > > > -DPARQUET_BUILD_TESTS=off \ > > > .. > > > > > > make -j4 > > > make install > > > popd > > > > > > > > > The make install step generates *libparquet.1.3.2.dylib* as one of the > > > artefacts, as illustrated below: > > > > > > -- Install configuration: "RELEASE"-- Installing: > > > /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet- > > > cppConfig.cmake-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/ > > > cmake/parquet-cppConfigVersion.cmake-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. > > > 1.3.2.dylib-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. > > > 1.dylib-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/ > > > libparquet.dylib-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > column_reader.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > column_page.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > column_scanner.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > column_writer.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > encoding.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > exception.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > file_reader.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > file_writer.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > metadata.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > printer.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > properties.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > schema.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > statistics.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > types.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > parquet_version.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/ > > > parquet.pc-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > > io.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > > reader.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > > writer.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ > > > schema.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > arrow/reader.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > arrow/schema.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ > > > arrow/writer.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > buffer-builder.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > comparison.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > logging.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > macros.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > memory.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > stopwatch.h-- > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ > > > visibility.h > > > > > > > > > Subsequently when I want to build a standalone pyarrow wheel by running > > > this step: > > > > > > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ > > > --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel > > > > > > > > > > > > Then I get an error where one of the build steps in the *CMakelists.txt* > > > expects to find *libparquet.1.0.0.dylib*. The error is illustrated > > below: > > > > > > running build_ext-- Runnning cmake for pyarrow > > > cmake > > -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev/bin/python > > > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on > > > -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release > > > /Users/simbarashenyatsanga/Projects/personal/oss/arrow/python > > > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++ > > > INFOCompiler version: Apple LLVM version 8.0.0 > > > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 > > > Thread model: posixInstalledDir: > > /Library/Developer/CommandLineTools/usr/bin > > > > > > INFOCompiler id: Clang > > > Selected compiler clang 3.8.0svn > > > Configured for RELEASE build (set with cmake > > > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: RELEASE-- Build > > > output directory: > > > /Users/simba/Projects/personal/oss/arrow/python/build/temp.macosx-10.9- > > > x86_64-2.7/release/-- > > > Checking for module 'arrow'-- Found arrow, version 0.9.0-SNAPSHOT-- > > > Arrow ABI version: 0.0.0-- Arrow SO version: 0-- Found the Arrow core > > > library: /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib-- > > > Found the Arrow Python library: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib > > > Added shared library dependency arrow: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib > > > Added shared library dependency arrow_python: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib-- > > > Found the Parquet library: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib > > > CMake Error: File > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.0.0.dylib does > > > not exist. > > > CMake Error at CMakeLists.txt:213 (configure_file): > > > configure_file Problem configuring file > > > Call Stack (most recent call first): > > > CMakeLists.txt:296 (bundle_arrow_lib) > > > > > > > > > Added shared library dependency parquet: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib-- Checking > > > for module 'plasma'-- Found plasma, version-- Plasma ABI version: > > > 0.0.0-- Plasma SO version: 0-- Found the Plasma core library: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- Found > > > Plasma executable: > > > /Users/simba/anaconda/envs/pyarrow-dev/bin/plasma_store > > > Added shared library dependency libplasma: > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- > > > Configuring incomplete, errors occurred! > > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ > > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeOutput.log". > > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ > > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeError.log".error: > > > command 'cmake' failed with exit status 1 > > > > > > > > > Also (might be) worth noting from above is that I'm picking up *arrow > > > 0.9.0-SNAPSHOT.* > > > > > > From what I can see in the */Users/simba/anaconda/envs/pyarrow-dev/lib* > > > folder the sym link is infact pointing to *libparquet.1.3.2.dylib > > *instead > > > of the expected *libparquet.1.0.0.dylib*: > > > > > > > pwd/Users/simba/anaconda/envs/pyarrow-dev/lib> ll | grep > > "libparquet"-rwxr-xr-x 1 simba staff 1.6M Jan 11 18:45 > > libparquet.1.3.2.dylib > > > lrwxr-xr-x 1 simba staff 22B Jan 11 18:45 libparquet.1.dylib -> > > > libparquet.1.3.2.dylib-rw-r--r-- 1 simba staff 3.0M Jan 11 18:45 > > > libparquet.a > > > lrwxr-xr-x 1 simba staff 18B Jan 11 18:45 libparquet.dylib -> > > > libparquet.1.dylib > > > > > > > > > > > > Just to clarify also, I'm attempting to build the wheel from within > > > *arrow/python* folder where the *setup.py* file is. > > > > > > Thanks again for the help. > > > > > > Simba > > > > > > > > > > > > On Thu, 11 Jan 2018 at 09:09 simba nyatsanga <simnyatsa...@gmail.com> > > wrote: > > > > > > > Hi Wes, > > > > > > > > Thanks for the response. I was following the development instructions > > on > > > > Github here: > > > > > > https://github.com/apache/arrow/blob/master/python/doc/source/development.rst > > > > > > > > I took MacOS option and installed my virtual env via conda. I must've > > > > missed an instruction when trying the 2.7 install, because I was able > > to > > > > successfully install for 3.6. > > > > > > > > Although it looks like the instructions on Github are similar to the > > ones > > > > you linked, I will give it another go with the later. > > > > > > > > Kind Regards > > > > Simba > > > > > > > > On Thu, 11 Jan 2018 at 00:51 Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > > >> hi Simba, > > > >> > > > >> Are you following development instructions in > > > >> > > > >> > > http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos > > > >> or something else? > > > >> > > > >> - Wes > > > >> > > > >> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga > > > >> <simnyatsa...@gmail.com> wrote: > > > >> > Hi, > > > >> > > > > >> > I've created a python 2.7 virtualenv in my attempt to build the > > pyarrow > > > >> > project. But I'm having trouble running one of commands as > > specified in > > > >> the > > > >> > development docs on Github, specifically this command: > > > >> > > > > >> > cd arrow/python > > > >> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ > > > >> > --with-parquet --with-plasma --inplace > > > >> > > > > >> > The error output looks like this: > > > >> > > > > >> > running build_ext-- Runnning cmake for pyarrow > > > >> > cmake > > > >> > > -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python > > > >> > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on > > > >> > -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python > > > >> > INFOCompiler command: > > /Library/Developer/CommandLineTools/usr/bin/c++ > > > >> > INFOCompiler version: Apple LLVM version 8.0.0 > > > >> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 > > > >> > Thread model: posixInstalledDir: > > > >> /Library/Developer/CommandLineTools/usr/bin > > > >> > > > > >> > INFOCompiler id: Clang > > > >> > Selected compiler clang 3.8.0svn > > > >> > Configured for DEBUG build (set with cmake > > > >> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build > > > >> > output directory: > > > >> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/-- > > > >> > Checking for module 'arrow'-- No package 'arrow' found-- Found the > > > >> > Arrow core library: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib-- > > > >> > Found the Arrow Python library: > > > >> > > > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib > > > >> > Added shared library dependency arrow: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib > > > >> > Added shared library dependency arrow_python: > > > >> > > > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib-- > > > >> > Checking for module 'parquet'-- No package 'parquet' found-- Found > > > >> > the Parquet library: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib > > > >> > Added shared library dependency parquet: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib-- > > > >> > Checking for module 'plasma'-- No package 'plasma' found-- Found > > the > > > >> > Plasma core library: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- > > > >> > Found Plasma executable: > > > >> > Added shared library dependency libplasma: > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- > > > >> > Configuring done-- Generating done-- Build files have been written > > to: > > > >> > /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for > > > >> > pyarrow-- Running cmake --build for pyarrow > > > >> > makemake: *** No targets specified and no makefile found. > > Stop.error: > > > >> > command 'make' failed with exit status 2 > > > >> > > > > >> > > > > >> > It looks like there's a change dir happening at this line in the > > > >> setup.py: > > > >> > https://github.com/apache/arrow/blob/master/python/setup.py#L136 > > > >> > Which, in my case, is switching to the temp build which doesn't > > have the > > > >> > required Makefile to run the make command. > > > >> > > > > >> > I could be missing something because I was able to build the project > > > >> > successfully for python3. But I'd like to build it in python2.7 to > > > >> attempt > > > >> > a bug fix for this issue: > > > >> https://issues.apache.org/jira/browse/ARROW-1976 > > > >> > > > > >> > Thanks for help. > > > >> > > > > >> > Kind Regards > > > >> > Simba > > > >> > > > > > >
