hi Simba -- the PyPI / pip wheels will only be updated when there is a new release. We'll either make a 0.8.1 release or 0.9.0 sometime in February depending on how development is progressing.
- Wes On Sun, Jan 14, 2018 at 9:19 AM, simba nyatsanga <simnyatsa...@gmail.com> wrote: > Thanks a lot. I see that there's a PR that's been opened to resolve the > encoding issue - https://github.com/apache/arrow/pull/1476 > > Do you think this PR (if merged ) will also roll out as part of version > 0.9.0, or I'll be able to pip install with the merge commit as soon as it's > merged? > > Kind Regards > > On Sun, 14 Jan 2018 at 15:50 Uwe L. Korn <uw...@xhochy.com> wrote: > >> Nice to hear that it worked. >> >> Updating the docs should not be necessary, we should rather see that we >> soon get a 0.9.0 release out (but that will also take some more weeks) >> >> Uwe >> >> On Sun, Jan 14, 2018, at 2:42 PM, simba nyatsanga wrote: >> > Amazing, thanks Uwe! >> > >> > I was able to build pyarrow successfully for python 2.7 using your >> > workaround. I appreciate that you've got a possible solution for the too. >> > >> > Besides the PR getting reviewed by more experienced maintainers, I'm >> > thinking to pull your branch and try the building process from scratch. >> > Otherwise I was wondering if it's valuable, in the meantime, to update >> the >> > docs with your work around? >> > >> > Kind Regards >> > Simba >> > >> > On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn <uw...@xhochy.com> wrote: >> > >> > > Hello Simba, >> > > >> > > it looks like you are running to >> > > https://issues.apache.org/jira/browse/ARROW-1856. >> > > >> > > To work around this issue, please "unset PARQUET_HOME" before you call >> the >> > > setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export >> > > PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". >> By >> > > doing this, you do the package discovery using pkg-config instead of >> the >> > > *_HOME variables. Currently this is the only path on which we can >> > > auto-detect the extension of the parquet shared library. >> > > >> > > Nevertheless, I will take a shot at fixing the issues as it seems that >> > > multiple users run into it. >> > > >> > > Uwe >> > > >> > > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote: >> > > > Hi Wes, >> > > > >> > > > Apologies for the ambiguity there. To clarify, I used the conda >> > > > instructions only to create a conda environment. So I did this >> > > > >> > > > conda create -y -q -n pyarrow-dev \ >> > > > python=2.7 numpy six setuptools cython pandas pytest \ >> > > > cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \ >> > > > gflags brotli jemalloc lz4-c zstd -c conda-forge >> > > > >> > > > >> > > > I followed the instructions closely and I've stumbled upon a >> different >> > > > error from the one I initially had encountered. Now the issue seems >> to be >> > > > that when I'm building the Arrow C++ i.e running the following steps: >> > > > >> > > > mkdir parquet-cpp/build >> > > > pushd parquet-cpp/build >> > > > >> > > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \ >> > > > -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \ >> > > > -DPARQUET_BUILD_BENCHMARKS=off \ >> > > > -DPARQUET_BUILD_EXECUTABLES=off \ >> > > > -DPARQUET_BUILD_TESTS=off \ >> > > > .. >> > > > >> > > > make -j4 >> > > > make install >> > > > popd >> > > > >> > > > >> > > > The make install step generates *libparquet.1.3.2.dylib* as one of >> the >> > > > artefacts, as illustrated below: >> > > > >> > > > -- Install configuration: "RELEASE"-- Installing: >> > > > >> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet- >> > > > cppConfig.cmake-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/ >> > > > cmake/parquet-cppConfigVersion.cmake-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. >> > > > 1.3.2.dylib-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet. >> > > > 1.dylib-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/ >> > > > libparquet.dylib-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > column_reader.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > column_page.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > column_scanner.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > column_writer.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > encoding.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > exception.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > file_reader.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > file_writer.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > metadata.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > printer.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > properties.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > schema.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > statistics.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > types.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > parquet_version.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/ >> > > > parquet.pc-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ >> > > > io.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ >> > > > reader.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ >> > > > writer.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/ >> > > > schema.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > arrow/reader.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > arrow/schema.h-- >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/ >> > > > arrow/writer.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > buffer-builder.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > comparison.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > logging.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > macros.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > memory.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > stopwatch.h-- >> > > > Installing: >> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/ >> > > > visibility.h >> > > > >> > > > >> > > > Subsequently when I want to build a standalone pyarrow wheel by >> running >> > > > this step: >> > > > >> > > > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ >> > > > --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel >> > > > >> > > > >> > > > >> > > > Then I get an error where one of the build steps in the >> *CMakelists.txt* >> > > > expects to find *libparquet.1.0.0.dylib*. The error is illustrated >> > > below: >> > > > >> > > > running build_ext-- Runnning cmake for pyarrow >> > > > cmake >> > > -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev/bin/python >> > > > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on >> > > > -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release >> > > > /Users/simbarashenyatsanga/Projects/personal/oss/arrow/python >> > > > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++ >> > > > INFOCompiler version: Apple LLVM version 8.0.0 >> > > > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 >> > > > Thread model: posixInstalledDir: >> > > /Library/Developer/CommandLineTools/usr/bin >> > > > >> > > > INFOCompiler id: Clang >> > > > Selected compiler clang 3.8.0svn >> > > > Configured for RELEASE build (set with cmake >> > > > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: RELEASE-- Build >> > > > output directory: >> > > > >> /Users/simba/Projects/personal/oss/arrow/python/build/temp.macosx-10.9- >> > > > x86_64-2.7/release/-- >> > > > Checking for module 'arrow'-- Found arrow, version 0.9.0-SNAPSHOT-- >> > > > Arrow ABI version: 0.0.0-- Arrow SO version: 0-- Found the Arrow core >> > > > library: /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib-- >> > > > Found the Arrow Python library: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib >> > > > Added shared library dependency arrow: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow.dylib >> > > > Added shared library dependency arrow_python: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libarrow_python.dylib-- >> > > > Found the Parquet library: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib >> > > > CMake Error: File >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.0.0.dylib >> does >> > > > not exist. >> > > > CMake Error at CMakeLists.txt:213 (configure_file): >> > > > configure_file Problem configuring file >> > > > Call Stack (most recent call first): >> > > > CMakeLists.txt:296 (bundle_arrow_lib) >> > > > >> > > > >> > > > Added shared library dependency parquet: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib-- >> Checking >> > > > for module 'plasma'-- Found plasma, version-- Plasma ABI version: >> > > > 0.0.0-- Plasma SO version: 0-- Found the Plasma core library: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- Found >> > > > Plasma executable: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/bin/plasma_store >> > > > Added shared library dependency libplasma: >> > > > /Users/simba/anaconda/envs/pyarrow-dev/lib/libplasma.dylib-- >> > > > Configuring incomplete, errors occurred! >> > > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ >> > > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeOutput.log". >> > > > See also "/Users/simba/Projects/personal/oss/arrow/python/build/ >> > > > temp.macosx-10.9-x86_64-2.7/CMakeFiles/CMakeError.log".error: >> > > > command 'cmake' failed with exit status 1 >> > > > >> > > > >> > > > Also (might be) worth noting from above is that I'm picking up *arrow >> > > > 0.9.0-SNAPSHOT.* >> > > > >> > > > From what I can see in the >> */Users/simba/anaconda/envs/pyarrow-dev/lib* >> > > > folder the sym link is infact pointing to *libparquet.1.3.2.dylib >> > > *instead >> > > > of the expected *libparquet.1.0.0.dylib*: >> > > > >> > > > > pwd/Users/simba/anaconda/envs/pyarrow-dev/lib> ll | grep >> > > "libparquet"-rwxr-xr-x 1 simba staff 1.6M Jan 11 18:45 >> > > libparquet.1.3.2.dylib >> > > > lrwxr-xr-x 1 simba staff 22B Jan 11 18:45 libparquet.1.dylib >> -> >> > > > libparquet.1.3.2.dylib-rw-r--r-- 1 simba staff 3.0M Jan 11 >> 18:45 >> > > > libparquet.a >> > > > lrwxr-xr-x 1 simba staff 18B Jan 11 18:45 libparquet.dylib -> >> > > > libparquet.1.dylib >> > > > >> > > > >> > > > >> > > > Just to clarify also, I'm attempting to build the wheel from within >> > > > *arrow/python* folder where the *setup.py* file is. >> > > > >> > > > Thanks again for the help. >> > > > >> > > > Simba >> > > > >> > > > >> > > > >> > > > On Thu, 11 Jan 2018 at 09:09 simba nyatsanga <simnyatsa...@gmail.com >> > >> > > wrote: >> > > > >> > > > > Hi Wes, >> > > > > >> > > > > Thanks for the response. I was following the development >> instructions >> > > on >> > > > > Github here: >> > > > > >> > > >> https://github.com/apache/arrow/blob/master/python/doc/source/development.rst >> > > > > >> > > > > I took MacOS option and installed my virtual env via conda. I >> must've >> > > > > missed an instruction when trying the 2.7 install, because I was >> able >> > > to >> > > > > successfully install for 3.6. >> > > > > >> > > > > Although it looks like the instructions on Github are similar to >> the >> > > ones >> > > > > you linked, I will give it another go with the later. >> > > > > >> > > > > Kind Regards >> > > > > Simba >> > > > > >> > > > > On Thu, 11 Jan 2018 at 00:51 Wes McKinney <wesmck...@gmail.com> >> wrote: >> > > > > >> > > > >> hi Simba, >> > > > >> >> > > > >> Are you following development instructions in >> > > > >> >> > > > >> >> > > >> http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos >> > > > >> or something else? >> > > > >> >> > > > >> - Wes >> > > > >> >> > > > >> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga >> > > > >> <simnyatsa...@gmail.com> wrote: >> > > > >> > Hi, >> > > > >> > >> > > > >> > I've created a python 2.7 virtualenv in my attempt to build the >> > > pyarrow >> > > > >> > project. But I'm having trouble running one of commands as >> > > specified in >> > > > >> the >> > > > >> > development docs on Github, specifically this command: >> > > > >> > >> > > > >> > cd arrow/python >> > > > >> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \ >> > > > >> > --with-parquet --with-plasma --inplace >> > > > >> > >> > > > >> > The error output looks like this: >> > > > >> > >> > > > >> > running build_ext-- Runnning cmake for pyarrow >> > > > >> > cmake >> > > > >> >> > > >> -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python >> > > > >> > -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on >> > > > >> > -DCMAKE_BUILD_TYPE= >> /Users/simba/Projects/personal/oss/arrow/python >> > > > >> > INFOCompiler command: >> > > /Library/Developer/CommandLineTools/usr/bin/c++ >> > > > >> > INFOCompiler version: Apple LLVM version 8.0.0 >> > > > >> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0 >> > > > >> > Thread model: posixInstalledDir: >> > > > >> /Library/Developer/CommandLineTools/usr/bin >> > > > >> > >> > > > >> > INFOCompiler id: Clang >> > > > >> > Selected compiler clang 3.8.0svn >> > > > >> > Configured for DEBUG build (set with cmake >> > > > >> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- >> Build >> > > > >> > output directory: >> > > > >> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/-- >> > > > >> > Checking for module 'arrow'-- No package 'arrow' found-- >> Found the >> > > > >> > Arrow core library: >> > > > >> > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib-- >> > > > >> > Found the Arrow Python library: >> > > > >> > >> > > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib >> > > > >> > Added shared library dependency arrow: >> > > > >> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib >> > > > >> > Added shared library dependency arrow_python: >> > > > >> > >> > > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib-- >> > > > >> > Checking for module 'parquet'-- No package 'parquet' found-- >> Found >> > > > >> > the Parquet library: >> > > > >> > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib >> > > > >> > Added shared library dependency parquet: >> > > > >> > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib-- >> > > > >> > Checking for module 'plasma'-- No package 'plasma' found-- >> Found >> > > the >> > > > >> > Plasma core library: >> > > > >> > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- >> > > > >> > Found Plasma executable: >> > > > >> > Added shared library dependency libplasma: >> > > > >> > >> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib-- >> > > > >> > Configuring done-- Generating done-- Build files have been >> written >> > > to: >> > > > >> > /Users/simba/Projects/personal/oss/arrow/python-- Finished >> cmake for >> > > > >> > pyarrow-- Running cmake --build for pyarrow >> > > > >> > makemake: *** No targets specified and no makefile found. >> > > Stop.error: >> > > > >> > command 'make' failed with exit status 2 >> > > > >> > >> > > > >> > >> > > > >> > It looks like there's a change dir happening at this line in the >> > > > >> setup.py: >> > > > >> > >> https://github.com/apache/arrow/blob/master/python/setup.py#L136 >> > > > >> > Which, in my case, is switching to the temp build which doesn't >> > > have the >> > > > >> > required Makefile to run the make command. >> > > > >> > >> > > > >> > I could be missing something because I was able to build the >> project >> > > > >> > successfully for python3. But I'd like to build it in python2.7 >> to >> > > > >> attempt >> > > > >> > a bug fix for this issue: >> > > > >> https://issues.apache.org/jira/browse/ARROW-1976 >> > > > >> > >> > > > >> > Thanks for help. >> > > > >> > >> > > > >> > Kind Regards >> > > > >> > Simba >> > > > >> >> > > > > >> > > >>
