[jira] [Created] (ARROW-2751) [GLib] Add garrow_table_replace_column()

2018-06-26 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-2751: --- Summary: [GLib] Add garrow_table_replace_column() Key: ARROW-2751 URL: https://issues.apache.org/jira/browse/ARROW-2751 Project: Apache Arrow Issue Type: New F

Re: Gandiva Initiative

2018-06-26 Thread Pindikura Ravindra
Sorry for the delay, Julian. My replies inline. On Fri, Jun 22, 2018 at 11:39 PM Julian Hyde wrote: > This is exciting. We have wanted to build an Arrow adapter in Calcite for > some time and have a prototype (see > https://issues.apache.org/jira/browse/CALCITE-2173 < > https://issues.apache.org

Re: [DISCUSS] Developing a standard memory layout for in-memory records / "row-oriented" data

2018-06-26 Thread Jacques Nadeau
I'm not sure this makes sense as an external stable api. I definitely think it is useful as an internal representation for use within a particular algorithm. I also think that can be informed by the particular algorithm that you're working on. We definitely had this requirement in Dremio and came

[jira] [Created] (ARROW-2750) Add MATLAB support for reading numeric types from Feather files

2018-06-26 Thread Kevin Gurney (JIRA)
Kevin Gurney created ARROW-2750: --- Summary: Add MATLAB support for reading numeric types from Feather files Key: ARROW-2750 URL: https://issues.apache.org/jira/browse/ARROW-2750 Project: Apache Arrow

Re: Housing longer-term Arrow development, design, and roadmap documents

2018-06-26 Thread Antoine Pitrou
Note: if/when we create a top-level Sphinx-based documentation, it can become an appropriate place for developer-facing docs as well. Writing reStructuredText prose is sufficiently fluid. (but submitting changes in that scheme would still require a PR, perhaps) Regards Antoine. Le 26/06/20

Re: Housing longer-term Arrow development, design, and roadmap documents

2018-06-26 Thread Julian Hyde
+1 And thanks, Wes, for your ongoing time-commitment mowing the many lawns in this project. > On Jun 26, 2018, at 10:29 AM, Wes McKinney wrote: > > I agree that wikis often become a mess -- I realize that I'm > essentially volunteering myself to be one of the librarians if we end > up with a s

Re: Housing longer-term Arrow development, design, and roadmap documents

2018-06-26 Thread Wes McKinney
I agree that wikis often become a mess -- I realize that I'm essentially volunteering myself to be one of the librarians if we end up with a substantial trove of documents. I'm definitely not in favor of putting user documentation in a wiki, only developer-facing design and planning documents. We c

Re: Housing longer-term Arrow development, design, and roadmap documents

2018-06-26 Thread Julian Hyde
I have a bias against wikis of all kinds. If left to their own devices, they tend to become an unstructured mess. Of course, the lack of structure is what makes them useful for what Wes is proposing: gathering knowledge and organizing it as it evolves. But someone will need to play the “librari

Re: [DISCUSS] Developing a standard memory layout for in-memory records / "row-oriented" data

2018-06-26 Thread Wes McKinney
hi Antoine, On Sun, Jun 24, 2018 at 1:06 PM, Antoine Pitrou wrote: > > Hi Wes, > > Le 24/06/2018 à 08:24, Wes McKinney a écrit : >> >> If this sounds interesting to the community, I could help to kickstart >> a design process which would likely take a significant amount of time. >> The requiremen

Re: Some initial GPU questions

2018-06-26 Thread Wes McKinney
hi Anthony, Antoine is right that a Device abstraction is needed. I hadn't seen ARROW-2447 (I was on vacation in April) but I will comment there. It would be helpful to collect more requirements from GPU users -- one of the reasons that I set up the arrow/gpu project to begin with was to help cat

Re: Some initial GPU questions

2018-06-26 Thread Antoine Pitrou
Hi Anthony, Le 26/06/2018 à 17:55, Anthony Scopatz a écrit : > >2. Should arrow compute assume that everything fits in memory? Arrow >seem to handle data that is larger than memory via the Buffer API. Are >there restrictions that using Buffers imply that we should be aware of? A Bu

Some initial GPU questions

2018-06-26 Thread Anthony Scopatz
Hello All, As some of you may know, a few of us at Quansight have started (in parntership with NVIDIA) have started looking at Arrow's GPU capabilites. We are excited to help improve and expand Arrow's GPU support, but we did have a few initial scoping questions. Feel free to break these out into

Re: Housing longer-term Arrow development, design, and roadmap documents

2018-06-26 Thread Wes McKinney
GitHub wiki pages lack collaboration features like commenting. It will be interesting to see what we can work up with JIRA integration, e.g. burndown charts for release management. I asked INFRA to create a Confluence space for us so we can give it a try to see if it works for us. Confluence seems

Re: [DISCUSS] Concerns about the Arrow Slack channel

2018-06-26 Thread Wes McKinney
I would personally prefer to have all questions on the mailing list for now. I don't know if the community is large enough to provide consistent attention on an additional communication channel. If there end up being too many user-centric questions on dev@, we can activate and use user@ as some oth

Re: [DISCUSS] Concerns about the Arrow Slack channel

2018-06-26 Thread Dhruv Madeka
It might be nice to have the discourse option before shutting it down. As someone who asks, that would be a nice way to get me to migrate On Tue, Jun 26, 2018 at 10:53 AM, Wes McKinney wrote: > hi folks, > > How would you like to proceed on the Slack channel discussion? It > seems there is reaso

Re: [DISCUSS] Concerns about the Arrow Slack channel

2018-06-26 Thread Wes McKinney
hi folks, How would you like to proceed on the Slack channel discussion? It seems there is reasonable consensus to close the channel. Should we have a vote? It would be a good idea to export the data / chat history from the channel before closing it down. Thanks Wes On Thu, Jun 21, 2018 at 11:3

[jira] [Created] (ARROW-2749) [GLib] Return GArrowDecimal128 instead of gchar * in garrow_decimal128_array_get_value

2018-06-26 Thread yosuke shiro (JIRA)
yosuke shiro created ARROW-2749: --- Summary: [GLib] Return GArrowDecimal128 instead of gchar * in garrow_decimal128_array_get_value Key: ARROW-2749 URL: https://issues.apache.org/jira/browse/ARROW-2749 Pr

[jira] [Created] (ARROW-2748) [GLib] Add garrow_decimal_data_type_get_scale() (and _precision())

2018-06-26 Thread yosuke shiro (JIRA)
yosuke shiro created ARROW-2748: --- Summary: [GLib] Add garrow_decimal_data_type_get_scale() (and _precision()) Key: ARROW-2748 URL: https://issues.apache.org/jira/browse/ARROW-2748 Project: Apache Arrow

Re: Build system discussion for Arrow (and Orc?)

2018-06-26 Thread Michael Sarahan
I believe the fix was in ORC - they greatly improved their CMake stuff since I first tried it. The brokenness was not necessarily in arrow, but in arrow's vendoring of orc, with orc using -werror by default. That is no longer the default. Anyway, all fixed now, and I think the CMake stuff is in

Re: Build system discussion for Arrow (and Orc?)

2018-06-26 Thread Wes McKinney
Thanks Michael. How did you hit the -Werror issue? I'm looking at your patch https://github.com/AnacondaRecipes/arrow-cpp-feedstock/blob/master/recipe/0001-remove-Werror-for-compilation-on-mac-and-linux-32.patch but this should not be impacting you. The default warning level is PRODUCTION, -Werr

[jira] [Created] (ARROW-2747) [CI] [Plasma] huge tables test failure on Travis

2018-06-26 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-2747: - Summary: [CI] [Plasma] huge tables test failure on Travis Key: ARROW-2747 URL: https://issues.apache.org/jira/browse/ARROW-2747 Project: Apache Arrow Issue

[jira] [Created] (ARROW-2746) [C++] Add option to use dynamic linking with Protobuf libraries

2018-06-26 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-2746: --- Summary: [C++] Add option to use dynamic linking with Protobuf libraries Key: ARROW-2746 URL: https://issues.apache.org/jira/browse/ARROW-2746 Project: Apache Arrow

Fixing linter warnings in Arrow Java?

2018-06-26 Thread Wes McKinney
hi folks, In light of ARROW-2608, we've begun building the Java libraries before running the C++ and Python unit tests. Unfortunately, the Java libraries have a little over 6000 lines of output, which is 2/3 of the size allowed by the Travis CI web UI (example: https://travis-ci.org/apache/arrow/

Re: Running Apache Arrow on Visual Studio

2018-06-26 Thread AggD
Sure. It will be up by the end of the week , when I find some free time .

Re: Running Apache Arrow on Visual Studio

2018-06-26 Thread Wes McKinney
Could you explain the resolution here in case someone else finds this thread? Much appreciated On Tue, Jun 26, 2018, 7:46 AM AggD wrote: > Okay , I managed to solve it. If anyone needs help feel free to ask me. If > you want , I can make a file explaining the operations needed. >

Re: Running Apache Arrow on Visual Studio

2018-06-26 Thread AggD
Okay , I managed to solve it. If anyone needs help feel free to ask me. If you want , I can make a file explaining the operations needed.

Re: Building against arrow static library

2018-06-26 Thread Wes McKinney
See note about this in https://github.com/apache/arrow/blob/master/python/doc/source/development.rst#known-issues On Tue, Jun 26, 2018 at 7:10 AM, Wes McKinney wrote: > Ah, ok, I think I know the issue then. You need to pass > -DCMAKE_CXX_FLAGS='-D_GLIBCXX_USE_CXX11_ABI=0' when building all your

Re: Building against arrow static library

2018-06-26 Thread Wes McKinney
Ah, ok, I think I know the issue then. You need to pass -DCMAKE_CXX_FLAGS='-D_GLIBCXX_USE_CXX11_ABI=0' when building all your libraries if you use the conda-forge libraries, since they are built using the old gcc ABI On Tue, Jun 26, 2018 at 5:11 AM, Praveen Kumar wrote: > Hi Wes, > > Thanks for t

Re: Running Apache Arrow on Visual Studio

2018-06-26 Thread AggD
I did everything from the beginning with no results. I include the .lib and .dll files but with no results. I have problem only with memory_pool .

Re: Building against arrow static library

2018-06-26 Thread Praveen Kumar
Hi Wes, Thanks for the suggestion. I purged the system boost libs and removed LLVM (suspecting it might be bringing in some static dependencies) too. Then recompiled Arrow and tried to link Gandiva again but no avail. It looks similar to the one seen here, https://issues.apache.org/jira/browse/AR

[jira] [Created] (ARROW-2745) [C++] ORC ExternalProject needs to declare dependency on vendored protobuf

2018-06-26 Thread Uwe L. Korn (JIRA)
Uwe L. Korn created ARROW-2745: -- Summary: [C++] ORC ExternalProject needs to declare dependency on vendored protobuf Key: ARROW-2745 URL: https://issues.apache.org/jira/browse/ARROW-2745 Project: Apache

Re: Running Apache Arrow on Visual Studio

2018-06-26 Thread AggD
I think I got it fine. Thank you for your time

[jira] [Created] (ARROW-2744) Writing to parquet crashes when writing a ListArray of empty lists

2018-06-26 Thread Anton Daitche (JIRA)
Anton Daitche created ARROW-2744: Summary: Writing to parquet crashes when writing a ListArray of empty lists Key: ARROW-2744 URL: https://issues.apache.org/jira/browse/ARROW-2744 Project: Apache Arr