[VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-28 Thread Micah Kornfield
I've opened a PR that updates the specification to allow for 256-bit Decimal types [1]. It updates both schema.fbs and the C-ABI to document this support. The decimal256 branch [2] contains implementations in Java and C++ and updates to the integration test to demonstrate interoperability. If th

Re: [DISCUSS] Big Endian support in Arrow (was: Re: [Java] Supporting Big Endian)

2020-09-28 Thread Micah Kornfield
OK, I will try to update documentation reflecting this in the next few days (in particular it would be good to document which implementations are willing to support byte flipping). On Tue, Sep 22, 2020 at 3:30 AM Antoine Pitrou wrote: > > > Le 22/09/2020 à 06:36, Micah Kornfield a écrit : > > I

[FlightRPC][Java][Python/c_lib] Encryption without validation in Flight clients

2020-09-28 Thread James Duong
Hi Arrow-Dev, I've created a JIRA for supporting using Flight clients to connect to Flight servers without verifying the server certificate. This is something I've seen done frequently for other connectivity tools such as JDBC drivers. I'll be looking to implement this one for the and hopefully

Re: [C++] Criteria for upgrading Arrow to C++14?

2020-09-28 Thread Wes McKinney
I see. gcc 4.9 doesn't support all of the C++14 standard, so if we definitely need to keep supporting gcc 4.9 for another 6 months, then we may have to wait. > 5 years is a long time to be frozen on gcc 4.x (4.8 was the minimum version when we started the project). On Mon, Sep 28, 2020 at 4:10 PM

Re: [C++] Criteria for upgrading Arrow to C++14?

2020-09-28 Thread Neal Richardson
R is looking much more favorable for C++14 or even C++17 since the 4.0 release in April: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-C_002b_002b14-and-later It's unclear how well the old Rtools Windows toolchain would support C++14--we can try and see, at least it's gcc 4.9.

[C++] Criteria for upgrading Arrow to C++14?

2020-09-28 Thread Wes McKinney
hi folks, I think this has come up before, but I'm curious what would be the roadblocks (if any) for upgrading our minimum C++ standard from 11 to 14. One of the last hangers-on is probably our Python manylinux1 build which uses gcc 4.8, but is there anything else? The minimum supported MSVC for C

Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC

2020-09-28 Thread Wes McKinney
hi Ryan, I'm interested in enabling this effort to move forward -- is there any part where feedback or other help could be useful? Thanks, Wes On Wed, Sep 2, 2020 at 4:45 PM Wes McKinney wrote: > > I haven't been able to scrutinize the details greatly, but I am > supportive of this effort, othe

Re: Pyarrow RecordBatchStreamWriter and dictionaries

2020-09-28 Thread Wes McKinney
hi Al, It's definitely wrong. I confirmed the behavior is present on master. https://issues.apache.org/jira/browse/ARROW-10121 I made this a blocker for the release. Thanks, Wes On Mon, Sep 28, 2020 at 10:52 AM Al Taylor wrote: > > Hi, > > I've found that when I serialize two recordbatches wh

Pyarrow RecordBatchStreamWriter and dictionaries

2020-09-28 Thread Al Taylor
Hi, I've found that when I serialize two recordbatches which have a dictionary-encoded field, but different encoding dictionaries to a sequence of pybytes with a RecordBatchStreamWriter, then deserialize using pa.ipc.open_stream(), the dictionaries get jumbled. (or at least, on deserialization

Re: [DISCUSS] Rethinking our approach to scheduling CPU and IO work in C++?

2020-09-28 Thread Antoine Pitrou
Le 28/09/2020 à 11:38, Antoine Pitrou a écrit : > > Hi Weston, > > Le 25/09/2020 à 23:21, Weston Pace a écrit : >> >> * The current thread pool implementation deadlocks when used in a >> "nested" case, an asynchronous solution can work around this > > If required it may be possible to hack aroun

[NIGHTLY] Arrow Build Report for Job nightly-2020-09-28-0

2020-09-28 Thread Crossbow
Arrow Build Report for Job nightly-2020-09-28-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-28-0 Failed Tasks: - conda-linux-gcc-py36-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-28-0-drone-conda-linux-gcc-py3

MEAN Stack Use-case understanding

2020-09-28 Thread thomasroshin
Hello , I am working on a proof-of-concept for which I am having a bit of trouble understanding apache-arrow with JS and wanted to clarify a few things with this regard. My use case- I have a MEAN (MongoDB/Express/Angular/NodeJS) that connects to customer databases and third-party dat

Re: [DISCUSS] Rethinking our approach to scheduling CPU and IO work in C++?

2020-09-28 Thread Antoine Pitrou
Hi Weston, Le 25/09/2020 à 23:21, Weston Pace a écrit : > > * The current thread pool implementation deadlocks when used in a > "nested" case, an asynchronous solution can work around this If required it may be possible to hack around this. For example, AFAIR TBB has a simple heuristic to ena