Re: [Flight] Consistency in C++ / Java error handling

The intent is that the application can define/use whatever keys it wants - though C++/Python don't expose that fully, which is another shortcoming that should be addressed. We should also perhaps consider standardizing/exposing a way to set that extra_info header in Python/C++ from Java and vice ve

Re: Polymorphism in DataFusion

I think B) is closer to what I was thinking. We may be using the term statically and dynamically typed a little differently -- I am sorry for the confusion. I have somewhat lost track of exactly what we are proposing and for that I apologize. I propose a next step of sketching out a proposed API

Questions about S3 options

To use S3 it appears I can either use `pyarrow.fs.S3FileSystem` or I can use s3fs and it gets wrapped (I think) with `pyarrow.fs.DaskFileSystem`. However, I don't see any documentation for `pyarrow.fs.DaskFileSystem`. Is this option supported going forwards? I'm currently configuring an s3fs ins

Re: [Flight] Consistency in C++ / Java error handling

Cool yeah, I had noticed that and it is definitely something that should be fixed. I think that is only one part of it though, right? We'd still need to toss the keys in the metadata for it to be picked up on the C++ side. I'm not sure if that set of keys is documented or even appropriate to rely

Authentication Redeisgn

Hi Arrow-Dev, I've written up a proposal to make some enhancements to the authentication process. I've captured the overall goals on JIRA here: https://issues.apache.org/jira/browse/ARROW-9804 The proposal itself is in this Google Doc: https://docs.google.com/document/d/1TutHf9ttw3rLzBvYxJXP2q6fZ

Re: Polymorphism in DataFusion

Hi, Thank you for this enlightening discussion, Andrew! So, just to make sure I understood, are you proposing A), B) or something else? A) we should not accept / declare polymorphic operations: all types should be known based on the operation name (e.g. sum_f32, plus_f32, etc.) B) we should cont

Re: [Flight] Consistency in C++ / Java error handling

Ah - I see. I filed https://issues.apache.org/jira/browse/ARROW-9802. At a quick glance, I think what's wrong is StatusUtils.toGrpcStatus doesn't copy the metadata from the CallStatus into the gRPC status it constructs. This should be an easy enough fix if you'd like to look at it. (Adding an inte

Re: Subject: [VOTE] Release Apache Arrow 1.0.1 - RC0

+1 (binding) In addition to the verifications on https://github.com/apache/arrow/pull/7985, I built and checked the R package locally. Neal On Tue, Aug 18, 2020 at 2:55 PM Sutou Kouhei wrote: > +1 (binding) > > I ran the followings on Debian GNU/Linux sid: > > * INSTALL_NODE=0 \ > CUDA

Re: [Flight] Consistency in C++ / Java error handling

Hey David, Appreciate the quick response! I had been doing it that way, but I found that If I was sending something along the lines of: ErrorFlightMetadata errorFlightMetadata = new ErrorFlightMetadata(); errorFlightMetadata.insert("my_custom_key", "hello"); listener.error(CallStatus.INTERNAL

Re: [Flight] Consistency in C++ / Java error handling

Hey Patrick, How are you sending the error to the client from the server side? You should be calling `listener.error` or `listener.onError` with a `FlightRuntimeException` (which you can get via CallStatus..withDescription(...).toRuntimeException); this will get the status code and message to the

Re: Arrow sync call August 19 at 12:00 US/Eastern, 16:00 UTC

Attendees: Projjal Chanda Micah Kornfield Neal Richardson Andrew Wieteska Discussion: * 1.0.1 patch release is open for vote now, please test it out * ARROW-1644 nested parquet reading in C++: discussed division of subtasks, progress On Tue, Aug 18, 2020 at 4:05 PM Neal Richardson wrote: > Hi a

Re: Best ways to implement push notifications over Flight?

Hey Radu, A. I don't think there's a standard way. You can also think about the DoAction endpoint (which is a server-side stream) if you don't want to abuse DoGet/if you want push notifications to be non-Arrow data. B. This sounds fine. Reading from a stream is blocking, and it's guaranteed to un

[Flight] Consistency in C++ / Java error handling

Hey all, I'm working on a project that has a Java arrow flight server and has a pyarrow client, both version 1.0.0. I've been trying to get richer errors back to the python client instead of the generic "grpc error ...". I see that the spec explicitly doesn't try to define metadata: https://arrow

Best ways to implement push notifications over Flight?

Hi, I am looking at the best way to push notifications from a server to clients over flight and I have a few questions on the approach: A. Is there a standard way of doing it and/or does this fundamentally go against flight philosophy? B. One approach is to run a doGet and then have the server

[NIGHTLY] Arrow Build Report for Job nightly-2020-08-19-0

Arrow Build Report for Job nightly-2020-08-19-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-08-19-0 Failed Tasks: - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-08-19-0-azure-conda-osx-clang-py38 - cond