Re: [DISCUSS] Leap seconds/days and day light saving for Duration types

2021-09-13 Thread QP Hou
Good point Weston. My proposal was written with the impression that Arrow does want to define semantic for some of these temporal types based on the existing comments in the Schema.fbs file. For example, here is a quote taken from the comments for the Time time: /// This definition doesn't allow

Re: [DISCUSS] Leap seconds/days and day light saving for Duration types

2021-09-13 Thread Weston Pace
One could define a sorting based on 30 days months, 365 day years, and 24 hour days. It would be consistent but can lead to some surprising results. It appears that this is what postgres does as I got the following ordering for an interval: 359 days, 12 months, 360 days, 1 year, 365 days, 366 da

Re: [DataFusion] Question about async/await?

2021-09-13 Thread Evan Chan
The other suggestion would be to have a way to monitor and watch for when the CPU-bound thread pool saturates, which can result in queues backing up into the main dispatch async threads as well…. Ie there might be some spillover if the CPU thread pool fills up to watch out for. -Evan > On Sep

Re: [DISCUSS] Leap seconds/days and day light saving for Duration types

2021-09-13 Thread QP Hou
On Mon, Sep 13, 2021 at 6:18 AM Antoine Pitrou wrote: > The Duration type is defined with a TimeUnit. You are probably thinking > about the Interval type. > Oops, my bad, yes, it should be Interval type not Duration. > Ok. How about daylight savings? I suppose they are taken into account > as

Re: Arrow Flight Connection Handling

2021-09-13 Thread David Li
Hello, For this, you will have to depend on the fact that Flight is currently implemented with gRPC. This means: 1. Take a dependency on flight-grpc, 2. Configure a plain gRPC server following the grpc-java docs, 3. Add the Flight service to the gRPC server via FlightGrpcUtils[1] Then you can

Re: arrow_iterator.cpython-38-darwin.so - the developer cannot be verified

2021-09-13 Thread Jason Withrow
Oh, I bet it is snowflake-connector! Apologies for the misunderstanding. On 9/13/21, 9:21 AM, "Antoine Pitrou" wrote: Hello Jason, Le 13/09/2021 à 03:06, Jason Withrow a écrit : > Apologies in advance if this is the wrong forum to raise this issue. I > would be happy to file a

Re: [Errno 25] fcntl(fd, F_RDADVISE, ...) failed

2021-09-13 Thread Antoine Pitrou
Le 13/09/2021 à 15:20, Antoine Pitrou a écrit : Perhaps unrelated, but executing pyarrow.parquet.read_table against an NFS source in pyarrow 5.0.0 yields the following exception: (pyarrow 4.0.1 succeeds with the same sample) [...] OSError: [Errno 25] fcntl(fd, F_RDADVISE, ...) failed. Det

Arrow Flight Connection Handling

2021-09-13 Thread Mohamed Abdelhakem
in Java Arrow Flight protocol, I would like to access the client address and port whenever a FlightClient connects to FlightServer. is there a way to do so? Thanks in advance

Arrow Flight Connection Handling

2021-09-13 Thread Mohamed Abdelhakem
in java Arrow Flight, I would like to access the client host and port whenever a FlightClient get connected to FlightServer is there a way to do so? Thanks in Advance

Re: arrow_iterator.cpython-38-darwin.so - the developer cannot be verified

2021-09-13 Thread Antoine Pitrou
Hello Jason, Le 13/09/2021 à 03:06, Jason Withrow a écrit : Apologies in advance if this is the wrong forum to raise this issue. I would be happy to file a bug in Jira If more appropriate.  I am experiencing  issues accessing files over NFS from Big Sure with pyarrow 5.0.0. I am running an a

Re: [DISCUSS] Leap seconds/days and day light saving for Duration types

2021-09-13 Thread Antoine Pitrou
Le 13/09/2021 à 06:00, QP Hou a écrit : Hi, I would like to draw some attention to a format PR aiming to clarify leap seconds, leap days and daylight saving handling semantics for duration types: https://github.com/apache/arrow/pull/11138. This came out of the effort [1] trying to implement P

Re: [DataFusion] Question about async/await?

2021-09-13 Thread Andrew Lamb
I have found the suggestion in the tokio docs to avoid using tokio for CPU bound work very confusing. I think the core suggestion is not to use the same threadpool for IO and CPU bound work (which makes a lot of sense) but it is perfectly feasible to create multiple tokio threadpools (`Runtimes`) i