Re: [C++] ResumeProducing Future Causing Blocking

2022-07-21 Thread Weston Pace
> Do you have any suggestions for a temporary workaround? I don't have any great ideas at the moment but I will continue to think it over. Unfortunately, this is somewhat tied up with the ordering issue. Details... Our current implementation of "serial execution" is "executor==nullptr" and that

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread James Duong
Feedback I got here was to use static_cast: https://github.com/apache/arrow/pull/13532#issuecomment-1177488433 I'm indifferent as to whether we want to do the static_casts or just suppress the warning as you've done. Your PR isn't building the 32-bit build in CI btw. It fails finding OpenSSL: htt

RE: [C++] ResumeProducing Future Causing Blocking

2022-07-21 Thread Ivan Chau
It seems like for this to work currently, we would want to call ResumeProducing on the source nodes, but outside of the processing thread. I tried a solution by making the ResumeProducing call with a fire-and-forget std::thread, but this doesn't seem like a great idea; It gets pretty hairy when

[GitHub] [arrow-datafusion-python] alamb commented on pull request #5: Latest from datafusion-contrib

2022-07-21 Thread GitBox
alamb commented on PR #5: URL: https://github.com/apache/arrow-datafusion-python/pull/5#issuecomment-1191888186 👍 to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
Hi James. My PR makes the compiler ignore the warnings. As far as I understand, this issue cannot be consistently resolved within the Google paradigm arrow follows on this subject. The google paradigm requires to treat all the sizes as signed 64 bit integers, regardless of the architecture.

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread James Duong
Hi Arkadiy, I've been working on a PR for fixing 32-bit Visual Studio here which has some of the same changes. It also enables the 32-bit VS build in CI, which fails due to a ton of integer implicit cast warnings: https://github.com/apache/arrow/pull/13532 Most of this commit is fixing 32-bit cas

[Rust] Arrow Call Notes - 2022-07-21

2022-07-21 Thread Andrew Lamb
Hi, We had a call today. See more details in the agenda document [1] Attendees: Andrew Lamb, Dan Harris, Brent, Ian Joiner, Paddy Horan, Xudong Topics: * FlightSQL – described history and status of the progress in Rust * Discussed history and status of Row Format. Ian said he was interested in

[Discuss][Rust][DataFusion] Errors with Context

2022-07-21 Thread Andrew Lamb
I wanted to mention that there is a proposed addition to DataFusionError, to add a `Context` variant in an open PR [1] . The addition to DataFusionError might not be obvious as the PR is large and is largely focused on subqueries, but given the variety of opinions on Error handling in Rust in gene

Re: [DISCUSS] Disable dependabot automated PRs

2022-07-21 Thread Antoine Pitrou
+1 for disabling. Le 21/07/2022 à 15:35, Raul Cumplido Dominguez a écrit : Hi, There was a discussion on Zulip dev about disabling dependabot alerts and updates [1] Based on this Apache INFRA wiki page we should be able to disable them [2]. There are currently several open PRs from dependa

Re: [C++] Moving from -O3 to -O2 optimization level in release builds

2022-07-21 Thread Antoine Pitrou
Le 21/07/2022 à 16:34, Wes McKinney a écrit : Based on the discussion in https://github.com/apache/arrow/pull/13661, it seems that one major issue with switching to -O2 is that auto-vectorization (which we rely on in places) and perhaps some other optimization passes would have to be manually

[GitHub] [arrow-datafusion-python] matthewmturner commented on pull request #5: Latest from datafusion-contrib

2022-07-21 Thread GitBox
matthewmturner commented on PR #5: URL: https://github.com/apache/arrow-datafusion-python/pull/5#issuecomment-1191605004 @andygrove good for me - i just left a governance related question on the original issue proposing this move -- This is an automated message from the Apache Git Servic

Re: [C++] Moving from -O3 to -O2 optimization level in release builds

2022-07-21 Thread Wes McKinney
Based on the discussion in https://github.com/apache/arrow/pull/13661, it seems that one major issue with switching to -O2 is that auto-vectorization (which we rely on in places) and perhaps some other optimization passes would have to be manually enabled in gcc. This benchmark run just completed

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Raul Cumplido Dominguez
Yes, issues 1-3 are not related to your PR. On Thu, Jul 21, 2022 at 4:04 PM Arkadiy Vertleyb (BLOOMBERG/ 120 PARK) < avertl...@bloomberg.net> wrote: > Thanks Raul. > > Does this mean issues 1-3 are not really caused by my PR and I just need > to wait for them to be fixed? > > > From: dev@arrow.ap

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
Thanks Raul. Does this mean issues 1-3 are not really caused by my PR and I just need to wait for them to be fixed? From: dev@arrow.apache.org At: 07/21/22 09:51:09 UTC-4:00To: Arkadiy Vertleyb (BLOOMBERG/ 120 PARK ) , dev@arrow.apache.org Subject: Re: Help needed with PR #13659: Fixing buil

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Sten Larsson
Regarding 1: Looks like Homebrew recently moved glib-mkenums into a separate package called glib-utils. /Sten

Re: Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Raul Cumplido Dominguez
Hi Arkadiy, For issues 2 and 3 there is currently an issue [1] with the protobuf version [2] distributed with homebrew [3] happening on master. These ones should be fixed once the upstream homebrew package is distributed. Issue 1 is also happening on master and I am not sure whether the issue is t

[DISCUSS] Disable dependabot automated PRs

2022-07-21 Thread Raul Cumplido Dominguez
Hi, There was a discussion on Zulip dev about disabling dependabot alerts and updates [1] Based on this Apache INFRA wiki page we should be able to disable them [2]. There are currently several open PRs from dependabot [3]. We don't seem to use them so I would like to close the current PRs and

Re: Arrow sync call July 20 at 12:00 US/Eastern, 16:00 UTC

2022-07-21 Thread Rok Mihevc
> One failing test is the R ubuntu test. Rok noted it is likely unrelated as he is seeing elsewhere [7]. > [7] https://github.com/apache/arrow/runs/7424773120?check_suite_focus=true Dragos is looking into this. It does seem to be an R timezone issue [1]. [1] https://github.com/apache/arrow/comp

RE: [C++] ResumeProducing Future Causing Blocking

2022-07-21 Thread Ivan Chau
Thanks Sasha and Weston! The diagrams would be helpful! Would the new first class support in the scheduler be something similar to what's available currently in BackpressureMonitor? We are looking to implement some more custom backpressure schemes that depend on batch ordering/completion rather

Help needed with PR #13659: Fixing build/unit test issues in msvc/win32

2022-07-21 Thread Arkadiy Vertleyb (BLOOMBERG/ 120 PARK)
Hi all. Can someone help me understand how the changes in this PR (https://github.com/apache/arrow/pull/13659/commits/e77ec9a84dab750bf016f9f5bd02ea48f2c8d77f) caused the following build failures? Thanks, Arkadiy Here are the failures: 1) AMD64 MacOS 10.15 GLib & Ruby c_glib/arrow-glib/meson

Re: [C++] Moving from -O3 to -O2 optimization level in release builds

2022-07-21 Thread Yaron Gvili
> only enable -O3 on source files selectively that can be demonstrated to > benefit from it Unfortunately, actual benefits from -O3 are application dependent. As https://www.linuxjournal.com/article/7269 explains: "Although -O3 can produce fast code, the increase in the size of the image can h

[GitHub] [arrow-datafusion-python] andygrove commented on pull request #5: Latest from datafusion-contrib

2022-07-21 Thread GitBox
andygrove commented on PR #5: URL: https://github.com/apache/arrow-datafusion-python/pull/5#issuecomment-1191250037 @jimexist @cpcloud @samkaufman @jychen7 @alamb @matthewmturner Could you please review when you have time. -- This is an automated message from the Apache Git Service. To r

Re: [Rust] [DataFusion] Discuss moving Python bindings back to Apache Arrow

2022-07-21 Thread Andy Grove
Thanks, Julian. I have now created a new repo [1] containing the code as it was before being removed from the DataFusion repo and I have created a PR [2] to update this code to reflect the changes that were made while the code was hosted in the datafusion-contrib GitHub organization. These changes

[GitHub] [arrow-datafusion-python] dependabot[bot] opened a new pull request, #1: Bump numpy from 1.21.3 to 1.22.0

2022-07-21 Thread GitBox
dependabot[bot] opened a new pull request, #1: URL: https://github.com/apache/arrow-datafusion-python/pull/1 Bumps [numpy](https://github.com/numpy/numpy) from 1.21.3 to 1.22.0. Release notes Sourced from https://github.com/numpy/numpy/releases";>numpy's releases. v1.22.0

[GitHub] [arrow-datafusion-python] andygrove opened a new pull request, #5: Latest from datafusion-contrib

2022-07-21 Thread GitBox
andygrove opened a new pull request, #5: URL: https://github.com/apache/arrow-datafusion-python/pull/5 This PR contributes the changes that were made to the project since it moved to the `datafusion-contrib` github organization. -- This is an automated message from the Apache Git Service.

[GitHub] [arrow-datafusion-python] dependabot[bot] opened a new pull request, #4: Bump regex from 1.5.4 to 1.6.0

2022-07-21 Thread GitBox
dependabot[bot] opened a new pull request, #4: URL: https://github.com/apache/arrow-datafusion-python/pull/4 Bumps [regex](https://github.com/rust-lang/regex) from 1.5.4 to 1.6.0. Changelog Sourced from https://github.com/rust-lang/regex/blob/master/CHANGELOG.md";>regex's changelog

[GitHub] [arrow-datafusion-python] dependabot[bot] opened a new pull request, #3: Bump tokio from 1.12.0 to 1.13.1

2022-07-21 Thread GitBox
dependabot[bot] opened a new pull request, #3: URL: https://github.com/apache/arrow-datafusion-python/pull/3 Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.12.0 to 1.13.1. Release notes Sourced from https://github.com/tokio-rs/tokio/releases";>tokio's releases. Tokio

[GitHub] [arrow-datafusion-python] dependabot[bot] opened a new pull request, #2: Bump num-bigint from 0.4.2 to 0.4.3

2022-07-21 Thread GitBox
dependabot[bot] opened a new pull request, #2: URL: https://github.com/apache/arrow-datafusion-python/pull/2 Bumps [num-bigint](https://github.com/rust-num/num-bigint) from 0.4.2 to 0.4.3. Changelog Sourced from https://github.com/rust-num/num-bigint/blob/master/RELEASES.md";>num-b