An update on Spark integration tests: the new error looks to be a regression so I made https://issues.apache.org/jira/browse/ARROW-7723 and marked as a blocker. It's possible to work around this bug, so I wouldn't call it a hard blocker if we need to proceed with the release.
On Wed, Jan 29, 2020 at 7:45 AM Neal Richardson <neal.p.richard...@gmail.com> wrote: > The place where the segfault is triggered in the R nightlies is a couple of > tests after the one I added in that patch. If that patch is causing the > segfaults, we can skip the new test ( > > https://github.com/apache/arrow/blob/master/r/tests/testthat/test-parquet.R#L125 > ) > and investigate later. The patch is exercising previously existing > codepaths that were not tested, so I don't think that identifying and > fixing the segfault should be release blocking (though we should clearly > fix it). > > Neal > > > > On Wed, Jan 29, 2020 at 7:33 AM David Li <li.david...@gmail.com> wrote: > > > The Flight leak should be unrelated to that commit, the failing test > > already existed before - it's a flaky test > > https://issues.apache.org/jira/browse/ARROW-7721. > > > > I'm hoping to look at the issue this week but we might just want to > > disable the test for now. > > > > David > > > > On 1/29/20, Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > > > Hi, > > > > > > - The fuzzit builds has been disabled by Neal on the current master. > > > - Created a PR to resolve occasionally failing python dataset tests [1] > > > - Merged the fix for C# timezone error [2] > > > - Merged various fixes for the release scripts. > > > - The nightly Gandiva OS X build is still failing, but because of a > > travis > > > deployment timeout, which shouldn't block the release. > > > > > > We still have failing tests: > > > - failing Spark integration test > > > - failing nightly R builds (see the 2020-01-29 nightly report) > > > - master reports a Java flight memory leak > > > > > > Spark: > > > Joris created a fix for the immediate issue [3], but now we have a > > > different > > > spark test error, see the discussion in the PR [3]. > > > I put up a PR [6] to check the regressions 0.16 arrow release would > > > introduce > > > for spark interoperability, and it turns out that arrow 0.15.1 is not > > > compatible > > > with neither spark 2.4.4 nor spark 2.4.5-rc1, so 0.16 arrow release > could > > > only be compatible with spark 3.0 or spark master which we have tests > > for. > > > So I'm a bit confused how to interpret arrow backward compatibility > with > > > Spark, thus what should and what should not block the release. > > > Either way we'll need to fix the remaining spark issues and add nightly > > > spark > > > integration tests for both the next spark release and spark master. > > > > > > R: > > > There is the same segfault in each R nightly builds [4]. There was a > > single > > > change [5] which could introduce the regression compared to the > previous > > > builds. > > > I've tried to reproduce the builds using docker-compose, but locally > > > 3.6-bionic > > > has passed for me. I'm trying to wipe my local cache and rerun to see > > > whether I can reproduce it. > > > > > > Java/Flight leak: > > > The current master reports memory leak [6] which I guess is surfaced by > > > change [7] > > > > > > If we manage to fix the issues above today than I can cut RC2 tomorrow. > > > > > > Thanks, Krisztian > > > > > > [1]: https://github.com/apache/arrow/pull/6319 > > > [2]: https://github.com/apache/arrow/pull/6309 > > > [3]: https://github.com/apache/arrow/pull/6312 > > > [4]: https://github.com/ursa-labs/crossbow/branches/all?query=r-base > > > [5]: > > > > > > https://github.com/apache/arrow/commit/8b7911b086d120359e2000fbedb0c38c0f13f683 > > > [6]: https://github.com/apache/arrow/runs/415037585#step:5:1533 > > > [7]: > > > > > > https://github.com/apache/arrow/commit/8b42288f58caa84a40bb7a13c1731ff919c934f2 > > > > > > On Wed, Jan 29, 2020 at 11:06 AM Sutou Kouhei <k...@clear-code.com> > > wrote: > > >> > > >> Hi, > > >> > > >> > Thank you. After the C# download fix, I have the following C# test > > >> > failure: > > >> > https://gist.github.com/pitrou/d82ed1ff80db43b63f0c3d5e5f2474a4 > > >> > > >> https://github.com/apache/arrow/pull/6309 > > >> will fix it. > > >> > > >> I think that this is a test problem, not an implementation > > >> problem. > > >> > > >> > > >> Workaround: > > >> > > >> TZ=UTC dev/release/verify-release-candidate.sh ... > > >> > > >> > > >> Thanks, > > >> -- > > >> kou > > >> > > >> In <2bd07a17-600b-7f49-3ea1-a0b1acc91...@python.org> > > >> "Re: [VOTE] Release Apache Arrow 0.16.0 - RC1" on Wed, 29 Jan 2020 > > >> 10:11:46 +0100, > > >> Antoine Pitrou <anto...@python.org> wrote: > > >> > > >> > > > >> > Thank you. After the C# download fix, I have the following C# test > > >> > failure: > > >> > https://gist.github.com/pitrou/d82ed1ff80db43b63f0c3d5e5f2474a4 > > >> > > > >> > Regards > > >> > > > >> > Antoine. > > >> > > > >> > > > >> > > > >> > Le 29/01/2020 à 00:42, Sutou Kouhei a écrit : > > >> >> Hi, > > >> >> > > >> >>> Source verification succeeded on Java and C++ and then failed > > >> >>> downloading some C# thing: > > >> >>> https://gist.github.com/pitrou/5c4a98387153ef415ef64b8aa2457e63 > > >> >> > > >> >> I've created a pull request to fix it: > > >> >> https://github.com/apache/arrow/pull/6307 > > >> >> > > >> >> > > >> >> Thanks, > > >> >> -- > > >> >> kou > > > > > >
