@mrkn submitted a PR to add backtraces which was merged recently: https://github.com/apache/arrow/pull/9524
However I think the abort is a red herring - the C++ integration client aborts when anything fails. The UNAVAILABLE makes me think that the C++ client is starting and running before the Rust server has started listening, and then it fails. Either the harness should wait-for-ready or the C++ client should perhaps retry a few times on UNAVAILABLE. (One thing I would like to do, but never got around to, is changing servers to bind to port 0 and printing the actual port; then the harness would wait for the port to be printed before continuing. That would fix this, assuming my guess is correct, and would also avoid occasional failures where we bind to a port already in use for some reason.) Best, David On Fri, Feb 19, 2021, at 11:26, Andrew Lamb wrote: > I noticed that the Rust/CPP integration tests are failing seemingly > intermittently on master (and on Rust PRs). Is anyone else was looking into > this? I will have time tomorrow, but likely not today. > > There didn't seem to be any obviously open PRs or JIRA > > There are several commits that the little red `X` meaning that CI didn't > pass > https://github.com/apache/arrow/commits/master > > Here are some Some example CI runs that are failing > https://github.com/apache/arrow/runs/1935673508 > https://github.com/apache/arrow/runs/1926705212 > > Here is an example: > > Testing file auth:basic_proto > 9201 > <https://github.com/apache/arrow/runs/1926705212#step:8:9201>========================================================== > > 9202 <https://github.com/apache/arrow/runs/1926705212#step:8:9202>Traceback > (most recent call last): > 9203 > <https://github.com/apache/arrow/runs/1926705212#step:8:9203>################# > FAILURES ################# > 9204 <https://github.com/apache/arrow/runs/1926705212#step:8:9204> File > "/arrow/dev/archery/archery/integration/util.py", line 139, in run_cmd > 9205 <https://github.com/apache/arrow/runs/1926705212#step:8:9205> output = > subprocess.check_output(cmd, stderr=subprocess.STDOUT) > 9206 <https://github.com/apache/arrow/runs/1926705212#step:8:9206> File > "/opt/conda/envs/arrow/lib/python3.8/subprocess.py", line 411, in > check_output > 9207 <https://github.com/apache/arrow/runs/1926705212#step:8:9207> return > run(*popenargs, stdout=PIPE, timeout=timeout, check=True, > 9208 <https://github.com/apache/arrow/runs/1926705212#step:8:9208> File > "/opt/conda/envs/arrow/lib/python3.8/subprocess.py", line 512, in run > 9209 <https://github.com/apache/arrow/runs/1926705212#step:8:9209> raise > CalledProcessError(retcode, process.args, > 9210 > <https://github.com/apache/arrow/runs/1926705212#step:8:9210>subprocess.CalledProcessError: > Command '['/build/cpp/debug/flight-test-integration-client', '-host', > 'localhost', '-port=45809', '-scenario', 'auth:basic_proto']' died with > <Signals.SIGABRT: 6>. > 9211 <https://github.com/apache/arrow/runs/1926705212#step:8:9211> > 9212 <https://github.com/apache/arrow/runs/1926705212#step:8:9212>During > handling of the above exception, another exception occurred: > 9213 <https://github.com/apache/arrow/runs/1926705212#step:8:9213> > 9214 <https://github.com/apache/arrow/runs/1926705212#step:8:9214>Traceback > (most recent call last): > 9215 <https://github.com/apache/arrow/runs/1926705212#step:8:9215> File > "/arrow/dev/archery/archery/integration/runner.py", line 308, in > _run_flight_test_case > 9216 <https://github.com/apache/arrow/runs/1926705212#step:8:9216> > consumer.flight_request(port, **client_args) > 9217 <https://github.com/apache/arrow/runs/1926705212#step:8:9217> File > "/arrow/dev/archery/archery/integration/tester_cpp.py", line 116, in > flight_request > 9218 <https://github.com/apache/arrow/runs/1926705212#step:8:9218> > run_cmd(cmd) > 9219 <https://github.com/apache/arrow/runs/1926705212#step:8:9219> File > "/arrow/dev/archery/archery/integration/util.py", line 148, in run_cmd > 9220 <https://github.com/apache/arrow/runs/1926705212#step:8:9220> raise > RuntimeError(sio.getvalue()) > 9221 > <https://github.com/apache/arrow/runs/1926705212#step:8:9221>RuntimeError: > Command failed: /build/cpp/debug/flight-test-integration-client -host > localhost -port=45809 -scenario auth:basic_proto > 9222 <https://github.com/apache/arrow/runs/1926705212#step:8:9222>With > output: > 9223 > <https://github.com/apache/arrow/runs/1926705212#step:8:9223>-------------- > > 9224 <https://github.com/apache/arrow/runs/1926705212#step:8:9224>-- Arrow > Fatal Error -- > 9225 <https://github.com/apache/arrow/runs/1926705212#step:8:9225>Invalid: > Expected UNAUTHENTICATED but got Unavailable > 9226 <https://github.com/apache/arrow/runs/1926705212#step:8:9226> >
