[
https://issues.apache.org/jira/browse/IMPALA-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18039531#comment-18039531
]
ASF subversion and git services commented on IMPALA-14512:
----------------------------------------------------------
Commit 3ce0004c12f587dbc6ac6d7451b5f362d148d033 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3ce0004c1 ]
IMPALA-14512: Remove dependency on sh python package
This modifies bin/single_node_perf_run.py to stop using the sh
python package. It replaces sh with calls to subprocess. It
stops installing sh for both the Python 2 and 3 virtualenvs.
Testing:
- Ran perf-AB-test job with it and examined the logs
Change-Id: Ic5f9316a5d83c5c0dc37d4a94c55b6a655765fe3
Reviewed-on: http://gerrit.cloudera.org:8080/23600
Reviewed-by: Riza Suminto <[email protected]>
Reviewed-by: Jason Fehr <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> impala-python3 needs a newer version of sh
> ------------------------------------------
>
> Key: IMPALA-14512
> URL: https://issues.apache.org/jira/browse/IMPALA-14512
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> bin/single_node_perf_run.py uses the sh package. When it switched to python3,
> this line stopped working and instead produces this error:
> {noformat}
> 14:15:59 Exception in thread Thread-30:
> 14:15:59 Traceback (most recent call last):
> 14:15:59 File
> "/home/ubuntu/Impala/toolchain/toolchain-packages-gcc10.4.0/python-3.8.18/lib/python3.8/threading.py",
> line 932, in _bootstrap_inner
> 14:15:59 self.run()
> 14:15:59 File
> "/home/ubuntu/Impala/toolchain/toolchain-packages-gcc10.4.0/python-3.8.18/lib/python3.8/threading.py",
> line 870, in run
> 14:15:59 self._target(*self._args, **self._kwargs)
> 14:15:59 File
> "/home/ubuntu/Impala/infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/sh.py",
> line 1484, in output_thread
> 14:15:59 done = stream.read()
> 14:15:59 File
> "/home/ubuntu/Impala/infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/sh.py",
> line 1974, in read
> 14:15:59 self.write_chunk(chunk)
> 14:15:59 File
> "/home/ubuntu/Impala/infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/sh.py",
> line 1949, in write_chunk
> 14:15:59 self.should_quit = self.process_chunk(chunk)
> 14:15:59 File
> "/home/ubuntu/Impala/infra/python/env-gcc10.4.0-py3/lib/python3.8/site-packages/sh.py",
> line 1847, in process
> 14:15:59 handler.write(chunk)
> 14:15:59 TypeError: write() argument must be str, not bytes{noformat}
> This is easy to reproduce locally by running impala-python3 from IMPALA_HOME:
> {noformat}
> import sh
> import sys
> sh.cat('README.md', _out=sys.stdout){noformat}
> This will fail. We are currently only an old sh version, and sh=2.2.2 works
> properly with our python3 version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
