This is an automated email from the ASF dual-hosted git repository.
dbecker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 451543a2e IMPALA-11785: Warn if Thrift fastbinary is not working for
impala-shell
451543a2e is described below
commit 451543a2e56cc6a5830c7beca74d2153315bb15d
Author: Joe McDonnell <[email protected]>
AuthorDate: Tue Apr 25 12:24:19 2023 -0700
IMPALA-11785: Warn if Thrift fastbinary is not working for impala-shell
Thrift's fastbinary module provides native code that
accelerations the BinaryProtocol. It can make a large
performance difference when using the Hiveserver2
protocol with impala-shell. If the fastbinary is not
working, it silently falls back to interpreted code.
This can happen because the fastbinary couldn't load
a particular library, etc.
This adds a warning on impala-shell startup when
it detects that Thrift's fastbinary is not working.
When bin/impala-shell.sh is modified to use python3,
impala-shell outputs this error (shortened for legibility):
WARNING: Failed to load Thrift's fastbinary module. Thrift's
BinaryProtocol will not be accelerated, which can reduce performance.
Error was '{path to Python2 thrift fastbinary.so}: undefined symbol:
_Py_ZeroStruct'
Testing:
- Added a simple test that verifies the impala-shell
does not output the warning
- Outputs warning when Python 2 thrift used for Python 3 shell
Change-Id: Id5d0e5db5cfdf1db4521b00f912b4697a7f646e8
Reviewed-on: http://gerrit.cloudera.org:8080/19806
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
shell/impala_shell.py | 8 ++++++++
tests/shell/test_shell_commandline.py | 6 ++++++
2 files changed, 14 insertions(+)
diff --git a/shell/impala_shell.py b/shell/impala_shell.py
index 9905dc988..e531d11b8 100755
--- a/shell/impala_shell.py
+++ b/shell/impala_shell.py
@@ -2193,6 +2193,14 @@ def impala_shell_main():
if options.verbose:
print("SSL is enabled", file=sys.stderr)
+ if options.verbose:
+ try:
+ import thrift.protocol.fastbinary
+ except Exception as e:
+ print("WARNING: Failed to load Thrift's fastbinary module. Thrift's "
+ "BinaryProtocol will not be accelerated, which can reduce
performance. "
+ "Error was '{0}'".format(e), file=sys.stderr)
+
if options.output_file:
try:
# Make sure the given file can be opened for writing. This will also
clear the file
diff --git a/tests/shell/test_shell_commandline.py
b/tests/shell/test_shell_commandline.py
index e91ccc51f..9b209f86d 100644
--- a/tests/shell/test_shell_commandline.py
+++ b/tests/shell/test_shell_commandline.py
@@ -186,6 +186,12 @@ class TestImpalaShell(ImpalaTestSuite):
results = run_impala_shell_cmd(vector, [], wait_until_connected=False)
assert "with no authentication" in results.stderr
+ def test_fastbinary_warning_message(self, vector):
+ results = run_impala_shell_cmd(vector, [], wait_until_connected=False)
+ # Verify that we don't print any message about fastbinary
+ # This doesn't check the full error string, because that could change.
+ assert "fastbinary" not in results.stderr
+
def test_print_header(self, vector, populated_table):
args = ['--print_header', '-B', '--output_delim=,', '-q',
'select * from {0}'.format(populated_table)]
