Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22330 )
Change subject: IMPALA-11980 (part 2): Fix absolute import issues for impala_shell ...................................................................... IMPALA-11980 (part 2): Fix absolute import issues for impala_shell Python 3 changed the behavior of imports with PEP328. Existing imports become absolute unless they use the new relative import syntax. This adapts the impala-shell code to use absolute imports, fixing issues where it is imported from our test code. There are several parts to this: 1. It moves impala shell code into shell/impala_shell. This matches the directory structure of the PyPi package. 2. It changes the imports in the shell code to be absolute paths (i.e. impala_shell.foo rather than foo). This fixes issues with Python 3 absolute imports. It also eliminates the need for ugly hacks in the PyPi package's __init__.py. 3. This changes Thrift generation to put it directly in $IMPALA_HOME/shell rather than $IMPALA_HOME/shell/gen-py. This means that the generated Thrift code is rooted in the same directory as the shell code. 4. This changes the PYTHONPATH to include $IMPALA_HOME/shell and not $IMPALA_HOME/shell/gen-py. This means that the test code is using the same import paths as the pypi package. With all of these changes, the source code is very close to the directory structure of the PyPi package. As long as CMake has generated the thrift files and the Python version file, only a few differences remain. This removes those differences by moving the setup.py / MANIFEST.in and other files from the packaging directory to the top-level shell/ directory. This means that one can pip install directly from the source code. i.e. pip install $IMPALA_HOME/shell This also moves the shell tarball generation script to the packaging directory and changes bin/impala-shell.sh to use Python 3. This sorts the imports using isort for the affected Python files. Testing: - Ran a regular core job with Python 2 - Ran a core job with Python 3 and verified that the absolute import issues are gone. Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124 Reviewed-on: http://gerrit.cloudera.org:8080/22330 Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com> Reviewed-by: Michael Smith <michael.sm...@cloudera.com> Tested-by: Riza Suminto <riza.sumi...@cloudera.com> --- M bin/impala-shell.sh M bin/rat_exclude_files.txt M bin/set-pythonpath.sh M common/thrift/CMakeLists.txt M shell/.gitignore M shell/CMakeLists.txt A shell/MANIFEST.in R shell/README.md M shell/gen_impala_build_version.sh R shell/impala_shell/ImpalaHttpClient.py R shell/impala_shell/TSSLSocketWithWildcardSAN.py R shell/impala_shell/__init__.py R shell/impala_shell/compatibility.py R shell/impala_shell/cookie_util.py R shell/impala_shell/exec_summary.py R shell/impala_shell/impala_client.py R shell/impala_shell/impala_shell.py R shell/impala_shell/impala_shell_config_defaults.py R shell/impala_shell/kerberos_util.py R shell/impala_shell/option_parser.py R shell/impala_shell/shell_exceptions.py R shell/impala_shell/shell_output.py R shell/impala_shell/thrift_printer.py R shell/impala_shell/value_converter.py D shell/packaging/MANIFEST.in D shell/packaging/__init__.py R shell/packaging/impala-shell M shell/packaging/make_python_package.sh R shell/packaging/make_shell_tarball.sh R shell/requirements.txt R shell/setup.py M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/custom_cluster/test_hs2_fault_injection.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/custom_cluster/test_thrift_socket.py M tests/shell/test_cookie_util.py M tests/shell/test_kerberos_util.py M tests/shell/test_shell_client.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 42 files changed, 268 insertions(+), 266 deletions(-) Approvals: Riza Suminto: Looks good to me, but someone else must approve; Verified Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/22330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124 Gerrit-Change-Number: 22330 Gerrit-PatchSet: 22 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
