[
https://issues.apache.org/jira/browse/IMPALA-11980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17953190#comment-17953190
]
ASF subversion and git services commented on IMPALA-11980:
----------------------------------------------------------
Commit ea0969a772631dd8bcd0f1c5b9a46f2377ebb6d3 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ea0969a77 ]
IMPALA-11980 (part 2): Fix absolute import issues for impala_shell
Python 3 changed the behavior of imports with PEP328. Existing
imports become absolute unless they use the new relative import
syntax. This adapts the impala-shell code to use absolute
imports, fixing issues where it is imported from our test code.
There are several parts to this:
1. It moves impala shell code into shell/impala_shell.
This matches the directory structure of the PyPi package.
2. It changes the imports in the shell code to be
absolute paths (i.e. impala_shell.foo rather than foo).
This fixes issues with Python 3 absolute imports.
It also eliminates the need for ugly hacks in the PyPi
package's __init__.py.
3. This changes Thrift generation to put it directly in
$IMPALA_HOME/shell rather than $IMPALA_HOME/shell/gen-py.
This means that the generated Thrift code is rooted in
the same directory as the shell code.
4. This changes the PYTHONPATH to include $IMPALA_HOME/shell
and not $IMPALA_HOME/shell/gen-py. This means that the
test code is using the same import paths as the pypi
package.
With all of these changes, the source code is very close
to the directory structure of the PyPi package. As long as
CMake has generated the thrift files and the Python version
file, only a few differences remain. This removes those
differences by moving the setup.py / MANIFEST.in and other
files from the packaging directory to the top-level
shell/ directory. This means that one can pip install
directly from the source code. i.e. pip install $IMPALA_HOME/shell
This also moves the shell tarball generation script to the
packaging directory and changes bin/impala-shell.sh to use
Python 3.
This sorts the imports using isort for the affected Python files.
Testing:
- Ran a regular core job with Python 2
- Ran a core job with Python 3 and verified that the absolute
import issues are gone.
Change-Id: Ica75a24fa6bcb78999b9b6f4f4356951b81c3124
Reviewed-on: http://gerrit.cloudera.org:8080/22330
Reviewed-by: Riza Suminto <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Riza Suminto <[email protected]>
> Impala-shell should switch to absolute imports
> ----------------------------------------------
>
> Key: IMPALA-11980
> URL: https://issues.apache.org/jira/browse/IMPALA-11980
> Project: IMPALA
> Issue Type: Sub-task
> Components: Clients
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Various tests import code from the shell codebase:
> {noformat}
> $ git grep 'from shell\..*import'
> custom_cluster/test_hs2_fault_injection.py:from shell.ImpalaHttpClient import
> ImpalaHttpClient
> custom_cluster/test_hs2_fault_injection.py:from shell.impala_client import
> ImpalaHS2Client
> custom_cluster/test_hs2_fault_injection.py:from shell.shell_exceptions import
> HttpError
> custom_cluster/test_shell_interactive_reconnect.py:from shell.impala_shell
> import ImpalaShell as ImpalaShellClass
> shell/test_cookie_util.py:from shell.cookie_util import (cookie_matches_path,
> get_cookie_expiry,
> shell/test_shell_client.py:from shell.impala_client import
> ImpalaBeeswaxClient, ImpalaHS2Client
> shell/test_shell_commandline.py:from shell.impala_shell import ImpalaShell as
> ImpalaShellClass
> shell/test_shell_interactive.py:from shell.impala_shell import ImpalaShell as
> ImpalaShellClass{noformat}
> The shell codebase currently uses relative imports, so when using Python 3,
> the imports within the shell codebase will not work. For example,
> impala_shell.py includes compatibility.py via a relative import:
> {noformat}
> from compatibility import _xrange as xrange{noformat}
> In order for this to work under Python 3, these relative imports will need to
> be absolute. However, some ways of invoking the shell will fail with absolute
> imports (e.g. bin/impala-shell.sh). Other form factors are fine with absolute
> imports (e.g. pip installs into a venv).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
