[
https://issues.apache.org/jira/browse/IMPALA-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18060813#comment-18060813
]
ASF subversion and git services commented on IMPALA-14770:
----------------------------------------------------------
Commit 4e2189391c6cefa118d9f7b915eabd19f4a38189 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4e2189391 ]
IMPALA-14770: Upgrade impala-python3 to 3.11.14
This upgrades the impala-python3 virtualenv to use Python 3.11.14.
Upgrading to Python 3.11 allows us to use the most recent version
of most packages. Python 3.11 delivered performance improvements
over previous versions. It seems to reduce the runtime of precommit
by about 15-20 minutes (~4:40 -> ~4:25).
This fixes code using removed library functions:
- XML's getiterator() was removed and code needs to use iter()
- thread's isAlive() was removed and code needs to use is_alive()
This fixes some pieces of code to avoid warnings:
- The "pipes" package is deprecated, so this changes code using
pipes.quote() to use shlex.quote() instead.
- This changes some regex patterns to use raw strings to avoid
warnings about invalid escape sequences.
This upgrades a variety of packages (some of them required, many of them
optional):
1. Upgrades setuptools to a version from about a month ago
2. Upgrades requests to the latest version
- Addresses some warnings that show for pytest
- Requests got stricter about certificates and server-cert.pem
is no longer considered valid.
- This regenerates the server-key.pem/server-key-password.pem/server-cert.pem
to add a subject alternative name of localhost.
3. Upgrades python-magic to the latest and fixes dump_breakpad_symbols.py
4. Upgrades psutil to the latest and fixes tests/common/impala_cluster.py to
handle the changes in how it represents zombie processes
5. Upgrades several other Python packages to more recent versions without
needing code changes (allpairspy, flake8, hdfs, kazoo, prometheus-client,
six, sasl, avro, distro, pytz, Cython). Removes unused pg8000 package.
There were issues upgrading some other packages, so this does not
upgrade them:
- Newer versions of sqlparse have a regression in split.split()
that makes it inoperable for us.
- Newer versions of pexpect would require adjustment in the
interactive shell tests.
- New versions of pytest are not difficult but require additional changes
that would be better suited to a separate commit
Testing:
- Ran precommit
- Ran exhaustive release jobs
Change-Id: I9383ebfb310a4ee30134e02b195c332ffbb6a7ec
Reviewed-on: http://gerrit.cloudera.org:8080/23976
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Upgrade the impala-python3 virtualenv to Python 3.11 or higher
> --------------------------------------------------------------
>
> Key: IMPALA-14770
> URL: https://issues.apache.org/jira/browse/IMPALA-14770
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Upgrading to a newer Python 3 could unblock our package upgrades and let us
> use the latest versions for most packages. Python 3.8 and 3.9 are EOL and
> Python 3.10 is getting close. Many Python packages have been requiring Python
> 3.9 and some even require Python 3.10. Python 3.11 has substantial
> performance improvements over previous versions, so it seems reasonable to
> target Python 3.11 or above for this upgrade.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
