kazdy created HUDI-5578:
---------------------------
Summary: Upgrade base docker image for java 8
Key: HUDI-5578
URL: https://issues.apache.org/jira/browse/HUDI-5578
Project: Apache Hudi
Issue Type: Improvement
Components: dev-experience
Reporter: kazdy
Assignee: kazdy
Fix For: 1.0.0
The existing Hudi docker image uses a pretty old Debian distribution, therefore
it comes with Python 3.5.3. When using it with Spark 3.2.1, running "pyspark"
command in container shell results in below error:
{code:java}
root@e9fb3f81bdc9:/opt# pyspark
Python 3.5.3 (default, Nov 4 2021, 15:29:10)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/opt/spark/python/pyspark/shell.py", line 29, in <module>
from pyspark.context import SparkContext
File "/opt/spark/python/pyspark/__init__.py", line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File "/opt/spark/python/pyspark/rdd.py", line 48, in <module>
from pyspark.traceback_utils import SCCallSiteSync
File "/opt/spark/python/pyspark/traceback_utils.py", line 23, in <module>
CallSite = namedtuple("CallSite", "function file linenum")
File "/opt/spark/python/pyspark/serializers.py", line 390, in namedtuple
for k, v in _old_namedtuple_kwdefaults.items():
AttributeError: 'NoneType' object has no attribute 'items' {code}
The image I used was:
[https://hub.docker.com/r/apachehudi/hudi-hadoop_3.1.0-hive_3.1.2-sparkadhoc_3.2.1/tags]
Spark 3.2.1 requires Python 3.6+, Spark 3.3 requires Python 3.7+.
Base image for Java 8 uses openjdk:8u212-jdk-slim-stretch.
The goal is to upgrade it to
[8u342-jdk-slim-bullseye.|https://hub.docker.com/layers/library/openjdk/8u342-jdk-slim-bullseye/images/sha256-ecb89bb055c1ee4db9da38713b953f6daafefe575c77c6439eabbb85e3168402?context=explore]
This will also sync with existing java 11 base image distro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
