shml created HIVE-30011:
---------------------------
Summary: MR jobs submitted outside ExecDriver (MergeFileTask,
ColumnTruncateTask, MRCompactor) miss the JDK 17 --add-opens flags
Key: HIVE-30011
URL: https://issues.apache.org/jira/browse/HIVE-30011
Project: Hive
Issue Type: Bug
Components: Query Processor, Transactions
Affects Versions: 4.1.0
Reporter: shml
HIVE-28869 makes Hive append the JDK 17 \{{--add-opens}} flags to the JVMs it
launches itself: Tez AM / containers (\{{DagUtils}}, \{{TezSessionState}}), MR
AM / tasks (\{{ExecDriver}}) and the WebHCat controller
(\{{TempletonControllerJob}}). The MR side of that injection lives in
\{{ExecDriver.execute()}}, but three code paths submit MR jobs through
\{{JobClient}} directly and never go through \{{ExecDriver}}:
|| Path || Trigger || | \{{ql/io/merge/MergeFileTask}} | \{{ALTER TABLE ...
CONCATENATE}} with \{{hive.execution.engine=mr}} | |
\{{ql/io/rcfile/truncate/ColumnTruncateTask}} | \{{TRUNCATE TABLE ... COLUMNS
(...)}} on RCFile tables | | \{{ql/txn/compactor/MRCompactor}} | ACID
compaction launched by the metastore \{{Worker}}; still the default for
full-CRUD tables because \{{hive.compactor.crud.query.based}} defaults to
\{{false}} |
None of them touches \{{mapreduce.map.java.opts}},
\{{mapreduce.reduce.java.opts}} or \{{yarn.app.mapreduce.am.command-opts}}, so
on a JDK 17 cluster whose \{{mapred-site.xml}} does not carry the flags the
task JVMs fail on the first reflective access into \{{java.base}}.
_Reproduction_ (JDK 17 on both the submitter and the containers, no
\{{--add-opens}} in \{{mapred-site.xml}}):
{code:sql} SET hive.execution.engine=mr; CREATE TABLE t_orc (id INT, s STRING)
STORED AS ORC; INSERT INTO t_orc VALUES (1,'a'); INSERT INTO t_orc VALUES
(2,'b'); ALTER TABLE t_orc CONCATENATE; \{code}
Task attempt log:
{noformat} Error: java.lang.reflect.InaccessibleObjectException: Unable to make
field private volatile java.lang.String java.net.URI.string accessible: module
java.base does not "opens java.net" to unnamed module at
org.apache.hadoop.hive.common.StringInternUtils. at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init \{noformat}
An \{{INSERT ... SELECT}} on the same cluster succeeds because \{{ExecDriver}}
injects the flags.
{_}Why CI does not catch it{_}: the MiniMR / qtest drivers run tasks in the
local job runner inside the test JVM, where the \{{*.java.opts}} keys are
ignored and the surefire \{{argLine}} already opens the packages. Only a real
YARN submission exercises the container command line.
{_}Proposed fix{_}: move the injection into a shared helper and call it from
every MR submission path.
{code:java} // org.apache.hadoop.hive.common.JavaVersionUtils public static
void addOpensFlags(Configuration job) \{ String addOpens = getAddOpensFlags();
for (String key : new String[] {"mapreduce.map.java.opts",
"mapreduce.reduce.java.opts", "yarn.app.mapreduce.am.command-opts"}) \{ String
current = job.get(key); job.set(key, (current == null ? "" : current) +
addOpens); } } \{code}
* {\{ExecDriver}} calls the helper instead of its private \{{addOpensFlags}} /
\{{addJavaOpts}}.
* {\{MergeFileTask.execute}}, \{{ColumnTruncateTask.execute}} and
\{{MRCompactor.launchCompactionJob}} call it right before \{{new
JobClient(job)}}.
* Append semantics are kept, so cluster / user \{{java.opts}} such as heap
sizes survive. String-literal keys keep \{{hive-common}} free of a
\{{hadoop-mapreduce-client-core}} dependency, as \{{ExecDriver}} already does
today.
* Unit test in \{{TestJavaVersionUtils}}: an existing \{{-Xmx}} value is
preserved and all three keys end with the flag set.
Not in scope: \{{MapredLocalTask}} (map-join local task) inherits
\{{HADOOP_CLIENT_OPTS}} from the parent process, which
\{{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
{_}Verification{_}: an equivalent patch on a 3.1.x-based build was run on a
3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
cluster-side \{{--add-opens}} removed). Before the patch \{{ALTER TABLE ...
CONCATENATE}} failed as above; after it the AM and task
\{{launch_container.sh}} of the merge job, the RCFile column-truncate job and
the metastore-launched major compaction all carry the flag set and the jobs
succeed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
