[
https://issues.apache.org/jira/browse/HIVE-30011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shml updated HIVE-30011:
------------------------
Description:
HIVE-28869 makes Hive append the JDK 17 {{--add-opens}} flags to the JVMs it
launches itself: Tez AM / containers ({{{}DagUtils{}}},
{{{}TezSessionState{}}}), MR AM / tasks ({{{}ExecDriver{}}}) and the WebHCat
controller ({{{}TempletonControllerJob{}}}). The MR side of that injection
lives in {{{}ExecDriver.execute(){}}}, but three code paths submit MR jobs
through {{JobClient}} directly and never go through {{{}ExecDriver{}}}:
|| Path || Trigger || | {{ql/io/merge/MergeFileTask}} | {{ALTER TABLE ...
CONCATENATE}} with {{hive.execution.engine=mr}} | |
{{ql/io/rcfile/truncate/ColumnTruncateTask}} | {{TRUNCATE TABLE ... COLUMNS
(...)}} on RCFile tables | | {{ql/txn/compactor/MRCompactor}} | ACID compaction
launched by the metastore {{{}Worker{}}}; still the default for full-CRUD
tables because {{hive.compactor.crud.query.based}} defaults to {{false}} |
None of them touches {{{}mapreduce.map.java.opts{}}},
{{mapreduce.reduce.java.opts}} or {{{}yarn.app.mapreduce.am.command-opts{}}},
so on a JDK 17 cluster whose {{mapred-site.xml}} does not carry the flags the
task JVMs fail on the first reflective access into {{{}java.base{}}}.
_Reproduction_ (JDK 17 on both the submitter and the containers, no
{{--add-opens}} in {{{}mapred-site.xml{}}}):
{code:sql}
{code}
SET hive.execution.engine=mr; CREATE TABLE t_orc (id INT, s STRING) STORED AS
ORC; INSERT INTO t_orc VALUES (1,'a'); INSERT INTO t_orc VALUES (2,'b'); ALTER
TABLE t_orc CONCATENATE; \{code}
Task attempt log:
{noformat}
{noformat}
Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
private volatile java.lang.String java.net.URI.string accessible: module
java.base does not "opens java.net" to unnamed module at
org.apache.hadoop.hive.common.StringInternUtils. at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init \{noformat}
An {{INSERT ... SELECT}} on the same cluster succeeds because {{ExecDriver}}
injects the flags.
{_}Why CI does not catch it{_}: the MiniMR / qtest drivers run tasks in the
local job runner inside the test JVM, where the {{*.java.opts}} keys are
ignored and the surefire {{argLine}} already opens the packages. Only a real
YARN submission exercises the container command line.
{_}Proposed fix{_}: move the injection into a shared helper and call it from
every MR submission path.
{code:java}
{code}
// org.apache.hadoop.hive.common.JavaVersionUtils public static void
addOpensFlags(Configuration job) { String addOpens = getAddOpensFlags(); for
(String key : new String[]
{"mapreduce.map.java.opts", "mapreduce.reduce.java.opts",
"yarn.app.mapreduce.am.command-opts"}
) \{ String current = job.get(key); job.set(key, (current == null ? "" :
current) + addOpens); } } \{code}
* {{ExecDriver}} calls the helper instead of its private {{addOpensFlags}} /
{{{}addJavaOpts{}}}.
* {{{}MergeFileTask.execute{}}}, {{ColumnTruncateTask.execute}} and
{{MRCompactor.launchCompactionJob}} call it right before {{{}new
JobClient(job){}}}.
* Append semantics are kept, so cluster / user {{java.opts}} such as heap
sizes survive. String-literal keys keep {{hive-common}} free of a
{{hadoop-mapreduce-client-core}} dependency, as {{ExecDriver}} already does
today.
* Unit test in {{{}TestJavaVersionUtils{}}}: an existing {{-Xmx}} value is
preserved and all three keys end with the flag set.
Not in scope: {{MapredLocalTask}} (map-join local task) inherits
{{HADOOP_CLIENT_OPTS}} from the parent process, which
{{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
{_}Verification{_}: an equivalent patch on a 3.1.x-based build was run on a
3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
cluster-side {{--add-opens}} removed). Before the patch {{ALTER TABLE ...
CONCATENATE}} failed as above; after it the AM and task {{launch_container.sh}}
of the merge job, the RCFile column-truncate job and the metastore-launched
major compaction all carry the flag set and the jobs succeed.
was:
HIVE-28869 makes Hive append the JDK 17 \{{--add-opens}} flags to the JVMs it
launches itself: Tez AM / containers (\{{DagUtils}}, \{{TezSessionState}}), MR
AM / tasks (\{{ExecDriver}}) and the WebHCat controller
(\{{TempletonControllerJob}}). The MR side of that injection lives in
\{{ExecDriver.execute()}}, but three code paths submit MR jobs through
\{{JobClient}} directly and never go through \{{ExecDriver}}:
|| Path || Trigger || | \{{ql/io/merge/MergeFileTask}} | \{{ALTER TABLE ...
CONCATENATE}} with \{{hive.execution.engine=mr}} | |
\{{ql/io/rcfile/truncate/ColumnTruncateTask}} | \{{TRUNCATE TABLE ... COLUMNS
(...)}} on RCFile tables | | \{{ql/txn/compactor/MRCompactor}} | ACID
compaction launched by the metastore \{{Worker}}; still the default for
full-CRUD tables because \{{hive.compactor.crud.query.based}} defaults to
\{{false}} |
None of them touches \{{mapreduce.map.java.opts}},
\{{mapreduce.reduce.java.opts}} or \{{yarn.app.mapreduce.am.command-opts}}, so
on a JDK 17 cluster whose \{{mapred-site.xml}} does not carry the flags the
task JVMs fail on the first reflective access into \{{java.base}}.
_Reproduction_ (JDK 17 on both the submitter and the containers, no
\{{--add-opens}} in \{{mapred-site.xml}}):
{code:sql} SET hive.execution.engine=mr; CREATE TABLE t_orc (id INT, s STRING)
STORED AS ORC; INSERT INTO t_orc VALUES (1,'a'); INSERT INTO t_orc VALUES
(2,'b'); ALTER TABLE t_orc CONCATENATE; \{code}
Task attempt log:
{noformat} Error: java.lang.reflect.InaccessibleObjectException: Unable to make
field private volatile java.lang.String java.net.URI.string accessible: module
java.base does not "opens java.net" to unnamed module at
org.apache.hadoop.hive.common.StringInternUtils. at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init \{noformat}
An \{{INSERT ... SELECT}} on the same cluster succeeds because \{{ExecDriver}}
injects the flags.
{_}Why CI does not catch it{_}: the MiniMR / qtest drivers run tasks in the
local job runner inside the test JVM, where the \{{*.java.opts}} keys are
ignored and the surefire \{{argLine}} already opens the packages. Only a real
YARN submission exercises the container command line.
{_}Proposed fix{_}: move the injection into a shared helper and call it from
every MR submission path.
{code:java} // org.apache.hadoop.hive.common.JavaVersionUtils public static
void addOpensFlags(Configuration job) \{ String addOpens = getAddOpensFlags();
for (String key : new String[] {"mapreduce.map.java.opts",
"mapreduce.reduce.java.opts", "yarn.app.mapreduce.am.command-opts"}) \{ String
current = job.get(key); job.set(key, (current == null ? "" : current) +
addOpens); } } \{code}
* {\{ExecDriver}} calls the helper instead of its private \{{addOpensFlags}} /
\{{addJavaOpts}}.
* {\{MergeFileTask.execute}}, \{{ColumnTruncateTask.execute}} and
\{{MRCompactor.launchCompactionJob}} call it right before \{{new
JobClient(job)}}.
* Append semantics are kept, so cluster / user \{{java.opts}} such as heap
sizes survive. String-literal keys keep \{{hive-common}} free of a
\{{hadoop-mapreduce-client-core}} dependency, as \{{ExecDriver}} already does
today.
* Unit test in \{{TestJavaVersionUtils}}: an existing \{{-Xmx}} value is
preserved and all three keys end with the flag set.
Not in scope: \{{MapredLocalTask}} (map-join local task) inherits
\{{HADOOP_CLIENT_OPTS}} from the parent process, which
\{{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
{_}Verification{_}: an equivalent patch on a 3.1.x-based build was run on a
3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
cluster-side \{{--add-opens}} removed). Before the patch \{{ALTER TABLE ...
CONCATENATE}} failed as above; after it the AM and task
\{{launch_container.sh}} of the merge job, the RCFile column-truncate job and
the metastore-launched major compaction all carry the flag set and the jobs
succeed.
> MR jobs submitted outside ExecDriver (MergeFileTask, ColumnTruncateTask,
> MRCompactor) miss the JDK 17 --add-opens flags
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30011
> URL: https://issues.apache.org/jira/browse/HIVE-30011
> Project: Hive
> Issue Type: Bug
> Components: Query Processor, Transactions
> Affects Versions: 4.1.0
> Reporter: shml
> Priority: Major
>
> HIVE-28869 makes Hive append the JDK 17 {{--add-opens}} flags to the JVMs it
> launches itself: Tez AM / containers ({{{}DagUtils{}}},
> {{{}TezSessionState{}}}), MR AM / tasks ({{{}ExecDriver{}}}) and the WebHCat
> controller ({{{}TempletonControllerJob{}}}). The MR side of that injection
> lives in {{{}ExecDriver.execute(){}}}, but three code paths submit MR jobs
> through {{JobClient}} directly and never go through {{{}ExecDriver{}}}:
> || Path || Trigger || | {{ql/io/merge/MergeFileTask}} | {{ALTER TABLE ...
> CONCATENATE}} with {{hive.execution.engine=mr}} | |
> {{ql/io/rcfile/truncate/ColumnTruncateTask}} | {{TRUNCATE TABLE ... COLUMNS
> (...)}} on RCFile tables | | {{ql/txn/compactor/MRCompactor}} | ACID
> compaction launched by the metastore {{{}Worker{}}}; still the default for
> full-CRUD tables because {{hive.compactor.crud.query.based}} defaults to
> {{false}} |
> None of them touches {{{}mapreduce.map.java.opts{}}},
> {{mapreduce.reduce.java.opts}} or {{{}yarn.app.mapreduce.am.command-opts{}}},
> so on a JDK 17 cluster whose {{mapred-site.xml}} does not carry the flags the
> task JVMs fail on the first reflective access into {{{}java.base{}}}.
> _Reproduction_ (JDK 17 on both the submitter and the containers, no
> {{--add-opens}} in {{{}mapred-site.xml{}}}):
> {code:sql}
> {code}
> SET hive.execution.engine=mr; CREATE TABLE t_orc (id INT, s STRING) STORED
> AS ORC; INSERT INTO t_orc VALUES (1,'a'); INSERT INTO t_orc VALUES (2,'b');
> ALTER TABLE t_orc CONCATENATE; \{code}
> Task attempt log:
> {noformat}
> {noformat}
> Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
> private volatile java.lang.String java.net.URI.string accessible: module
> java.base does not "opens java.net" to unnamed module at
> org.apache.hadoop.hive.common.StringInternUtils. at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init \{noformat}
> An {{INSERT ... SELECT}} on the same cluster succeeds because {{ExecDriver}}
> injects the flags.
> {_}Why CI does not catch it{_}: the MiniMR / qtest drivers run tasks in the
> local job runner inside the test JVM, where the {{*.java.opts}} keys are
> ignored and the surefire {{argLine}} already opens the packages. Only a real
> YARN submission exercises the container command line.
> {_}Proposed fix{_}: move the injection into a shared helper and call it from
> every MR submission path.
> {code:java}
> {code}
> // org.apache.hadoop.hive.common.JavaVersionUtils public static void
> addOpensFlags(Configuration job) { String addOpens = getAddOpensFlags(); for
> (String key : new String[]
> {"mapreduce.map.java.opts", "mapreduce.reduce.java.opts",
> "yarn.app.mapreduce.am.command-opts"}
> ) \{ String current = job.get(key); job.set(key, (current == null ? "" :
> current) + addOpens); } } \{code}
> * {{ExecDriver}} calls the helper instead of its private {{addOpensFlags}} /
> {{{}addJavaOpts{}}}.
> * {{{}MergeFileTask.execute{}}}, {{ColumnTruncateTask.execute}} and
> {{MRCompactor.launchCompactionJob}} call it right before {{{}new
> JobClient(job){}}}.
> * Append semantics are kept, so cluster / user {{java.opts}} such as heap
> sizes survive. String-literal keys keep {{hive-common}} free of a
> {{hadoop-mapreduce-client-core}} dependency, as {{ExecDriver}} already does
> today.
> * Unit test in {{{}TestJavaVersionUtils{}}}: an existing {{-Xmx}} value is
> preserved and all three keys end with the flag set.
> Not in scope: {{MapredLocalTask}} (map-join local task) inherits
> {{HADOOP_CLIENT_OPTS}} from the parent process, which
> {{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
> {_}Verification{_}: an equivalent patch on a 3.1.x-based build was run on a
> 3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
> cluster-side {{--add-opens}} removed). Before the patch {{ALTER TABLE ...
> CONCATENATE}} failed as above; after it the AM and task
> {{launch_container.sh}} of the merge job, the RCFile column-truncate job and
> the metastore-launched major compaction all carry the flag set and the jobs
> succeed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
