[
https://issues.apache.org/jira/browse/HIVE-30011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shml updated HIVE-30011:
------------------------
Description:
HIVE-28869 makes Hive append the JDK 17 {{--add-opens}} flags to the JVMs it
launches itself: Tez AM / containers ({{DagUtils}}, {{TezSessionState}}), MR AM
/ tasks ({{ExecDriver}}) and the WebHCat controller
({{TempletonControllerJob}}). The MR side of that injection lives in
{{ExecDriver.execute()}}, but three code paths submit MR jobs through
{{JobClient}} directly and never go through {{ExecDriver}}:
|| Path || Trigger ||
| {{ql/io/merge/MergeFileTask}} | {{ALTER TABLE ... CONCATENATE}} with
{{hive.execution.engine=mr}} |
| {{ql/io/rcfile/truncate/ColumnTruncateTask}} | {{TRUNCATE TABLE ... COLUMNS
(...)}} on RCFile tables |
| {{ql/txn/compactor/MRCompactor}} | ACID compaction launched by the metastore
{{Worker}}; still the default for full-CRUD tables because
{{hive.compactor.crud.query.based}} defaults to {{false}} |
None of them touches {{mapreduce.map.java.opts}},
{{mapreduce.reduce.java.opts}} or {{yarn.app.mapreduce.am.command-opts}}, so on
a JDK 17 cluster whose {{mapred-site.xml}} does not carry the flags the task
JVMs fail on the first reflective access into {{java.base}}.
*Reproduction* (JDK 17 on both the submitter and the containers, no
{{--add-opens}} in {{mapred-site.xml}}):
{code:sql}
SET hive.execution.engine=mr;
CREATE TABLE t_orc (id INT, s STRING) STORED AS ORC;
INSERT INTO t_orc VALUES (1,'a');
INSERT INTO t_orc VALUES (2,'b');
ALTER TABLE t_orc CONCATENATE;
{code}
Task attempt log:
{noformat}
Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
private volatile java.lang.String java.net.URI.string accessible:
module java.base does not "opens java.net" to unnamed module
at org.apache.hadoop.hive.common.StringInternUtils.<clinit>
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init
{noformat}
An {{INSERT ... SELECT}} on the same cluster succeeds because {{ExecDriver}}
injects the flags.
*Why CI does not catch it*: the MiniMR / qtest drivers run tasks in the local
job runner inside the test JVM, where the {{*.java.opts}} keys are ignored and
the surefire {{argLine}} already opens the packages. Only a real YARN
submission exercises the container command line.
*Proposed fix*: move the injection into a shared helper and call it from every
MR submission path.
{code:java}
// org.apache.hadoop.hive.common.JavaVersionUtils
public static void addOpensFlags(Configuration job) {
String addOpens = getAddOpensFlags();
for (String key : new String[] {"mapreduce.map.java.opts",
"mapreduce.reduce.java.opts", "yarn.app.mapreduce.am.command-opts"}) {
String current = job.get(key);
job.set(key, (current == null ? "" : current) + addOpens);
}
}
{code}
* {{ExecDriver}} calls the helper instead of its private {{addOpensFlags}} /
{{addJavaOpts}}.
* {{MergeFileTask.execute}}, {{ColumnTruncateTask.execute}} and
{{MRCompactor.launchCompactionJob}} call it right before {{new JobClient(job)}}.
* Append semantics are kept, so cluster / user {{java.opts}} such as heap sizes
survive. String-literal keys keep {{hive-common}} free of a
{{hadoop-mapreduce-client-core}} dependency, as {{ExecDriver}} already does
today.
* Unit test in {{TestJavaVersionUtils}}: an existing {{-Xmx}} value is
preserved and all three keys end with the flag set.
Not in scope: {{MapredLocalTask}} (map-join local task) inherits
{{HADOOP_CLIENT_OPTS}} from the parent process, which
{{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
*Verification*: an equivalent patch on a 3.1.x-based build was run on a 3-node
YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
cluster-side {{--add-opens}} removed). Before the patch {{ALTER TABLE ...
CONCATENATE}} failed as above; after it the AM and task {{launch_container.sh}}
of the merge job, the RCFile column-truncate job and the metastore-launched
major compaction all carry the flag set and the jobs succeed.
was:
HIVE-28869 makes Hive append the JDK 17 {{--add-opens}} flags to the JVMs it
launches itself: Tez AM / containers ({{{}DagUtils{}}},
{{{}TezSessionState{}}}), MR AM / tasks ({{{}ExecDriver{}}}) and the WebHCat
controller ({{{}TempletonControllerJob{}}}). The MR side of that injection
lives in {{{}ExecDriver.execute(){}}}, but three code paths submit MR jobs
through {{JobClient}} directly and never go through {{{}ExecDriver{}}}:
|| Path || Trigger || | {{ql/io/merge/MergeFileTask}} | {{ALTER TABLE ...
CONCATENATE}} with {{hive.execution.engine=mr}} | |
{{ql/io/rcfile/truncate/ColumnTruncateTask}} | {{TRUNCATE TABLE ... COLUMNS
(...)}} on RCFile tables | | {{ql/txn/compactor/MRCompactor}} | ACID compaction
launched by the metastore {{{}Worker{}}}; still the default for full-CRUD
tables because {{hive.compactor.crud.query.based}} defaults to {{false}} |
None of them touches {{{}mapreduce.map.java.opts{}}},
{{mapreduce.reduce.java.opts}} or {{{}yarn.app.mapreduce.am.command-opts{}}},
so on a JDK 17 cluster whose {{mapred-site.xml}} does not carry the flags the
task JVMs fail on the first reflective access into {{{}java.base{}}}.
_Reproduction_ (JDK 17 on both the submitter and the containers, no
{{--add-opens}} in {{{}mapred-site.xml{}}}):
{code:sql}
{code}
SET hive.execution.engine=mr; CREATE TABLE t_orc (id INT, s STRING) STORED AS
ORC; INSERT INTO t_orc VALUES (1,'a'); INSERT INTO t_orc VALUES (2,'b'); ALTER
TABLE t_orc CONCATENATE; \{code}
Task attempt log:
{noformat}
{noformat}
Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
private volatile java.lang.String java.net.URI.string accessible: module
java.base does not "opens java.net" to unnamed module at
org.apache.hadoop.hive.common.StringInternUtils. at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init \{noformat}
An {{INSERT ... SELECT}} on the same cluster succeeds because {{ExecDriver}}
injects the flags.
{_}Why CI does not catch it{_}: the MiniMR / qtest drivers run tasks in the
local job runner inside the test JVM, where the {{*.java.opts}} keys are
ignored and the surefire {{argLine}} already opens the packages. Only a real
YARN submission exercises the container command line.
{_}Proposed fix{_}: move the injection into a shared helper and call it from
every MR submission path.
{code:java}
{code}
// org.apache.hadoop.hive.common.JavaVersionUtils public static void
addOpensFlags(Configuration job) { String addOpens = getAddOpensFlags(); for
(String key : new String[]
{"mapreduce.map.java.opts", "mapreduce.reduce.java.opts",
"yarn.app.mapreduce.am.command-opts"}
) \{ String current = job.get(key); job.set(key, (current == null ? "" :
current) + addOpens); } } \{code}
* {{ExecDriver}} calls the helper instead of its private {{addOpensFlags}} /
{{{}addJavaOpts{}}}.
* {{{}MergeFileTask.execute{}}}, {{ColumnTruncateTask.execute}} and
{{MRCompactor.launchCompactionJob}} call it right before {{{}new
JobClient(job){}}}.
* Append semantics are kept, so cluster / user {{java.opts}} such as heap
sizes survive. String-literal keys keep {{hive-common}} free of a
{{hadoop-mapreduce-client-core}} dependency, as {{ExecDriver}} already does
today.
* Unit test in {{{}TestJavaVersionUtils{}}}: an existing {{-Xmx}} value is
preserved and all three keys end with the flag set.
Not in scope: {{MapredLocalTask}} (map-join local task) inherits
{{HADOOP_CLIENT_OPTS}} from the parent process, which
{{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
{_}Verification{_}: an equivalent patch on a 3.1.x-based build was run on a
3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
cluster-side {{--add-opens}} removed). Before the patch {{ALTER TABLE ...
CONCATENATE}} failed as above; after it the AM and task {{launch_container.sh}}
of the merge job, the RCFile column-truncate job and the metastore-launched
major compaction all carry the flag set and the jobs succeed.
> MR jobs submitted outside ExecDriver (MergeFileTask, ColumnTruncateTask,
> MRCompactor) miss the JDK 17 --add-opens flags
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30011
> URL: https://issues.apache.org/jira/browse/HIVE-30011
> Project: Hive
> Issue Type: Bug
> Components: Query Processor, Transactions
> Affects Versions: 4.1.0
> Reporter: shml
> Priority: Major
>
> HIVE-28869 makes Hive append the JDK 17 {{--add-opens}} flags to the JVMs it
> launches itself: Tez AM / containers ({{DagUtils}}, {{TezSessionState}}), MR
> AM / tasks ({{ExecDriver}}) and the WebHCat controller
> ({{TempletonControllerJob}}). The MR side of that injection lives in
> {{ExecDriver.execute()}}, but three code paths submit MR jobs through
> {{JobClient}} directly and never go through {{ExecDriver}}:
> || Path || Trigger ||
> | {{ql/io/merge/MergeFileTask}} | {{ALTER TABLE ... CONCATENATE}} with
> {{hive.execution.engine=mr}} |
> | {{ql/io/rcfile/truncate/ColumnTruncateTask}} | {{TRUNCATE TABLE ... COLUMNS
> (...)}} on RCFile tables |
> | {{ql/txn/compactor/MRCompactor}} | ACID compaction launched by the
> metastore {{Worker}}; still the default for full-CRUD tables because
> {{hive.compactor.crud.query.based}} defaults to {{false}} |
> None of them touches {{mapreduce.map.java.opts}},
> {{mapreduce.reduce.java.opts}} or {{yarn.app.mapreduce.am.command-opts}}, so
> on a JDK 17 cluster whose {{mapred-site.xml}} does not carry the flags the
> task JVMs fail on the first reflective access into {{java.base}}.
> *Reproduction* (JDK 17 on both the submitter and the containers, no
> {{--add-opens}} in {{mapred-site.xml}}):
> {code:sql}
> SET hive.execution.engine=mr;
> CREATE TABLE t_orc (id INT, s STRING) STORED AS ORC;
> INSERT INTO t_orc VALUES (1,'a');
> INSERT INTO t_orc VALUES (2,'b');
> ALTER TABLE t_orc CONCATENATE;
> {code}
> Task attempt log:
> {noformat}
> Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
> private volatile java.lang.String java.net.URI.string accessible:
> module java.base does not "opens java.net" to unnamed module
> at org.apache.hadoop.hive.common.StringInternUtils.<clinit>
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init
> {noformat}
> An {{INSERT ... SELECT}} on the same cluster succeeds because {{ExecDriver}}
> injects the flags.
> *Why CI does not catch it*: the MiniMR / qtest drivers run tasks in the local
> job runner inside the test JVM, where the {{*.java.opts}} keys are ignored
> and the surefire {{argLine}} already opens the packages. Only a real YARN
> submission exercises the container command line.
> *Proposed fix*: move the injection into a shared helper and call it from
> every MR submission path.
> {code:java}
> // org.apache.hadoop.hive.common.JavaVersionUtils
> public static void addOpensFlags(Configuration job) {
> String addOpens = getAddOpensFlags();
> for (String key : new String[] {"mapreduce.map.java.opts",
> "mapreduce.reduce.java.opts", "yarn.app.mapreduce.am.command-opts"}) {
> String current = job.get(key);
> job.set(key, (current == null ? "" : current) + addOpens);
> }
> }
> {code}
> * {{ExecDriver}} calls the helper instead of its private {{addOpensFlags}} /
> {{addJavaOpts}}.
> * {{MergeFileTask.execute}}, {{ColumnTruncateTask.execute}} and
> {{MRCompactor.launchCompactionJob}} call it right before {{new
> JobClient(job)}}.
> * Append semantics are kept, so cluster / user {{java.opts}} such as heap
> sizes survive. String-literal keys keep {{hive-common}} free of a
> {{hadoop-mapreduce-client-core}} dependency, as {{ExecDriver}} already does
> today.
> * Unit test in {{TestJavaVersionUtils}}: an existing {{-Xmx}} value is
> preserved and all three keys end with the flag set.
> Not in scope: {{MapredLocalTask}} (map-join local task) inherits
> {{HADOOP_CLIENT_OPTS}} from the parent process, which
> {{bin/ext/hiveserver2.sh}} already exports since HIVE-26473.
> *Verification*: an equivalent patch on a 3.1.x-based build was run on a
> 3-node YARN cluster (Hadoop 3.4.3, JDK 17.0.20 on submitter and containers,
> cluster-side {{--add-opens}} removed). Before the patch {{ALTER TABLE ...
> CONCATENATE}} failed as above; after it the AM and task
> {{launch_container.sh}} of the merge job, the RCFile column-truncate job and
> the metastore-launched major compaction all carry the flag set and the jobs
> succeed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
