[
https://issues.apache.org/jira/browse/HIVE-24730?focusedWorklogId=548350&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-548350
]
ASF GitHub Bot logged work on HIVE-24730:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Feb/21 10:13
Start Date: 05/Feb/21 10:13
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570026024
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is
set, otherwise there
+ // is a good chance that the user of the minicluster set another value
intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key,
int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key,
defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
Would it make sense to LOG even if we do not override?
It took me several attempts to understand your comments in the description
(I have kept looking for 256 in the logs), so either it is too early morning
for me or some additional log messages would be good 😄
If I understand correctly this is test code only change so it would not be a
problem if we add more log.
Your thoughts? And BTW thanks for the patch!
Peter
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
was this intentional?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 548350)
Time Spent: 1.5h (was: 1h 20m)
> Shims classes override values from hive-site.xml and tez-site.xml silently
> --------------------------------------------------------------------------
>
> Key: HIVE-24730
> URL: https://issues.apache.org/jira/browse/HIVE-24730
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Since HIVE-14887,
> [Hadoop23Shims|https://github.com/apache/hive/blob/master/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java]
> silently overrides e.g. hive.tez.container.size which is defined in
> data/conf/hive/llap/hive-site.xml. This way, the developer will have no idea
> about what happened after setting those values in the xml.
> My proposal:
> 1. don't set those values, unless they contain the default value (e.g.: -1
> for hive.tez.container.size)
> 2. put an INFO level log message about the override
> OR:
> put a comment in hive-site.xml and tez-site.xml files that shims override it
> while creating a tez mini cluster
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
