[
https://issues.apache.org/jira/browse/HIVE-26984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683368#comment-17683368
]
László Bodor edited comment on HIVE-26984 at 2/2/23 11:54 AM:
--------------------------------------------------------------
okay, I think I understand your point
regarding the comfort and probability of conf issues (how often would I need
this): yeah, I'm indeed after comfort: just as we speak I can tell 2 different
code paths where some hive props behave unexpectedly causing a huge headache
for me (one is a tez vertex manager setting, another is "tez.queue.name"
cleared somewhere in the code...okay we already found maybe why the latter was,
but the story is the same: it caused a headache
I've already faced a similar problem while investigating the absence of hbase
related properties somewhere in the code, so 3 different issues from the last
couple of months, instantly
chicken-egg problem: you're right, it's possible theoretically if you change
the property to *true* on the fly...so worst case, we might end up setting this
property on cluster level (brute force, involves hiveserver2 restart) it's not
optimal, I know, let me think about this: what about trying this out on a
cluster, and putting a temporary log message to the original set HiveConf
methods: if I can see the usage of original HiveConf methods after setting
this prop to *true*, the chicken-egg problem just happened, and I can
investigate it, is it acceptable in case of this solution?
"when someone just wrote `new HiveConf()`..." <<- that's what @Deprecated is
for, sometimes it's a common need to hide object creation details
" as about passing&launching the agent: I'm not sure - but maybe the agent can
be placed inside say hive-exec or something; and then tweak the tez launch
params (from inside HS2) to add the -agent to the launch cmdline.... ; probably
similar for the HS2 startup...but that should be done somewhere in the
scripts..." <<- with all due respect, as far as I understand, here you're still
trying to convince me this is better than using a single hive setting (again,
comfort, I'm lazy I know)
so answering the question of whether I explored these paths? not yet, and don't
want to seem ignorant regarding your idea of the agents, let me think about it
also
was (Author: abstractdog):
okay, I think I understand your point
regarding the comfort and probability of conf issues (how often would I need
this): yeah, I'm indeed after comfort: just as we speak I can tell 2 different
code paths where some hive props behave unexpectedly causing a huge headache
for me (one is a tez vertex manager setting, another is "tez.queue.name"
cleared somewhere in the code...okay we already found maybe why the latter was,
but the story is the same: it caused a headache
I've already faced a similar problem while investigating the absence of hbase
related properties somewhere in the code, so 3 different issues from the last
couple of months, instantly
chicken-egg problem: you're right, it's possible theoretically if you change
the property to *true* on the fly...so worst case, we might end up setting this
property on cluster level (brute force, involves hiveserver2 restart) it's not
optimal, I know, let me think about this: what about trying this out on a
cluster, and putting a temporary log message to the original set HiveConf
methods: if I can see the usage of original HiveConf methods after setting
this prop to *true*, the chicken-egg problem just happened, and I can
investigate it, is it acceptable in case of this solution?
"when someone just wrote `new HiveConf()`..." <-- that's what @Deprecated is
for, sometimes it's a common need to hide object creation details
" as about passing&launching the agent: I'm not sure - but maybe the agent can
be placed inside say hive-exec or something; and then tweak the tez launch
params (from inside HS2) to add the -agent to the launch cmdline.... ; probably
similar for the HS2 startup...but that should be done somewhere in the
scripts..." <-- with all due respect, as far as I understand, here you're still
trying to convince me this is better than using a single hive setting (again,
comfort, I'm lazy I know)
so answering the question of whether I explored these paths? not yet, and don't
want to seem ignorant regarding your idea of the agents, let me think about it
also
> Deprecate public HiveConf constructors
> --------------------------------------
>
> Key: HIVE-26984
> URL: https://issues.apache.org/jira/browse/HIVE-26984
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> From time to time we investigate configuration object problems that are hard
> to investigate. We can improve this area, e.g. with HIVE-26985, but first, we
> need to introduce a public static factory method to hook into the creation
> process. I can see this pattern in another projects as well, like:
> HBaseConfiguration.
> Creating custom HiveConf subclasses can be useful because putting optional
> (say: if else branches or whatever) stuff into the original HiveConf object's
> hot codepaths can turn it less performant instantly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
