> On March 30, 2017, 8:41 p.m., Aihua Xu wrote: > > common/src/java/org/apache/hadoop/hive/common/LogUtils.java > > Lines 52 (patched) > > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52> > > > > Any suggestions? > > > > Seems that is what it does. Can't think of anything else. > > Peter Vary wrote: > // Constants of the key strings for the log4j MDC. > > Maybe? > > Aihua Xu wrote: > They are the same thing. Actually Log4J uses ThreadContext. See > https://logging.apache.org/log4j/2.x/manual/thread-context.html.
Thanks for the clarification! > On March 30, 2017, 8:41 p.m., Aihua Xu wrote: > > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > > Lines 1376-1380 (original) > > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376> > > > > We were using OperationLog incorrectly before. This log should not be > > for direct write, but we should be able to write to it through > > LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to > > acquire compile lock: " + command); above. > > > > With this patch, it will write this message at DEBUG level. > > > > The loglevels in operationLog (execution, verbose) are actually to > > control which classes are allowed to output logs. It's very confusion > > though. > > Peter Vary wrote: > When we used OperationLog for direct write, we might thought that there > are some messages we would like to show the user regardless of the log > settings. Like the MR deprecation warning. I am not saying that it was good, > merely stating that it was the case, and we will not be able to archive this > after this change :D > > > I was thinking about HiveCLI/BeeLine compatibility. > HiveCLI uses SessionState._console, to print out information to the user. > There are some cases when we use _console.getInfoStream() to write directly > to the HiveCLI console. Also there are some places where we use > SessionState.out to print a message to the HiveCLI user. I am not sure how > can print these messages to the BeeLine user, when we do not want to show the > other log messages of the given class to him. > > Aihua Xu wrote: > If you want to print any message, you should still use LOG.info() which > will print to console, file, etc which are configured. We are using those > improperly, probably because we only had HiveCLI originally but later we add > HiveServer2. So consoles in 2 HiveCLI and HiveServer2 mean different things. > I will have followup jiras to 1) make sure the logs are logged at the > right level, 2) cleanup the console output of HiveCLI and Beeline. > > Regarding some messages like "MR deprecation warning", I think it still > makes sense not to show if the user disable it (like run in silense). It's > just a warning like other warnings. I agree with you, I just wanted to make sure, that we do it intentionally. :) - Peter ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58085/#review170638 ----------------------------------------------------------- On March 30, 2017, 9:01 p.m., Aihua Xu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58085/ > ----------------------------------------------------------- > > (Updated March 30, 2017, 9:01 p.m.) > > > Review request for hive. > > > Bugs: hive-16061 > https://issues.apache.org/jira/browse/hive-16061 > > > Repository: hive-git > > > Description > ------- > > HIVE-16061: Some of console output is not printed to the beeline console > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java > e344e0f > ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119 > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97 > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java > 08d0544 > ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2 > ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out > 993329e > ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53 > > service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java > 8f08c2e > > service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java > eaf1acb > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 11a820f > > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java > 3f8f68e > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > f41092e > > > Diff: https://reviews.apache.org/r/58085/diff/2/ > > > Testing > ------- > > Test is done locally. You can see the beeline output now. > > 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on > t1.key=t2.key limit 10; > INFO : Compiling > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): > select t1.key from src t1 join src t2 on t1.key=t2.key limit 10 > INFO : Semantic Analysis Completed > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, > type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); > Time taken: 0.219 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): > select t1.key from src t1 join src t2 on t1.key=t2.key limit 10 > WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the > future versions. Consider using a different execution engine (i.e. spark, > tez) or using Hive 1.X releases. > INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961 > INFO : Total jobs = 1 > INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode > INFO : Execution completed successfully > INFO : MapredLocal task succeeded > INFO : Launching Job 1 out of 1 > INFO : Starting task [Stage-3:MAPRED] in serial mode > INFO : Number of reduce tasks is set to 0 since there's no reduce operator > INFO : Starting Job = job_local1894165710_0002, Tracking URL = > http://localhost:8080/ > INFO : Kill Command = > /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job > -kill job_local1894165710_0002 > INFO : Hadoop job information for Stage-3: number of mappers: 0; number of > reducers: 0 > INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0% > ERROR : Ended Job = job_local1894165710_0002 with errors > ERROR : FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > INFO : MapReduce Jobs Launched: > INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL > INFO : Total MapReduce CPU Time Spent: 0 msec > > > Thanks, > > Aihua Xu > >
