> On March 30, 2017, 8:41 p.m., Aihua Xu wrote: > > common/src/java/org/apache/hadoop/hive/common/LogUtils.java > > Lines 52 (patched) > > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52> > > > > Any suggestions? > > > > Seems that is what it does. Can't think of anything else. > > Peter Vary wrote: > // Constants of the key strings for the log4j MDC. > > Maybe?
They are the same thing. Actually Log4J uses ThreadContext. See https://logging.apache.org/log4j/2.x/manual/thread-context.html. > On March 30, 2017, 8:41 p.m., Aihua Xu wrote: > > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > > Lines 1376-1380 (original) > > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376> > > > > We were using OperationLog incorrectly before. This log should not be > > for direct write, but we should be able to write to it through > > LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to > > acquire compile lock: " + command); above. > > > > With this patch, it will write this message at DEBUG level. > > > > The loglevels in operationLog (execution, verbose) are actually to > > control which classes are allowed to output logs. It's very confusion > > though. > > Peter Vary wrote: > When we used OperationLog for direct write, we might thought that there > are some messages we would like to show the user regardless of the log > settings. Like the MR deprecation warning. I am not saying that it was good, > merely stating that it was the case, and we will not be able to archive this > after this change :D > > > I was thinking about HiveCLI/BeeLine compatibility. > HiveCLI uses SessionState._console, to print out information to the user. > There are some cases when we use _console.getInfoStream() to write directly > to the HiveCLI console. Also there are some places where we use > SessionState.out to print a message to the HiveCLI user. I am not sure how > can print these messages to the BeeLine user, when we do not want to show the > other log messages of the given class to him. If you want to print any message, you should still use LOG.info() which will print to console, file, etc which are configured. We are using those improperly, probably because we only had HiveCLI originally but later we add HiveServer2. So consoles in 2 HiveCLI and HiveServer2 mean different things. I will have followup jiras to 1) make sure the logs are logged at the right level, 2) cleanup the console output of HiveCLI and Beeline. Regarding some messages like "MR deprecation warning", I think it still makes sense not to show if the user disable it (like run in silense). It's just a warning like other warnings. - Aihua ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58085/#review170638 ----------------------------------------------------------- On March 30, 2017, 9:01 p.m., Aihua Xu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58085/ > ----------------------------------------------------------- > > (Updated March 30, 2017, 9:01 p.m.) > > > Review request for hive. > > > Bugs: hive-16061 > https://issues.apache.org/jira/browse/hive-16061 > > > Repository: hive-git > > > Description > ------- > > HIVE-16061: Some of console output is not printed to the beeline console > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java > e344e0f > ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119 > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97 > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java > 08d0544 > ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2 > ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out > 993329e > ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53 > > service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java > 8f08c2e > > service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java > eaf1acb > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 11a820f > > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java > 3f8f68e > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > f41092e > > > Diff: https://reviews.apache.org/r/58085/diff/2/ > > > Testing > ------- > > Test is done locally. You can see the beeline output now. > > 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on > t1.key=t2.key limit 10; > INFO : Compiling > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): > select t1.key from src t1 join src t2 on t1.key=t2.key limit 10 > INFO : Semantic Analysis Completed > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, > type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); > Time taken: 0.219 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): > select t1.key from src t1 join src t2 on t1.key=t2.key limit 10 > WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the > future versions. Consider using a different execution engine (i.e. spark, > tez) or using Hive 1.X releases. > INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961 > INFO : Total jobs = 1 > INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode > INFO : Execution completed successfully > INFO : MapredLocal task succeeded > INFO : Launching Job 1 out of 1 > INFO : Starting task [Stage-3:MAPRED] in serial mode > INFO : Number of reduce tasks is set to 0 since there's no reduce operator > INFO : Starting Job = job_local1894165710_0002, Tracking URL = > http://localhost:8080/ > INFO : Kill Command = > /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job > -kill job_local1894165710_0002 > INFO : Hadoop job information for Stage-3: number of mappers: 0; number of > reducers: 0 > INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0% > ERROR : Ended Job = job_local1894165710_0002 with errors > ERROR : FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > INFO : MapReduce Jobs Launched: > INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL > INFO : Total MapReduce CPU Time Spent: 0 msec > > > Thanks, > > Aihua Xu > >
