Hi, Needed info on where I can get detailed job counters for Hive on Tez. Am running this on a HDP cluster with Hive 0.13 and see only the following job counters through Hive Tez in Yarn application logs which I got through( yarn logs -applicationId ...) .
a. Cannot see any ReduceOperator counters and also only DESERIALIZE_ERRORS is the only counter present in MapOperator b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS accurate c. What does COMMITTED_HEAP_BYTES indicate? d. Is there any other place I should be checking the counters? [[File System Counters FILE: BYTES_READ=512, FILE: BYTES_WRITTEN=3079881, FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS: BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS: LARGE_READ_OPS=0, HDFS: WRITE_OPS=0] [org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543, GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*, PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248, COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543, OUTPUT_RECORDS=222543, OUTPUT_BYTES=23543896, OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0] [*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter DESERIALIZE_ERRORS=0]] Thanks Suma
