> On Oct. 31, 2018, 5:29 p.m., Antal Sinkovits wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java > > Line 62 (original), 83 (patched) > > <https://reviews.apache.org/r/69107/diff/3/?file=2102836#file2102836line83> > > > > Probably these three rows can move to the MemoryInfoLogger class (as > > static method), as from now on its his responsibility. > > > > Something like MemoryInfoLogger.start() or schedule
It is used by the close method as well, so I will probably leave it there itself. > On Oct. 31, 2018, 5:29 p.m., Antal Sinkovits wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java > > Line 574 (original), 572 (patched) > > <https://reviews.apache.org/r/69107/diff/3/?file=2102837#file2102837line574> > > > > Is this correct? In batch processing increasing it with only one? I see that when I go to implementations of reducer.process(batch, 0);, it considers the batch as a single row. And there too they are incrementing runTimeRows++ So that is why it is probably done in such a way to increment row count by 1 itself. - Bharathkrishna ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69107/#review210230 ----------------------------------------------------------- On Oct. 26, 2018, 5:13 p.m., Bharathkrishna Guruvayoor Murali wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69107/ > ----------------------------------------------------------- > > (Updated Oct. 26, 2018, 5:13 p.m.) > > > Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang > Karajgaonkar. > > > Repository: hive-git > > > Description > ------- > > Improve record and memory usage logging in SparkRecordHandler > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java > 88dd12c05ade417aca4cdaece4448d31d4e1d65f > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java > 8880bb604e088755dcfb0bcb39689702fab0cb77 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java > cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java > 20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 > > > Diff: https://reviews.apache.org/r/69107/diff/3/ > > > Testing > ------- > > > Thanks, > > Bharathkrishna Guruvayoor Murali > >
