> On Nov. 20, 2019, 1:58 a.m., Harish Jaiprakash wrote: > > Thanks for the change. This does solve the memory problem and it looks good > > for me. > > > > We need a follow up JIRA to address why the queue size was 17,000 events. > > Was this hdfs or s3fs? In either case we should have some more > > optimizations like: > > * if there are lot of events, batch the flush to hdfs. > > * if its one event per file mode, increase parallelism since writes are not > > happening in different files. > > > > FYI, the events are lost since it is not written to the hdfs file and DAS > > will not get these events. But that is better than crashing hiveserver2.
Thanks for the review. The hive.hook.proto.base-directory points to an s3a path. - Attila ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71784/#review218709 ----------------------------------------------------------- On Nov. 19, 2019, 3:43 p.m., Attila Magyar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71784/ > ----------------------------------------------------------- > > (Updated Nov. 19, 2019, 3:43 p.m.) > > > Review request for hive, Laszlo Bodor, Harish Jaiprakash, Mustafa Iman, and > Panos Garefalakis. > > > Bugs: HIVE-22514 > https://issues.apache.org/jira/browse/HIVE-22514 > > > Repository: hive-git > > > Description > ------- > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > > Since ScheduledThreadPoolExecutor does not support changing the default > unbounded queue to a bounded one, the queue capacity is checked manually by > the patch. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a7687d59004 > ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java > 8eab54859bf > ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHiveProtoLoggingHook.java > 450a0b544d6 > > > Diff: https://reviews.apache.org/r/71784/diff/1/ > > > Testing > ------- > > unittest > > > Thanks, > > Attila Magyar > >
