I saw bunch of failures related to disk space issue https://issues.apache.org/jira/browse/HIVE-11416?focusedCommentId=14650616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14650616 http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4792/console
Some of then throws IOException failing to create directories. https://issues.apache.org/jira/browse/HIVE-11304?focusedCommentId=14651250&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14651250 https://issues.apache.org/jira/browse/HIVE-11405?focusedCommentId=14654006&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14654006 Some are 404 not found issue http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4793/console The patches that I submitted recently had multiple runs and many attempts failed with one of the above reasons. I initially thought HIVE-11304 patches are generating huge log files because of log4j2 migration but then later found out that even trunk is generating huge log files (Sergio also found the same). Commits before HIVE-10166 seems to be generating smaller files in my test runs as well but I don’t have the exact number. I didn’t spend much time on debugging the issue but running TestJDBCWithMiniHS2 in trunk seems to emitting way too many thrift warnings than before. > On Aug 5, 2015, at 11:09 AM, Xuefu Zhang <xzh...@cloudera.com> wrote: > > Prasanth, > > Could you send me a URL that shows the warning msg that you were seeing? > I'm not sure if it's related to the thrift version that were used to > regenerate the code. > > Thanks, > Xuefu > > On Wed, Aug 5, 2015 at 10:54 AM, Prasanth Jayachandran < > pjayachand...@hortonworks.com> wrote: > >> Thanks Sergio for creating a follow up JIRA! >> >>> On Aug 5, 2015, at 10:49 AM, Sergio Pena <sergio.p...@cloudera.com> >> wrote: >>> >>> Thanks Prasanth. >>> >>> I run the same test you did, and I found the following sizes: >>> >>> BEFORE HIVE-10166 >>> *13M* Aug 5 11:57 ./hive-unit/target/tmp/log/hive.log >>> >>> WITH HIVE-10166 >>> *2.4G* Aug 5 12:07 ./hive-unit/target/tmp/log/hive.log >>> >>> CURRENT HEAD >>> *3.2G* Aug 5 12:36 ./hive-unit/target/tmp/log/hive.log >>> >>> HIVE-10166 is adding more size to the file. But, there are other commits >>> that are adding more values to it that we should investigate. >>> >>> I created a JIRA to track this issue. >>> https://issues.apache.org/jira/browse/HIVE-11466 >>> >>> >>> On Mon, Aug 3, 2015 at 3:29 PM, Prasanth Jayachandran < >>> pjayachand...@hortonworks.com> wrote: >>> >>>> Hi Sergio >>>> >>>> This seems to be related to recent merge in HIVE-10166. I checked out a >>>> commit prior to HIVE-10166 and ran TestJdbcWithMiniHS2 and the log seems >>>> reasonable. >>>> After HIVE-10166, the log file is getting too many WARN msgs (this log >> msg >>>> repeats and fills up the disk) >>>> >>>> org.apache.thrift.transport.TTransportException: No underlying server >>>> socket. >>>> at >>>> >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:126) >>>> at >>>> >> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) >>>> at >>>> >> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60) >>>> at >>>> >> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161) >>>> at >>>> >> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:100) >>>> >>>> Can someone take a look? >>>> >>>> Thanks >>>> Prasanth >>>> >>>>> On Aug 2, 2015, at 9:29 PM, Prasanth Jayachandran < >>>> pjayachand...@hortonworks.com> wrote: >>>>> >>>>> Hi Sergio >>>>> >>>>> Thanks for looking into this. It could be related to my patch >> HIVE-11304 >>>> (Log4j2 migration). I might have mistakenly specified log4j2 threshold >> to >>>> ALL level somewhere resulting in DEBUG level logging. I will look into >> it. >>>>> >>>>> Thanks >>>>> Prasanth >>>>> >>>>>> On Aug 2, 2015, at 9:21 PM, Sergio Pena <sergio.p...@cloudera.com> >>>> wrote: >>>>>> >>>>>> Hi Prasanth, >>>>>> >>>>>> I see there are some logs in the system that are too big, and using >> many >>>>>> space. Jenkins will delete those logs eventually. >>>>>> These are some of the logs bigger than 1G that I found: >>>>>> >>>>>> *13G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4789/succeeded/TestJdbcWithMiniHS2/hive.log* >>>>>> *9.9G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4790/succeeded/TestJdbcWithMiniHS2/hive.log >>>>>> <<< HIVE-11416* >>>>>> *5.5G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4790/succeeded/TestSchedulerQueue/hive.log* >>>>>> *4.9G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4789/succeeded/TestSchedulerQueue/hive.log* >>>>>> *4.6G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4792/succeeded/TestSchedulerQueue/hive.log* >>>>>> *4.1G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-Upload-10/succeeded/TestSchedulerQueue/hive.log* >>>>>> 2.0G ./logs/PreCommit-HIVE-TRUNK-Build-4792/succeeded/TestSSL/hive.log >>>>>> 1.9G ./logs/PreCommit-HIVE-TRUNK-Build-4790/failed/TestSSL/hive.log >>>>>> 1.8G ./logs/PreCommit-HIVE-TRUNK-Build-4789/succeeded/TestSSL/hive.log >>>>>> 1.8G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-Upload-10/succeeded/TestJdbcWithMiniHS2/hive.log >>>>>> 1.7G >>>>>> >>>> >> ./logs/HIVE-TRUNK-HADOOP-2-1/succeeded/TestSparkCliDriver-date_udf.q-join23.q-auto_join4.q-and-12-more/spark.log >>>>>> 1.7G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4789/succeeded/TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more/spark.log >>>>>> 1.7G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4790/succeeded/TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more/spark.log >>>>>> 1.7G >>>>>> >>>> >> ./logs/PreCommit-HIVE-TRUNK-Build-4792/succeeded/TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more/spark.log >>>>>> >>>>>> *TestJdbcWithMiniHS2* is one causing this issue. Is debug enabled on >>>> this >>>>>> log? >>>>>> >>>>>> - Sergio >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Aug 2, 2015 at 7:01 PM, Prasanth Jayachandran < >>>>>> pjayachand...@hortonworks.com> wrote: >>>>>> >>>>>>> Looks like there is something wrong with the precommit tests. >>>>>>> The tests runs through but throws IOException or runs out of disk. >>>>>>> https://issues.apache.org/jira/browse/HIVE-11416 >>>>>>> https://issues.apache.org/jira/browse/HIVE-11304 >>>>>>> >>>>>>> Can someone take a look whats going on? >>>>>>> >>>>>>> Thanks >>>>>>> Prasanth >>>>>>> >>>>> >>>>> >>>> >>>> >> >>
