Hi, You are probably right, I am probably just desensitized to the flakiness of Hive's testing methods. Generally I think the following issues contribute:
1) Hive unit testing framework is just plain flaky. There are many 1/10,000 times this fails items and thousands of tests. 2) Hive unit tests on a single machine take an incredible amount of time (> 15 hours) so we run them in parallel on many servers. 3) Many hive unit tests have race conditions exasperated by 1) slow CPUs in virtualized environments 2) high cpu usage due to many tests running on the same server 4) We use Amazon Spot instances to keep costs down. They often die during a run and we have to re-run the test which was running on that server. At present Cloudera sponsors all the EC2 instances for Hive testing. If we could get some more corporate sponsors such as MSFT or HWx to setup some dedicated EC2 instances we could eliminate the "spot" instances which would increase stability. Brock On Tue, Feb 11, 2014 at 1:05 PM, Remus Rusanu <rem...@microsoft.com> wrote: > Hi Brock, > > It doesn't have to be dire to have impact :) Even one false failure > triggers the need to investigate and causes friction during the submit as > it has to be explained that the failure is not caused by the patch. > What is causing all these failures? > > Thanks, > ~Remus > > ________________________________________ > From: Brock Noland <br...@cloudera.com> > Sent: Tuesday, February 11, 2014 7:10 PM > To: dev@hive.apache.org > Cc: hive-...@hadoop.apache.org > Subject: Re: Pre-commit builds are all failing past 2 days > > Hi, > > I don't think it's as dire as it looks. For example here: > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1270/ > > two tests failed. However, it was just stuck testing HIVE-6254. I restarted > the test server. > > Brock > > > On Tue, Feb 11, 2014 at 2:52 AM, Remus Rusanu <rem...@microsoft.com> > wrote: > > > * Last successful build (#1262), 2 days 2 hr ago< > > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/lastSuccessfulBuild/ > > >? > > > > I had a couple of patches uploaded yesterday and for both I got the red > > -1, but then I noticed all pre-commit builds are failing. What's going > on? > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org > -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
