Hi Lefty, thanks a lot for looking at it, I replied to you on HIVE-7254, I guess we can continue our conversation there.
On Sun, Jul 13, 2014 at 11:54 PM, Lefty Leverenz <leftylever...@gmail.com> wrote: > But the wiki page shouldn't be retired altogether, because it's still valid > for releases prior to 0.14.0. So some of those linking docs might need > revision as well as MiniMR and PTest2 > <https://cwiki.apache.org/confluence/display/Hive/MiniMR+and+PTest2>. > > -- Lefty > > > On Mon, Jul 14, 2014 at 2:47 AM, Lefty Leverenz <leftylever...@gmail.com> > wrote: > > > If you retire the wiki page MiniMR and PTest2 > > <https://cwiki.apache.org/confluence/display/Hive/MiniMR+and+PTest2> > then > > five links from other docs will have to be removed: > > > > Page: HiveDeveloperFAQ > > <https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ> > > Page: TestingDocs > > <https://cwiki.apache.org/confluence/display/Hive/TestingDocs> > > Home page: Home > > <https://cwiki.apache.org/confluence/display/Hive/Home> > > Page: Hive PreCommit Patch Testing > > < > https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing > > > > > > Page: DeveloperDocs > > <https://cwiki.apache.org/confluence/display/Hive/DeveloperDocs> > > > > -- Lefty > > > > > > On Mon, Jul 14, 2014 at 12:58 AM, Szehon Ho <sze...@cloudera.com> wrote: > > > >> Hi, > >> > >> This is now done, with some help from Gunther the Pre-commit test > >> framework > >> pick from the itests/qtest/testconfiguration.properties to find the > >> MiniXCliDriver tests, same as the normal test runner. New tests are > picked > >> automatically, no need to do as mentioned above (and we can probably > >> retire > >> that wiki page). > >> > >> There are just 1-2 failing MiniXCliDriver tests that hasn't been run as > >> part of pre-commit suite until this, that may show up in the failures > now. > >> > >> Thanks > >> Szehon > >> > >> > >> > >> > >> > >> > >> On Thu, Jun 19, 2014 at 7:09 AM, Szehon Ho <sze...@cloudera.com> wrote: > >> > >> > (changing subject) > >> > > >> > The MiniTezCliDriver tests have timed-out lately in the pre-commit > >> tests, > >> > reducing coverage of the test as Ashutosh reported. I now configured > >> the > >> > parallel-test framework to run MiniTezCliDriver in batches of 15 > qtest, > >> > like the others. Now the timeout issue is fixed, and test reports are > >> > showing up for those. > >> > > >> > A nice thing is it speeds up the average speed of pre-commit tests by > a > >> > lot, as it was bottlenecked on running all the 79 MiniTezCliDriver > >> tests on > >> > one node. > >> > > >> > The only impact is, now if you are adding new MiniTezCliDriver tests, > >> they > >> > need to be manually added in the Ptest config on the build machine , > >> like > >> > explained in: > >> > https://cwiki.apache.org/confluence/display/Hive/MiniMR+and+PTest2. > >> I've > >> > added all 79 current tests manually. It might be a bigger impact for > >> this > >> > driver than others, as Hive-Tez is under heavy development. I filed > >> > HIVE-7254 <https://issues.apache.org/jira/browse/HIVE-7254> to > explore > >> > improving it, but for now please follow that or notify me, to add the > >> new > >> > test to the pre-commit test coverage. > >> > > >> > Thanks > >> > Szehon > >> > > >> > > >> > > >> > On Fri, Jun 13, 2014 at 3:16 PM, Brock Noland <br...@cloudera.com> > >> wrote: > >> > > >> >> + dev > >> >> > >> >> Good call, yep that will need to be configured. > >> >> > >> >> Brock > >> >> > >> >> > >> >> On Fri, Jun 13, 2014 at 10:29 AM, Szehon Ho <sze...@cloudera.com> > >> wrote: > >> >> > >> >>> I was studying this a bit more, I believe the MiniTezCliDriver tests > >> are > >> >>> hitting timeout after 2 hours as error code is 124. The framework > is > >> >>> running all of them in one call, I'll try to chunk the tests into > >> batches > >> >>> like the other q-tests. > >> >>> > >> >>> I'll try to take a look next week at this. > >> >>> > >> >>> Thanks > >> >>> Szehon > >> >>> > >> >>> > >> >>> On Mon, Jun 9, 2014 at 1:13 PM, Szehon Ho <sze...@cloudera.com> > >> wrote: > >> >>> > >> >>>> It looks like JVM OOM crash during MiniTezCliDriver tests, or its > >> >>>> otherwise crashing. The 407 log has failures, but the 408 log is > >> cut off. > >> >>>> > >> >>>> > >> >>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-407/failed/TestMiniTezCliDriver/maven-test.txt > >> >>>> > >> >>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/maven-test.txt > >> >>>> > >> >>>> The MAVEN_OPTS is already set to "-XmX2g -XX:MaxPermSize=256M". Do > >> you > >> >>>> guys know of any such issues? > >> >>>> > >> >>>> Thanks, > >> >>>> Szehon > >> >>>> > >> >>>> > >> >>>> > >> >>>> On Sun, Jun 8, 2014 at 12:05 PM, Brock Noland <br...@cloudera.com> > >> >>>> wrote: > >> >>>> > >> >>>>> Looks like it's failing to generate a to generate a test output: > >> >>>>> > >> >>>>> > >> >>>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/ > >> >>>>> > >> >>>>> > >> >>>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-408/failed/TestMiniTezCliDriver/TestMiniTezCliDriver.txt > >> >>>>> > >> >>>>> exiting with 124 here: > >> >>>>> > >> >>>>> + wait 21961 > >> >>>>> + timeout 2h mvn -B -o test > >> -Dmaven.repo.local=/home/hiveptest//ip-10-31-188-232-hiveptest-2/maven > >> -Phadoop-2 -Phadoop-2 -Dtest=TestMiniTezCliDriver > >> >>>>> + ret=124 > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> On Sun, Jun 8, 2014 at 11:25 AM, Ashutosh Chauhan < > >> >>>>> hashut...@apache.org> wrote: > >> >>>>> > >> >>>>>> Build #407 ran MiniTezCliDriver > >> >>>>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/407/testReport/org.apache.hadoop.hive.cli/ > >> >>>>>> > >> >>>>>> but Build #408 didn't > >> >>>>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/408/testReport/org.apache.hadoop.hive.cli/ > >> >>>>>> > >> >>>>>> > >> >>>>>> On Sat, Jun 7, 2014 at 12:25 PM, Szehon Ho <sze...@cloudera.com> > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>>> Sounds like there's randomness, either in PTest test-parser or > in > >> >>>>>>> the maven test itself. In the history now, its running between > >> 5633-5707, > >> >>>>>>> which is similar to your range. > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/394/testReport/history/ > >> >>>>>>> > >> >>>>>>> I didnt see any in history without MiniTezCLIDriver, can you > point > >> >>>>>>> me to a build no. if you see one? If nobody else knows > >> immediately, I can > >> >>>>>>> dig deeper at it next week to try to find out. > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Sat, Jun 7, 2014 at 9:00 AM, Ashutosh Chauhan < > >> >>>>>>> hashut...@apache.org> wrote: > >> >>>>>>> > >> >>>>>>>> I noticed that PTest2 framework runs different number of tests > on > >> >>>>>>>> various runs. e.g., on yesterday's runs I saw it ran 5585 & > 5510 > >> tests on > >> >>>>>>>> subsequent runs. In particular, it seems its running > >> MiniTezCliDriver tests > >> >>>>>>>> in only half the runs. Anyone observed this? > >> >>>>>>>> > >> >>>>>>>> > >> > > >> > > > > >
