Hi Arun, thanks for chiming in!
On Sun, Jun 16, 2013 at 4:50 PM, Arun C Murthy <a...@hortonworks.com> wrote: >> But there's a bit of bad news too (or at least the news that need to >> be triaged). At >> this point I don't know whether the Hadoop code is to blame or the >> tests/components >> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha: >> 1. HDFS append integration tests failed: >> >> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/ >> It seems that there's some incompatibility between the client code >> that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 >> release) >> and the current Hadoop 2.1.0. > > This is well known, we need to recompile against hadoop-2.1.0-beta. That's what I suspected. Thanks for confirming. Like I said -- given the tight schedule I didn't have the time to recompile the entire Bigtop stack against Hadoop 2.1.0-beta (including tests). > Essentially, this is a Sqoop error - we could investigate why it took 600s, > but doesn't look like a blocker to me. > > We'll probably need another 2.1.1-beta anyway... Agreed. There seems to be some weird interaction between Sqoop and RM, though. But it definitely doesn't qualify as a blocker from what I see in the logs and my debugger and whether it ends up being a Sqoop issue or Hadoop issue we can tackle it a tad later. > Both errors seemed to be related to Hive unit tests failing since > MiniHDFSCluster didn't come up: Bigtop Hive tests don't use MiniHDFSCluster -- they are executed against an honest fully distributed cluster. Like I said -- the weird part here is that the failing two are in the middle of a whole bunch that worked just fine. And I haven't really seen them fail before like that ever. I wish I had more time to investigate. >> 4. All of the Mahout tests failed with the following: >> >> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/ >> This odd, and as I said -- all I know at this point is that >> the very same >> tests running the very same Mahout pass with Hadoop 2.0.5-alpha. > > Again, this is related to fact that we need to recompile Mahout against > 2.1.0-beta - in > particular this was due to the compatibility work done via MAPREDUCE-5156 > (part of > MAPREDUCE-5108). Ah! Didn't know that. Well, it sounds like we've got enough mileage out of of the quick application of Bigtop stack/tests from Bigtop 0.6.0. At this point I feel like the first level of smoke testing passed and if we want to drill down we'd have to spend time and rebuild everything against Hadoop 2.1.0. That'll take some time and should happen sometimes this week. Hope the amount of validation I was capable of was useful so far. Thanks, Roman.
