----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22329/#review44992 -----------------------------------------------------------
1. I think webhcat-default.xml should be modified to include the jars that are now required in templeton.libjars to minimize out-of-the-box config for end users. 2. Is there any test (e2e) that can be added for this? (with reasonable amount of effort) 3. When you tested that Pig/Hive jobs get properly tagged, you mean you tested that MR jobs that are generated by Pig/Hive are tagged, correct? hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/JobSubmissionConstants.java <https://reviews.apache.org/r/22329/#comment79625> I think it would be useful to add a more detailed description of these props. Something like what is in the JIRA ticket. I would have added the ticket number to the comment, but Hive prohibits that. hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/LaunchMapper.java <https://reviews.apache.org/r/22329/#comment79632> Which user will this use? Is it the user running WebHCat or the value of 'doAs' parameter? shims/0.23/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim23.java <https://reviews.apache.org/r/22329/#comment79613> Is LOG.info() the right log level? Seems like it will pollute the log file. shims/0.23/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim23.java <https://reviews.apache.org/r/22329/#comment79615> Is LOG.info() the right level? shims/0.23/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim23.java <https://reviews.apache.org/r/22329/#comment79631> log level - Eugene Koifman On June 6, 2014, 10:02 p.m., Ivan Mitic wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22329/ > ----------------------------------------------------------- > > (Updated June 6, 2014, 10:02 p.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > ------- > > Approach in the patch is similar to what Oozie does to handle this situation. > Specifically, all child map jobs get tagged with the launcher MR job id. On > launcher task restart, launcher queries RM for the list of jobs that have the > tag and kills them. After that it moves on to start the same child job again. > Again, similarly to what Oozie does, a new templeton.job.launch.time property > is introduced that captures the launcher job submit timestamp and later used > to reduce the search window when RM is queried. > > To validate the patch, you will need to add webhcat shim jars to > templeton.libjars as now webhcat launcher also has a dependency on hadoop > shims. > > I have noticed that in case of the SqoopDelegator webhcat currently does not > set the MR delegation token when optionsFile flag is used. This also creates > the problem in this scenario. This looks like something that should be > handled via a separate Jira. > > > Diffs > ----- > > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveDelegator.java > 23b1c4f > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarDelegator.java > 41b1dc5 > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LauncherDelegator.java > 04a5c6f > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/PigDelegator.java > 04e061d > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SqoopDelegator.java > adcd917 > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/JobSubmissionConstants.java > a6355a6 > > hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/LaunchMapper.java > 556ee62 > shims/0.20S/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim20S.java > d3552c1 > shims/0.23/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim23.java > 5a728b2 > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java > 299e918 > > Diff: https://reviews.apache.org/r/22329/diff/ > > > Testing > ------- > > I have validated that MR, Pig and Hive jobs do get tagged appropriately. I > have also validated that previous child jobs do get killed on RM > failover/task failure. > > > Thanks, > > Ivan Mitic > >
