Changes for Build #1163 Changes for Build #1164 [heyongqiang] HIVE-2666 [jira] StackOverflowError when using custom UDF in map join (Kevin Wilfong via Yongqiang He)
Summary: Resource files are now added to the class path as soon as they are added via the CLI. This fixes the stack overflow error mentioned in the JIRA by ensuring a consistent class loader between serializers and deserializers for the same query. Note that now serdes which contain a static block to register themselves are now registered twice, once when adding the file to the class loader, and once when an instance of the class is created. Previously, registering a serde twice resulted in an exception, to avoid this, I have downgraded it to a warning. When a custom UDF is used as part of a join which is converted to a map join, the XMLEncoder enters an infinite loop when serializing the map reduce task for the second time, as part of sending it to be executed. This results in a stack overflow error. Test Plan: I ran the unit tests to verify nothing was broken. I ran several queries which used custom UDFs and involved a join which was converted to a map join. I verified these completed successfully consistently Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: heyongqiang, kevinwilfong Differential Revision: 957 [namit] HIVE-2642 fix Hive-2566 and make union optimization more aggressive (Yongqiang He via namit) Changes for Build #1166 Changes for Build #1167 Changes for Build #1168 [heyongqiang] HIVE-2600: Enable/Add type-specific compression for rcfile (Krishna Kumar via He Yongqiang) Changes for Build #1169 Changes for Build #1170 [cws] HIVE-1877. Add java_method() as a synonym for the reflect() UDF (Zhenxiao Luo via cws) Changes for Build #1171 Changes for Build #1172 Changes for Build #1173 Changes for Build #1174 Changes for Build #1175 [hashutosh] HIVE-2681 : SUCESS is misspelled (jonchang via Ashutosh Chauhan) Changes for Build #1176 [hashutosh] HIVE-2616 : Passing user identity from metastore client to server in non-secure mode (Ashutosh Chauhan) Changes for Build #1177 Changes for Build #1178 Changes for Build #1179 Changes for Build #1180 Changes for Build #1181 Changes for Build #1182 [heyongqiang] HIVE-2621:Allow multiple group bys with the same input data and spray keys to be run on the same reducer. (Kevin via He Yongqiang) Changes for Build #1184 [namit] HIVE-2690 a bug in 'alter table concatenate' that causes filenames getting double url encoded (He Yongqiang via namit) Changes for Build #1185 Changes for Build #1186 Changes for Build #1187 Changes for Build #1188 Changes for Build #1189 Changes for Build #1190 [amareshwari] HIVE-2629. Make a single Hive binary work with both 0.20.x and 0.23.0. (Thomas Weise via amareshwari) Changes for Build #1191 [amareshwari] HIVE-2629. Reverting previous commit Changes for Build #1192 [heyongqiang] HIVE-2706 [jira] StackOverflowError when using custom UDF after adding archive after adding jars (Kevin Wilfong via Yongqiang He) Summary: https://issues.apache.org/jira/browse/HIVE-2706 The issue was that the current thread's classloader and the classloader in the conf differed due to the prehook updating only the current thread's classloader with new jars. Now, it updates both classloaders, fixing the issue. When a custom UDF is used in a query after add an archive, such as a zip file, after adding jars, the XMLEncoder enters an infinite loop when serializing the map reduce task, as part of sending it to be executed. This results in a stack overflow error. Test Plan: Verified it fixed the stack overflow error. Reviewers: JIRA, heyongqiang, njain Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D1167 Changes for Build #1193 [hashutosh] HIVE-2705: SemanticAnalyzer twice swallows an exception it shouldn't (jghoman via hashutosh) 3 tests failed. FAILED: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown Error Message: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown(TestCliDriver.java:25901) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) FAILED: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:9476) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) FAILED: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl5 Error Message: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. Stack Trace: junit.framework.AssertionFailedError: Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get more logs. at junit.framework.Assert.fail(Assert.java:50) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl5(TestCliDriver.java:16877) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1193) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1193/ to view the results.
