Hello, I am trying to evaluate Hive 0.11.0-RC1, in particular I am very interested in the ORC storage mechanism. We have a need to have one column be a Map<String,String> in a table and from what I have read this is supported with the ORC format, however when trying to do a select on a table with a Map column I get an exception (the stack trace and more details is below). Here is what I have done to test this:
Environment: OS X 10.8, Java 1.6 Hadoop 1.0.4 running in local mode Hive 0.11.0-RC1 (from -- http://people.apache.org/~hashutosh/hive-0.11.0-rc1/) using Deby for Metadata Steps to repeat * Create table: create table tempWithMap(name String, Map<String,String>); * Load data into the table from a local file. * Create an ORC backed table off of the TextFile backed table doing: create table orcTableWithMap stored as ORC as select * from tempWithMap; * Select name field from orcTableWithMap table doing: select name from orcTableWithMap; I then get the following output: Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201304301245_0033, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201304301245_0033 Kill Command = /Users/hadoop/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201304301245_0033 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 2013-05-03 10:55:52,796 Stage-1 map = 0%, reduce = 0% 2013-05-03 10:56:22,889 Stage-1 map = 100%, reduce = 100% Ended Job = job_201304301245_0033 with errors Error during job, obtaining debugging information... Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201304301245_0033 Examining task ID: task_201304301245_0033_m_000002 (and more) from job job_201304301245_0033 Task with the most failures(4): ----- Task ID: task_201304301245_0033_m_000000 URL: http://localhost:50030/taskdetails.jsp?jobid=job_201304301245_0033&tipid=task_201304301245_0033_m_000000 ----- Diagnostic Messages for this Task: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:522) at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90) ... 22 more Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138) at org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:270) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:482) ... 23 more FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask MapReduce Jobs Launched: Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec Seeing this exception raised at least two questions in my mind: 1. I was under the impression, albeit perhaps wrong, that with ORC only the columns being selected would be deserialized. If that is true, then why would the map be deserialized when my query was for the column that is a string type and the map is not needed to satisfy the query? 2. Is there something I am doing wrong here? If not what can I do to help track down the source of the problem? I have tried this test using a map<int,int> and get the same results. Also, I have been trying to run the ORC Junit tests with Eclipse but have been having a dickens of time getting that to work. Any input/insight would be greatly appreciated. Thanks in advance, Andrew
