[
https://issues.apache.org/jira/browse/HIVE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828397#comment-13828397
]
Eric Hanson commented on HIVE-5817:
-----------------------------------
I was able to create a smaller repro as follows.
First, create the table alltypesorc (this is a standard table that is in the
Hive source code as test data).
set hive.vectorized.execution.enabled = false;
create table store(s_store_sk int, s_city string)
stored as orc;
insert overwrite table store
select cint, cstring1
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);
create table store_sales(ss_store_sk int, ss_hdemo_sk int, ss_net_profit double)
stored as orc;
insert overwrite table store_sales
select cint, cint, cdouble
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);
create table household_demographics(hd_demo_sk int)
stored as orc;
insert overwrite table household_demographics
select cint
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);
-- the NOT IN condition makes sure all the cint values are unique
-- finally, run this:
set hive.vectorized.execution.enabled = true;
select store.s_city, ss_net_profit
from store_sales
JOIN store ON store_sales.ss_store_sk = store.s_store_sk
JOIN household_demographics ON store_sales.ss_hdemo_sk =
household_demographics.hd_demo_sk
limit 100;
Expected result: 100 rows of output are produced
Actual result:
...
2013-11-20 17:57:37,487 Stage-4 map = 0%, reduce = 0%
2013-11-20 17:58:04,585 Stage-4 map = 100%, reduce = 100%
Ended Job = job_201311191600_0022 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0022
Examining task ID: task_201311191600_0022_m_000002 (and more) from job
job_201311191600_0022
Task with the most failures(4):
-----
Task ID:
task_201311191600_0022_m_000000
URL:
http://localhost:50030/taskdetails.jsp?jobid=job_201311191600_0022&tipid=task_201311191600_0022_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:181)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Long
vector column and primitive category STRING
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory.buildObjectAssign(VectorColumnAssignFactory.java:278)
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory.buildAssigners(VectorColumnAssignFactory.java:365)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:235)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:675)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:224)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.processOp(VectorMapJoinOperator.java:293)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.flushOutput(VectorMapJoinOperator.java:249)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:244)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:675)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:224)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.processOp(VectorMapJoinOperator.java:293)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
... 9 more
> column name to index mapping in VectorizationContext is broken
> --------------------------------------------------------------
>
> Key: HIVE-5817
> URL: https://issues.apache.org/jira/browse/HIVE-5817
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Priority: Critical
> Attachments: HIVE-5817-uniquecols.broken.patch,
> HIVE-5817.00-broken.patch
>
>
> Columns coming from different operators may have the same internal names
> ("_colNN"). There exists a query in the form {{select b.cb, a.ca from a JOIN
> b ON ... JOIN x ON ...;}} (distilled from a more complex query), which runs
> ok w/o vectorization. With vectorization, it will run ok for most ca, but for
> some ca it will fail (or can probably return incorrect results). That is
> because when building column-to-VRG-index map in VectorizationContext,
> internal column name for ca that the first map join operator adds to the
> mapping may be the same as internal name for cb that the 2nd one tries to
> add. 2nd VMJ doesn't add it (see code in ctor), and when it's time for it to
> output stuff, it retrieves wrong index from the map by name, and then wrong
> vector from VRG.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
