[ 
https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049878#comment-15049878
 ] 

Sergey Shelukhin edited comment on HIVE-12632 at 12/10/15 2:23 AM:
-------------------------------------------------------------------

The patch. [~prasanth_j] looks like LLAP IO was also broken since 
MiniLlapcluster integration, it was not initialized. I am fixing QTestUtils 
toremove all the defaults from ctors to avoid accidental settings, andadding 
back LLAP IO init. That works now; if some other tests breakbecause it was not 
running with them I will move that into separate bug(and assign to you :P)

The main patch changes the explain output, and also checks splits foracidity to 
determine which input format to return, falling back to thewrapped one for ACID 
splits.However, this part is not really tested because ACID code doesn't 
runproperly for the attached test. I modeled this on other ACID tests, 
exceptfor using the alter statements to get some partitions with original 
filesand some with base files.I see no signs of ACID in the logs and LLAP just 
logs several originalfiles for repeated inserts.[~alangates] [~ekoifman] is 
this a correct usage of ACID? (explains, dropetc. omitted for 
brevity):{noformat}set hive.mapred.mode=nonstrict; 
SEThive.vectorized.execution.enabled=true; SET hive.llap.io.enabled=false;SET 
hive.exec.orc.default.buffer.size=32768; 
SEThive.exec.orc.default.row.index.stride=1000; 
SEThive.optimize.index.filter=true; set hive.fetch.task.conversion=none;

set hive.support.concurrency=true;set 
hive.exec.dynamic.partition.mode=nonstrict;set 
hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

CREATE TABLE orc_llap (...) partitioned by (csmallint smallint) clusteredby 
(cint) into 2 buckets stored as orc;

insert into table orc_llap partition (csmallint = 1) select ... fromalltypesorc 
...;insert into table orc_llap partition (csmallint = 2) select ... 
fromalltypesorc ...;

alter table orc_llap SET TBLPROPERTIES ('transactional'='true');

insert into table orc_llap partition (csmallint = 3) select ... fromalltypesorc 
...;

SET hive.llap.io.enabled=true;...select cint, csmallint, cbigint from orc_llap 
where cint is not null;...insert into table orc_llap partition (csmallint = 1) 
values (1, 1, 1, 1);...select cint, csmallint, cbigint from orc_llap where cint 
is not null;{noformat}


was (Author: sershe):
The patch. [~prasanth_j] looks like LLAP IO was also broken since MiniLlap
cluster integration, it was not initialized. I am fixing QTestUtils to
remove all the defaults from ctors to avoid accidental settings, and
adding back LLAP IO init. That works now; if some other tests break
because it was not running with them I will move that into separate bug
(and assign to you :P)

The main patch changes the explain output, and also checks splits for
acidity to determine which input format to return, falling back to the
wrapped one for ACID splits.
However, this part is not really tested because ACID code doesn't run
properly for the attached test. I modeled this on other ACID tests, except
for using the alter statements to get some partitions with original files
and some with base files.
I see no signs of ACID in the logs and LLAP just logs several original
files for repeated inserts.
[~alangates] [~ekoifman] is this a correct usage of ACID? (explains, drop
etc. omitted for brevity):
{noformat}
set hive.mapred.mode=nonstrict; SET
hive.vectorized.execution.enabled=true; SET hive.llap.io.enabled=false;
SET hive.exec.orc.default.buffer.size=32768; SET
hive.exec.orc.default.row.index.stride=1000; SET
hive.optimize.index.filter=true; set hive.fetch.task.conversion=none;

set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

CREATE TABLE orc_llap (...) partitioned by (csmallint smallint) clustered
by (cint) into 2 buckets stored as orc;

insert into table orc_llap partition (csmallint = 1) select ... from
alltypesorc ...;
insert into table orc_llap partition (csmallint = 2) select ... from
alltypesorc ...;

alter table orc_llap SET TBLPROPERTIES ('transactional'='true');

insert into table orc_llap partition (csmallint = 3) select ... from
alltypesorc ...;

SET hive.llap.io.enabled=true;
...
select cint, csmallint, cbigint from orc_llap where cint is not null;
...
insert into table orc_llap partition (csmallint = 1) values (1, 1, 1, 1);
...
select cint, csmallint, cbigint from orc_llap where cint is not null;
{noformat}

> LLAP: don't use IO elevator for ACID tables 
> --------------------------------------------
>
>                 Key: HIVE-12632
>                 URL: https://issues.apache.org/jira/browse/HIVE-12632
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Takahiko Saito
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-12632.patch
>
>
> Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right 
> now, a FileNotFound error is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to