[jira] [Created] (HIVE-15194) Hive on Tez - Hive Runtime Error while closing operators

[Shankar M (JIRA)]

Shankar M created HIVE-15194:


 Summary: Hive on Tez - Hive Runtime Error while closing operators
 Key: HIVE-15194
 URL: https://issues.apache.org/jira/browse/HIVE-15194
 Project: Hive
  Issue Type: Bug
  Components: Hive, Tez
Affects Versions: 2.1.0
 Environment: Hive 2.1.0 
Tez 0.8.4
4 Nodes x CentOS-6 x64 (32GB Memory, 8 CPUs)
Hadoop 2.7.1
Reporter: Shankar M


Please help me to solve below issue.. 
--
I am setting below commands in hive CLI: 

set hive.execution.engine=tez;

set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
SET hive.tez.container.size=4096;
SET hive.tez.java.opts=-Xmx3072m;


--
{code}
hive> CREATE TABLE tmp_parquet_newtable STORED AS PARQUET AS 
> select a.* from orc_very_large_table a where a.event = 1 and EXISTS 
(SELECT 1 FROM tmp_small_parquet_table b WHERE b.session_id = a.session_id ) ;
Query ID = hadoop_20161114132930_65843cb3-557c-4b42-b662-2901caf5be2d
Total jobs = 1
Launching Job 1 out of 1


Status: Running (Executing on YARN cluster with App id 
application_1479059955967_0049)

--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED  
--
Map 1 .  containerFAILED384 440  340
  26   0  
Map 2 .. container SUCCEEDED  1  100
   0   0  
--
VERTICES: 01/02  [===>>---] 11%   ELAPSED TIME: 43.76 s
--
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1479059955967_0049_2_01, 
diagnostics=[Task failed, taskId=task_1479059955967_0049_2_01_48, 
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( 
failure ) : 
attempt_1479059955967_0049_2_01_48_0:java.lang.RuntimeException: 
java.lang.RuntimeException: Hive Runtime Error while closing operators
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing 
operators
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:422)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
... 14 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:513)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:411)
... 15 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : 
attempt_1479059955967_0049_2_01_48_1:java.lang.RuntimeException: 
java.lang.RuntimeException: Hive Runtime Error while closing operators
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializ

Re: [DISCUSS] Yetus pre-commit tests

[Peter Vary]

Hi Sid,

Good to know, that we have idle resources to run the checks. If it is done in 
parallel then I think it should be possible to run all relevant tests you 
mentioned. It will take some time to make it work, since I will try to contact 
the Yetus project to check if it is possible to incorporate the changes we 
need, so we do not have to have our “own” Yetus.

As for the checkstyle test, it will help to check the Java code that adheres to 
a coding standard. Our one is defined here: 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CodingConventions

"Code should be formatted according to Sun's conventions 
,
 with two exceptions:
Indent two (2) spaces per level, not four (4).
Line length limit is 100 chars, instead of 80 chars."

There is an existing checkstyle configuration which I will compare with the 
standard sun checkstyle configuration to see the differences.

My plan for the rollout is the following:
Create a working version which could be run from command line
Review and commit it - we do not automate it this time. I would like to make it 
available for others to test too.
Open an umbrella jira to collect the problems. Others can add problems here, 
and I will run the tests for multiple patches.
We can discuss the intended changes, and make them
After we are satisfied with the check we should enable in on jenkins

I would like to create tests which we are able to keep all green, so if there 
is a -1 in any one aspect then it should clearly indicate that there is a valid 
problem. I am ready to define a little more lax rules for it if needed. If we 
can do this with the pre-commit tests, and with the ptests as well, then I 
think we made a big step forward to an even better quality project.

Do we agree on this rollout plan, do anyone have any suggestions or ideas to 
consider?

Thanks,
Peter

> On Nov 10, 2016, at 10:29 PM, Siddharth Seth  wrote:
> 
> Peter
> In terms of the modules - ignoring the time taken - I would vote for
> asflicense, author, findbugs, javac, maybe javadoc, wrhitespace. Not sure
> what checkstyle does, and some form of test4tests is already covered in
> ptest. This will at least help preventing new issues. Fixing the existing
> set would be quite an exercise.
> 
> The numbers that you have posted - I think they are on your local system?
> I'd expect these to be higher on the build machines. Not too keen on having
> the runtime go up by 10+ minutes though. Would this run before ptest is
> actually started? Is it possible to start this from within ptest as a
> parallel phase? The ptest server doesn't do much while tests are running.
> Running the regular ptest flow, and this set of checks could be
> parallelized there.
> 
> Thank you for taking this up.
> 
> Sid
> 
> On Thu, Nov 10, 2016 at 7:57 AM, Peter Vary  > wrote:
> 
>> Hi there,
>> 
>> Previously we discussed that it would be good to integrate some automated
>> checks to the pre-commit flow.
>> Alan Gates suggested Apache Yetus and I checked what it can do for us
>> (Yetus 0.3.0).
>> 
>> The good things that I have found:
>> 
>>   - Several existing tests (asflicense, author, checkstyle, findbugs,
>>   javac, javadoc, test4tests, unitveto, whitespace, xml, junit)
>>   - It shows changes in errors/failures so we do not have to clean up
>>   the original code, but new code will be checked.
>>   - Used by multiple ASF projects already - so we will be Apache conform
>>   using it.
>>   - Extensible, so if we decide to add the ptest framework to these test
>>   this could be done
>>   - It is possible to run the test only on the modules which contain
>>   changed files
>> 
>> The bad thing is it could take long time to run the tests even with
>> patches touching a single module.
>> 
>> I think we should decide on which test to include into our pre-commit flow
>> based on our needs and the resource requirements. For reference I have run
>> the test for a fairly small patch on my macbook pro 2 times:
>> 
>>   1. Adding 3 new files to the beeline module (1 java, 1 xml, 1 q.out) -
>>   took ~4 mins - see the result in the attached beeline.out file
>>   2. Adding 3 new files (same as before) to the ql module (1 java, 1
>>   xml, 1 q.out) - took ~12 mins - see the result in the attached ql.out file
>> 
>> In nutshell, the out of the box tests which are available in Yetus are
>> (the numbers are the time in seconds required to run the test in beeline/ql
>> plugin):
>> 
>>   - asflicense (24/23) - apache-rat:check - currently this runs for the
>>   full path
>>   - author (0/0) - Checks for @author tags
>>   - checkstyle (31/66) - checkstyle:checksyle
>>   - findbugs (73/353) - findbugs:findbugs
>>   - javac (53/147) - install compilation warnings (the runtime presented
>>   in the tables are not valid)
>>   - javadoc (34/92) - javadoc warnings
>>   - test4test

[jira] [Created] (HIVE-15195) Drop non-existing macro executes successfully while it must give an error.

[Sangeeta Gulia (JIRA)]

Sangeeta Gulia created HIVE-15195:
-

 Summary: Drop non-existing macro executes successfully while it 
must give an error.
 Key: HIVE-15195
 URL: https://issues.apache.org/jira/browse/HIVE-15195
 Project: Hive
  Issue Type: Bug
Reporter: Sangeeta Gulia


I have created a macro :
CREATE TEMPORARY MACRO simple_add (x int, y int) x + y;
then i dropped the macro.
> drop temporary macro simple_add;
OK
Time taken: 0.038 seconds
hive> 
> 
> select simple_add(2,3);
FAILED: SemanticException [Error 10011]: Line 1:7 Invalid function 'simple_add'
then i again tried to drop the same macro and it again executed without any 
exception:
> drop temporary macro simple_add;
OK
Time taken: 0.016 seconds




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15196) LLAP UI: HIVE-14984 broke LLAP UI

[Gopal V (JIRA)]

Gopal V created HIVE-15196:
--

 Summary: LLAP UI: HIVE-14984 broke LLAP UI
 Key: HIVE-15196
 URL: https://issues.apache.org/jira/browse/HIVE-15196
 Project: Hive
  Issue Type: Bug
  Components: llap, Web UI
Affects Versions: 2.2.0
Reporter: Gopal V


Hard-rewrite rule to the HS2 UI in HttpServer impl

{code}
RewriteRegexRule rootRule = new RewriteRegexRule();
rootRule.setRegex("^/$");
rootRule.setReplacement("/hiveserver2.jsp");
rootRule.setTerminating(true);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15197) count sum query on empty table, returning empty output

[vishal.rajan (JIRA)]

vishal.rajan created HIVE-15197:
---

 Summary: count sum query on empty table, returning empty output 
 Key: HIVE-15197
 URL: https://issues.apache.org/jira/browse/HIVE-15197
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: vishal.rajan


When the below query is run in hive 1.2 it returns 'NULLNULL0'  but 
when the same query is run on hive 2.1.0, nothing is returned.
hive 1.2.0 -
hive>  SELECT sum(destination_pincode),sum(length(source_city)),count(*)  from 
test_stage.geo_zone;
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 4.79 sec   HDFS Read: 7354 HDFS Write: 
114 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 790 msec
OK
NULLNULL0
Time taken: 38.168 seconds, Fetched: 1 row(s)

-hive 2.1.0-
hive> SELECT sum(destination_pincode),sum(length(source_city)),count(*)  from 
test_stage.geo_zone
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the 
future versions. Consider using a different execution engine (i.e. spark, tez) 
or using Hive 1.X releases.

Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
2016-11-14 19:06:15,421 WARN  [Thread-215] mapreduce.JobResourceUploader 
(JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing 
not performed. Implement the Tool interface and execute your application with 
ToolRunner to remedy this.
2016-11-14 19:06:19,222 INFO  [Thread-215] input.FileInputFormat 
(FileInputFormat.java:listStatus(283)) - Total input paths to process : 1
2016-11-14 19:06:20,000 INFO  [Thread-215] mapreduce.JobSubmitter 
(JobSubmitter.java:submitJobInternal(198)) - number of splits:0
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0

2016-11-14 19:06:39,405 Stage-1 map = 0%,  reduce = 0%
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 28.302 seconds





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15198) Capitalisation of columns/aliases is lost in query results

[Michael O'Kane (JIRA)]

Michael O'Kane created HIVE-15198:
-

 Summary: Capitalisation of columns/aliases is lost in query results
 Key: HIVE-15198
 URL: https://issues.apache.org/jira/browse/HIVE-15198
 Project: Hive
  Issue Type: Bug
Reporter: Michael O'Kane


RDBMS systems, while generally case-insensitive, retain case information for 
table metadata, e.g. under postgres

{code}
root=# create table foo as select 1 as "FOO" from users;
SELECT 7
root=# select * from foo;
 FOO 
-
1
1
...
{code}

Hive displays a deviation from this behaviour in that:
1) Persisted table metadata (i.e. from CREATE/ALTER) table is physically 
coerced to lower case upon writing to the DB
2) Transient metadata (i.e. column aliases) are displayed as lower case, 
regardless of the case provided by the query.

In hive:

{code}0: jdbc:hive2://localhost:1/default> create table foo as select 1 as 
FOO from users;
...
0: jdbc:hive2://localhost:1/default> select * from foo;
+--+--+
| foo.foo  |
+--+--+
| 1|
| 1|
...
0: jdbc:hive2://localhost:1/default> select foo as FOO from foo;
+--+--+
| foo  |
+--+--+
| 1|
| 1|
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 53730: HIVE-15196: LLAP UI: HIVE-14984 broke LLAP UI

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53730/
---

Review request for hive and Gopal V.


Repository: hive-git


Description
---

HIVE-15196: LLAP UI: HIVE-14984 broke LLAP UI


Diffs
-

  common/src/java/org/apache/hive/http/HttpServer.java 
42d29592aec0f0794e9c039ac87ac9b8033a4f29 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/services/impl/TestLlapWebServices.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 
9c94611a2497f17b5f4b95b5730425cc7514e567 
  service/src/test/org/apache/hive/service/server/TestHS2HttpServer.java 
d918c6451f6c459f97715053d91e11445b8b376a 

Diff: https://reviews.apache.org/r/53730/diff/


Testing
---

Tested that HS2 WebUI is still being redirected.
Added unit test for LLAP UI and confirmed that it was failing before and 
passing now.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-15199) INSERT INTO data on S3 is replacing the old rows with the new ones

[JIRA]

Sergio Peña created HIVE-15199:
--

 Summary: INSERT INTO data on S3 is replacing the old rows with the 
new ones
 Key: HIVE-15199
 URL: https://issues.apache.org/jira/browse/HIVE-15199
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Sergio Peña
Assignee: Sergio Peña
Priority: Critical


Any INSERT INTO statement run on S3 tables and when the scratch directory is 
saved on S3 is deleting old rows of the table.

{noformat}
hive> set hive.blobstore.use.blobstore.as.scratchdir=true;

hive> create table t1 (id int, name string) location 's3a://spena-bucket/t1';

hive> insert into table t1 values (1,'name1');

hive> select * from t1;
1   name1

hive> insert into table t1 values (2,'name2');

hive> select * from t1;
2   name2
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

[Sergey Shelukhin]

> On Nov. 11, 2016, 12:28 a.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java,
> >  line 329
> > 
> >
> > nit: Can the batchkey be null?

no


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155661
---


On Nov. 9, 2016, 2:34 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 9, 2016, 2:34 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  eb8ee6c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> bcb54d6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 76a1286 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/
---

(Updated Nov. 14, 2016, 8:45 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 a27266f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 2fa68a9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 72f733a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 c9b0a4d 
  orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
4d09dcd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
a4925b9 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 ebbdf8d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
  ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
  ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
  ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642 
  ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
 907181e 

Diff: https://reviews.apache.org/r/53593/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 53744: Optimize mutli column distinct queries

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53744/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-10901


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
 c243266 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
 7d4411a 
  ql/src/test/queries/clientpositive/multi_count_distinct.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/multi_count_distinct.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53744/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-15200) Support setOp in subQuery with parentheses

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-15200:
--

 Summary: Support setOp in subQuery with parentheses
 Key: HIVE-15200
 URL: https://issues.apache.org/jira/browse/HIVE-15200
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


{code}
explain select key from ((select key from src) union (select key from src))subq;
{code}
will throw
{code}
FAILED: ParseException line 1:47 cannot recognize input near 'union' '(' 
'select' in subquery source
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53593: HIVE-14089 complex type support in LLAP IO is broken

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53593/#review155870
---


Ship it!




Ship It!

- Prasanth_J


On Nov. 14, 2016, 8:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53593/
> ---
> 
> (Updated Nov. 14, 2016, 8:45 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  a27266f 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  2fa68a9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  72f733a 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  c9b0a4d 
>   orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c347181 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java 245e3ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java a9dbc3e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ccb39da 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
> 4d09dcd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
> a4925b9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  ebbdf8d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 4405232 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 5cc3663 
>   ql/src/test/queries/clientpositive/vector_complex_all.q 91a7368 
>   ql/src/test/results/clientpositive/llap/vector_complex_all.q.out 08d49bc 
>   ql/src/test/results/clientpositive/llap/vector_complex_join.q.out 97d5642 
>   ql/src/test/results/clientpositive/vector_complex_all.q.out a54a371 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java
>  907181e 
> 
> Diff: https://reviews.apache.org/r/53593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-15201) Flaky test: TestMiniLlapCliDriver.testCliDriver[transform_ppr2]

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-15201:


 Summary: Flaky test: 
TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
 Key: HIVE-15201
 URL: https://issues.apache.org/jira/browse/HIVE-15201
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


{code}
<   Statistics: Num rows: 1000 Data size: 178000 Basic stats: 
COMPLETE Column stats: PARTIAL
---
>   Statistics: Num rows: 1000 Data size: 178000 Basic stats: 
> COMPLETE Column stats: COMPLETE
46c46
< Statistics: Num rows: 1000 Data size: 272000 Basic stats: 
COMPLETE Column stats: PARTIAL
---
> Statistics: Num rows: 1000 Data size: 272000 Basic stats: 
> COMPLETE Column stats: COMPLETE
59c59
<   Statistics: Num rows: 1000 Data size: 272000 Basic 
stats: COMPLETE Column stats: PARTIAL
---
>   Statistics: Num rows: 1000 Data size: 272000 Basic 
> stats: COMPLETE Column stats: COMPLETE
63c63
< Statistics: Num rows: 333 Data size: 2664 Basic 
stats: COMPLETE Column stats: PARTIAL
---
> Statistics: Num rows: 333 Data size: 2664 Basic 
> stats: COMPLETE Column stats: COMPLETE
69c69
<   Statistics: Num rows: 333 Data size: 2664 Basic 
stats: COMPLETE Column stats: PARTIAL
---
>   Statistics: Num rows: 333 Data size: 2664 Basic 
> stats: COMPLETE Column stats: COMPLETE
178c178
< Statistics: Num rows: 333 Data size: 2664 Basic stats: 
COMPLETE Column stats: PARTIAL
---
> Statistics: Num rows: 333 Data size: 2664 Basic stats: 
> COMPLETE Column stats: COMPLETE
184c184
<   Statistics: Num rows: 333 Data size: 2664 Basic stats: 
COMPLETE Column stats: PARTIAL
---
>   Statistics: Num rows: 333 Data size: 2664 Basic stats: 
> COMPLETE Column stats: COMPLETE

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

[Rui Li (JIRA)]

Rui Li created HIVE-15202:
-

 Summary: Concurrent compactions for the same partition may 
generate malformed folder structure
 Key: HIVE-15202
 URL: https://issues.apache.org/jira/browse/HIVE-15202
 Project: Hive
  Issue Type: Bug
Reporter: Rui Li


If two compactions run concurrently on a single partition, it may generate 
folder structure like this: (nested base dir)
{noformat}
drwxr-xr-x   - root supergroup  0 2016-11-14 22:23 
/user/hive/warehouse/test/z=1/base_007/base_007
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_0
-rw-r--r--   3 root supergroup611 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_1
-rw-r--r--   3 root supergroup614 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_2
-rw-r--r--   3 root supergroup621 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_3
-rw-r--r--   3 root supergroup621 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_4
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_5
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_6
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_7
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_8
-rw-r--r--   3 root supergroup201 2016-11-14 21:46 
/user/hive/warehouse/test/z=1/base_007/bucket_9
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15203) Hive export command does export to non HDFS file system

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-15203:


 Summary: Hive export command does export to non HDFS file system
 Key: HIVE-15203
 URL: https://issues.apache.org/jira/browse/HIVE-15203
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Thejas M Nair


Hive export command does export to non HDFS file system.
If a non hdfs filessystem is the default file system, then export command tries 
to use hdfs scheme against the url of the default file system.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15204) Hive-Hbase integration thorws "java.lang.ClassNotFoundException: NULL::character varying" (Postgres)

[Anshuman (JIRA)]

Anshuman created HIVE-15204:
---

 Summary: Hive-Hbase integration thorws 
"java.lang.ClassNotFoundException: NULL::character varying" (Postgres)
 Key: HIVE-15204
 URL: https://issues.apache.org/jira/browse/HIVE-15204
 Project: Hive
  Issue Type: Bug
  Components: HBase Handler
Affects Versions: 2.1.0
 Environment: apache-hive-2.1.0-bin
hbase-1.1.1
Reporter: Anshuman


When doing hive to hbase integration, we have observed that current Apache Hive 
2.x is not able to recognise 'NULL::character varying' (Variant data type of 
NULL in prostgres) properly and throws the java.lang.ClassNotFoundException 
exception.

Exception:
ERROR ql.Driver: FAILED: RuntimeException java.lang.ClassNotFoundException: 
NULL::character varying
java.lang.RuntimeException: java.lang.ClassNotFoundException: NULL::character 
varying

Caused by: java.lang.ClassNotFoundException: NULL::character varying
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

Reason:

org.apache.hadoop.hive.ql.metadata.Table.java

final public Class getInputFormatClass() {
if (inputFormatClass == null) {
  try {
String className = tTable.getSd().getInputFormat();

if (className == null) {  /*If the className is one of the postgres 
variant of NULL i.e. 'NULL::character varying' control is going to else block 
and throwing error.*/

  if (getStorageHandler() == null) {
return null;
  }
  inputFormatClass = getStorageHandler().getInputFormatClass();
} else {
  inputFormatClass = (Class)
Class.forName(className, true, 
Utilities.getSessionSpecifiedClassLoader());
}
  } catch (ClassNotFoundException e) {
throw new RuntimeException(e);
  }
}
return inputFormatClass;
  }

Steps to reproduce:
Hive 2.x (e.g. apache-hive-2.1.0-bin) and HBase (e.g. hbase-1.1.1)

1. Install and configure Hive, if it is not already installed.
2. Install and configure HBase, if it is not already installed.
3. Configure the hive-site.xml File (as per recommended steps)
4. Provide necessary jars to Hive (as per recommended steps)
4. Create table in HBase as shown below -

create 'hivehbase', 'ratings'
put 'hivehbase', 'row1', 'ratings:userid', 'user1'
put 'hivehbase', 'row1', 'ratings:bookid', 'book1'
put 'hivehbase', 'row1', 'ratings:rating', '1'
 
put 'hivehbase', 'row2', 'ratings:userid', 'user2'
put 'hivehbase', 'row2', 'ratings:bookid', 'book1'
put 'hivehbase', 'row2', 'ratings:rating', '3'
 
put 'hivehbase', 'row3', 'ratings:userid', 'user2'
put 'hivehbase', 'row3', 'ratings:bookid', 'book2'
put 'hivehbase', 'row3', 'ratings:rating', '3'
 
put 'hivehbase', 'row4', 'ratings:userid', 'user2'
put 'hivehbase', 'row4', 'ratings:bookid', 'book4'
put 'hivehbase', 'row4', 'ratings:rating', '1'

5. Create external table as shown below 
CREATE EXTERNAL TABLE hbasehive_table
(key string, userid string,bookid string,rating int) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES 
("hbase.columns.mapping" = ":key,ratings:userid,ratings:bookid,ratings:rating")
TBLPROPERTIES ("hbase.table.name" = "hivehbase");

6. select * from hbasehive_table;
FAILED: RuntimeException java.lang.ClassNotFoundException: NULL::character 
varying



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hive-Hbase integration thorws "java.lang.ClassNotFoundException: NULL::character varying" (Postgres)

[Anshuman Dwivedi]

 Hi Guys

When  doing hive to hbase integration, we have observed that current Apache  
Hive 2.x is not able to recognise 'NULL::character varying' (Variant  data type 
of NULL in prostgres) properly and throws the  java.lang.ClassNotFoundException 
exception.

Exception:
FAILED: RuntimeException java.lang.ClassNotFoundException: NULL::character 
varying

Found out more this JIRA: https://issues.apache.org/jira/browse/HIVE-15204

BR
Anshuman Dwivedi
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you