[jira] [Created] (HIVE-3790) UDF to find DATE OFFSET for a given date or timestamp

[Jithin John (JIRA)]

Jithin John created HIVE-3790:
-

 Summary: UDF to find DATE OFFSET for a given date or timestamp 
 Key: HIVE-3790
 URL: https://issues.apache.org/jira/browse/HIVE-3790
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Affects Versions: 0.9.0
Reporter: Jithin John


Current releases of Hive lacks a  generic function which would find the date 
offset to a date / timestamp. Current releases have date_add (date) and 
date_sub(date) which allows user to add or substract days only.But we could not 
use year or month as a unit.


The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
start_date according to the unit. Here the unit can be year , month and day.

The function could be used for date range queries and is more flexible than the 
existing functions.

Functionality :-

Function Name: DATE_OFFSET(date,offset,unit)
 
Add a offset value to the unit part of the date.
Returns  the date in the format of "-MM-dd" .
Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
-> 2009-06-29
Usage :-

Case  : To calculate the expiry date of a item from manufacturing date 
Table :- ITEM_TAB
 Manufacturing_date  |item id|store id|value|unit|price
  2012-12-01|110001|00003|0.99|1.00|0.99
  2012-12-02|110001|00008|0.99|0.00|0.00
  2012-12-03|110001|00009|0.99|0.00|0.00
  2012-12-04|110001|001112002|0.99|0.00|0.00
  2012-12-05|110001|001112003|0.99|0.00|0.00
  2012-12-06|110001|001112006|0.99|1.00|0.99
  2012-12-07|110001|001112007|0.99|0.00|0.00
  2012-12-08|110001|001112008|0.99|0.00|0.00
  2012-12-09|110001|001112009|0.99|0.00|0.00
  2012-12-10|110001|001112010|0.99|0.00|0.00
  2012-12-11|110001|001113003|0.99|0.00|0.00
  2012-12-12|110001|001113006|0.99|0.00|0.00
  2012-12-13|110001|001113008|0.99|0.00|0.00
  2012-12-14|110001|001113010|0.99|0.00|0.00
  2012-12-15|110001|001114002|0.99|0.00|0.00
  2012-12-16|110001|001114004|0.99|1.00|0.99
  2012-12-17|110001|001114005|0.99|0.00|0.00
  2012-12-18|110001|001121004|0.99|0.00|0.00 

QUERY:
select man_date , date_offset(man_date ,5 ,'year') as expiry_date from item_tab;


RESULT:
2012-12-01  2017-12-01
2012-12-02  2017-12-02
2012-12-03  2017-12-03
2012-12-04  2017-12-04
2012-12-05  2017-12-05
2012-12-06  2017-12-06
2012-12-07  2017-12-07
2012-12-08  2017-12-08
2012-12-09  2017-12-09
2012-12-10  2017-12-10
2012-12-11  2017-12-11
2012-12-12  2017-12-12
2012-12-13  2017-12-13
2012-12-14  2017-12-14
2012-12-15  2017-12-15
2012-12-16  2017-12-16
2012-12-17  2017-12-17
2012-12-18  2017-12-18



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3790) UDF to find DATE OFFSET for a given date or timestamp

[Jithin John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jithin John updated HIVE-3790:
--

Description: 
Current releases of Hive lacks a  generic function which would find the date 
offset to a date / timestamp. Current releases have date_add (date) and 
date_sub(date) which allows user to add or substract days only.But we could not 
use year or month as a unit.


The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
start_date according to the unit. Here the unit can be year , month and day.

The function could be used for date range queries and is more flexible than the 
existing functions.

Functionality :-

Function Name: DATE_OFFSET(date,offset,unit)
 
Add a offset value to the unit part of the date/timestamp.
Returns  the date in the format of "-MM-dd" .
Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
-> 2009-06-29
Usage :-

Case  : To calculate the expiry date of a item from manufacturing date 
Table :- ITEM_TAB
 Manufacturing_date  |item id|store id|value|unit|price
  2012-12-01|110001|00003|0.99|1.00|0.99
  2012-12-02|110001|00008|0.99|0.00|0.00
  2012-12-03|110001|00009|0.99|0.00|0.00
  2012-12-04|110001|001112002|0.99|0.00|0.00
  2012-12-05|110001|001112003|0.99|0.00|0.00
  2012-12-06|110001|001112006|0.99|1.00|0.99
  2012-12-07|110001|001112007|0.99|0.00|0.00
  2012-12-08|110001|001112008|0.99|0.00|0.00
  2012-12-09|110001|001112009|0.99|0.00|0.00
  2012-12-10|110001|001112010|0.99|0.00|0.00
  2012-12-11|110001|001113003|0.99|0.00|0.00
  2012-12-12|110001|001113006|0.99|0.00|0.00
  2012-12-13|110001|001113008|0.99|0.00|0.00
  2012-12-14|110001|001113010|0.99|0.00|0.00
  2012-12-15|110001|001114002|0.99|0.00|0.00
  2012-12-16|110001|001114004|0.99|1.00|0.99
  2012-12-17|110001|001114005|0.99|0.00|0.00
  2012-12-18|110001|001121004|0.99|0.00|0.00 

QUERY:
select man_date , date_offset(man_date ,5 ,'year') as expiry_date from item_tab;


RESULT:
2012-12-01  2017-12-01
2012-12-02  2017-12-02
2012-12-03  2017-12-03
2012-12-04  2017-12-04
2012-12-05  2017-12-05
2012-12-06  2017-12-06
2012-12-07  2017-12-07
2012-12-08  2017-12-08
2012-12-09  2017-12-09
2012-12-10  2017-12-10
2012-12-11  2017-12-11
2012-12-12  2017-12-12
2012-12-13  2017-12-13
2012-12-14  2017-12-14
2012-12-15  2017-12-15
2012-12-16  2017-12-16
2012-12-17  2017-12-17
2012-12-18  2017-12-18



  was:
Current releases of Hive lacks a  generic function which would find the date 
offset to a date / timestamp. Current releases have date_add (date) and 
date_sub(date) which allows user to add or substract days only.But we could not 
use year or month as a unit.


The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
start_date according to the unit. Here the unit can be year , month and day.

The function could be used for date range queries and is more flexible than the 
existing functions.

Functionality :-

Function Name: DATE_OFFSET(date,offset,unit)
 
Add a offset value to the unit part of the date.
Returns  the date in the format of "-MM-dd" .
Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
-> 2009-06-29
Usage :-

Case  : To calculate the expiry date of a item from manufacturing date 
Table :- ITEM_TAB
 Manufacturing_date  |item id|store id|value|unit|price
  2012-12-01|110001|00003|0.99|1.00|0.99
  2012-12-02|110001|00008|0.99|0.00|0.00
  2012-12-03|110001|00009|0.99|0.00|0.00
  2012-12-04|110001|001112002|0.99|0.00|0.00
  2012-12-05|110001|001112003|0.99|0.00|0.00
  2012-12-06|110001|001112006|0.99|1.00|0.99
  2012-12-07|110001|001112007|0.99|0.00|0.00
  2012-12-08|110001|001112008|0.99|0.00|0.00
  2012-12-09|110001|001112009|0.99|0.00|0.00
  2012-12-10|110001|001112010|0.99|0.00|0.00
  2012-12-11|110001|001113003|0.99|0.00|0.00
  2012-12-12|110001|001113006|0.99|0.00|0.00
  2012-12-13|110001|001113008|0.99|0.00|0.00
  2012-12-14|110001|001113010|0.99|0.00|0.00
  2012-12-15|110001|001114002|0.99|0.00|0.00
  2012-12-16|110001|001114004|0.99|1.00|0.99
  2012-12-17|110001|001114005|0.99|0.00|0.00
  2012-12-18|110001|001121004|0.99|0.00|0.00 

QUERY:
select man_date , date_offset(man_date ,5 ,'year') as expiry_date from item_tab;


RESULT:
2012-12-01  2017-12-01
2012-12-02  2017-12-02
2012-12-03  2017-12-03
2012-12-04  2017-12-04
2012-12-05  2017-12-05
2012-12-06  2017-12-06
2012-12-07  2017-12-07
2012-12-08  2017-12-08
2012-12-09  2017-12-09
2012-12-10  2017-12-10
2012-12-11  2017-12-11
2012-12-12  2017-12-12
2012-12-13  2017-12-13
2012-12-14  2017-12-14
2012-12-15  2017-12-15
2012-1

[jira] [Commented] (HIVE-3790) UDF to find DATE OFFSET for a given date or timestamp

[Georgy B Abraham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528836#comment-13528836
 ] 

Georgy B Abraham commented on HIVE-3790:


UDF would be useful in manipulating the day , month or year of any given date 
by the offset passed.
eg: date_offset('2009-07-29', -1 ,'MONTH' )  would give us the date 2009-06-29. 
Should have to consider Leapyears Februvary scenario too

> UDF to find DATE OFFSET for a given date or timestamp 
> --
>
> Key: HIVE-3790
> URL: https://issues.apache.org/jira/browse/HIVE-3790
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Jithin John
>
> Current releases of Hive lacks a  generic function which would find the date 
> offset to a date / timestamp. Current releases have date_add (date) and 
> date_sub(date) which allows user to add or substract days only.But we could 
> not use year or month as a unit.
> 
> The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
> start_date according to the unit. Here the unit can be year , month and day.
> The function could be used for date range queries and is more flexible than 
> the existing functions.
> Functionality :-
> Function Name: DATE_OFFSET(date,offset,unit)
>
> Add a offset value to the unit part of the date/timestamp.
> Returns  the date in the format of "-MM-dd" .
> Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
> -> 2009-06-29
> Usage :-
> Case  : To calculate the expiry date of a item from manufacturing date 
> Table :- ITEM_TAB
>  Manufacturing_date  |item id|store id|value|unit|price
>   2012-12-01|110001|00003|0.99|1.00|0.99
>   2012-12-02|110001|00008|0.99|0.00|0.00
>   2012-12-03|110001|00009|0.99|0.00|0.00
>   2012-12-04|110001|001112002|0.99|0.00|0.00
>   2012-12-05|110001|001112003|0.99|0.00|0.00
>   2012-12-06|110001|001112006|0.99|1.00|0.99
>   2012-12-07|110001|001112007|0.99|0.00|0.00
>   2012-12-08|110001|001112008|0.99|0.00|0.00
>   2012-12-09|110001|001112009|0.99|0.00|0.00
>   2012-12-10|110001|001112010|0.99|0.00|0.00
>   2012-12-11|110001|001113003|0.99|0.00|0.00
>   2012-12-12|110001|001113006|0.99|0.00|0.00
>   2012-12-13|110001|001113008|0.99|0.00|0.00
>   2012-12-14|110001|001113010|0.99|0.00|0.00
>   2012-12-15|110001|001114002|0.99|0.00|0.00
>   2012-12-16|110001|001114004|0.99|1.00|0.99
>   2012-12-17|110001|001114005|0.99|0.00|0.00
>   2012-12-18|110001|001121004|0.99|0.00|0.00 
> QUERY:
> select man_date , date_offset(man_date ,5 ,'year') as expiry_date from 
> item_tab;
> RESULT:
> 2012-12-01  2017-12-01
> 2012-12-02  2017-12-02
> 2012-12-03  2017-12-03
> 2012-12-04  2017-12-04
> 2012-12-05  2017-12-05
> 2012-12-06  2017-12-06
> 2012-12-07  2017-12-07
> 2012-12-08  2017-12-08
> 2012-12-09  2017-12-09
> 2012-12-10  2017-12-10
> 2012-12-11  2017-12-11
> 2012-12-12  2017-12-12
> 2012-12-13  2017-12-13
> 2012-12-14  2017-12-14
> 2012-12-15  2017-12-15
> 2012-12-16  2017-12-16
> 2012-12-17  2017-12-17
> 2012-12-18  2017-12-18

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3790) UDF to introduce an OFFSET(day,month or year) for a given date or timestamp

[Georgy B Abraham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Georgy B Abraham updated HIVE-3790:
---

Summary: UDF to introduce an OFFSET(day,month or year) for a given date or 
timestamp   (was: UDF to find DATE OFFSET for a given date or timestamp )

> UDF to introduce an OFFSET(day,month or year) for a given date or timestamp 
> 
>
> Key: HIVE-3790
> URL: https://issues.apache.org/jira/browse/HIVE-3790
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Jithin John
>
> Current releases of Hive lacks a  generic function which would find the date 
> offset to a date / timestamp. Current releases have date_add (date) and 
> date_sub(date) which allows user to add or substract days only.But we could 
> not use year or month as a unit.
> 
> The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
> start_date according to the unit. Here the unit can be year , month and day.
> The function could be used for date range queries and is more flexible than 
> the existing functions.
> Functionality :-
> Function Name: DATE_OFFSET(date,offset,unit)
>
> Add a offset value to the unit part of the date/timestamp.
> Returns  the date in the format of "-MM-dd" .
> Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
> -> 2009-06-29
> Usage :-
> Case  : To calculate the expiry date of a item from manufacturing date 
> Table :- ITEM_TAB
>  Manufacturing_date  |item id|store id|value|unit|price
>   2012-12-01|110001|00003|0.99|1.00|0.99
>   2012-12-02|110001|00008|0.99|0.00|0.00
>   2012-12-03|110001|00009|0.99|0.00|0.00
>   2012-12-04|110001|001112002|0.99|0.00|0.00
>   2012-12-05|110001|001112003|0.99|0.00|0.00
>   2012-12-06|110001|001112006|0.99|1.00|0.99
>   2012-12-07|110001|001112007|0.99|0.00|0.00
>   2012-12-08|110001|001112008|0.99|0.00|0.00
>   2012-12-09|110001|001112009|0.99|0.00|0.00
>   2012-12-10|110001|001112010|0.99|0.00|0.00
>   2012-12-11|110001|001113003|0.99|0.00|0.00
>   2012-12-12|110001|001113006|0.99|0.00|0.00
>   2012-12-13|110001|001113008|0.99|0.00|0.00
>   2012-12-14|110001|001113010|0.99|0.00|0.00
>   2012-12-15|110001|001114002|0.99|0.00|0.00
>   2012-12-16|110001|001114004|0.99|1.00|0.99
>   2012-12-17|110001|001114005|0.99|0.00|0.00
>   2012-12-18|110001|001121004|0.99|0.00|0.00 
> QUERY:
> select man_date , date_offset(man_date ,5 ,'year') as expiry_date from 
> item_tab;
> RESULT:
> 2012-12-01  2017-12-01
> 2012-12-02  2017-12-02
> 2012-12-03  2017-12-03
> 2012-12-04  2017-12-04
> 2012-12-05  2017-12-05
> 2012-12-06  2017-12-06
> 2012-12-07  2017-12-07
> 2012-12-08  2017-12-08
> 2012-12-09  2017-12-09
> 2012-12-10  2017-12-10
> 2012-12-11  2017-12-11
> 2012-12-12  2017-12-12
> 2012-12-13  2017-12-13
> 2012-12-14  2017-12-14
> 2012-12-15  2017-12-15
> 2012-12-16  2017-12-16
> 2012-12-17  2017-12-17
> 2012-12-18  2017-12-18

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2439) Upgrade antlr version to 3.4

[Thiruvel Thirumoolan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528855#comment-13528855
 ] 

Thiruvel Thirumoolan commented on HIVE-2439:


Looks like Phabricator is on version 5 and arc is on version 6. Unable to post 
patch on phabricator. Phabricator doesnt seem to have any branches, so gets 
difficult trying to get it from one of their trees before the version bump.

Will try posting couple of days later.

> Upgrade antlr version to 3.4
> 
>
> Key: HIVE-2439
> URL: https://issues.apache.org/jira/browse/HIVE-2439
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Ashutosh Chauhan
> Attachments: HIVE-2439_branch9_2.patch, HIVE-2439_branch9_3.patch, 
> HIVE-2439_branch9.patch, hive-2439_incomplete.patch, HIVE-2439_trunk.patch
>
>
> Upgrade antlr version to 3.4

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3768) Document JDBC client configuration for secure clusters

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528861#comment-13528861
 ] 

Lefty Leverenz commented on HIVE-3768:
--

The patch converts the wiki page [Hive JDBC 
Interface|https://cwiki.apache.org/confluence/display/Hive/HiveJDBCInterface] 
to xml and adds a one-sentence section for secure cluster setup: 

* Hive JDBC Driver
** Integration with Pentaho
** Integration with SQuirrel SQL Client
* Hive JDBC Client
** JDBC Client Setup for a Secure Cluster
\\ To configure Hive on a secure cluster, add the directory containing 
hive-site.xml to the CLASSPATH of the JDBC client.


> Document JDBC client configuration for secure clusters
> --
>
> Key: HIVE-3768
> URL: https://issues.apache.org/jira/browse/HIVE-3768
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 0.9.0
>Reporter: Lefty Leverenz
>Assignee: Lefty Leverenz
> Fix For: 0.10.0
>
> Attachments: HIVE-3768.1.patch
>
>
> Document the JDBC client configuration required for starting Hive on a secure 
> cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #226

[Apache Jenkins Server]

See 


--
[...truncated 9916 lines...]

compile-test:
 [echo] Project: serde
[javac] Compiling 26 source files to 

[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.

create-dirs:
 [echo] Project: service
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: service

ivy-init-settings:
 [echo] Project: service

ivy-resolve:
 [echo] Project: service
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: service

compile:
 [echo] Project: service

ivy-resolve-test:
 [echo] Project: service

ivy-retrieve-test:
 [echo] Project: service

compile-test:
 [echo] Project: service
[javac] Compiling 2 source files to 


test:
 [echo] Project: hive

test-shims:
 [echo] Project: hive

test-conditions:
 [echo] Project: shims

gen-test:
 [echo] Project: shims

create-dirs:
 [echo] Project: shims
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: shims

ivy-init-settings:
 [echo] Project: shims

ivy-resolve:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: shims

compile:
 [echo] Project: shims
 [echo] Building shims 0.20

build_shims:
 [echo] Project: shims
 [echo] Compiling 

 against hadoop 0.20.2 
(

ivy-init-settings:
 [echo] Project: shims

ivy-resolve-hadoop-shim:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 


ivy-retrieve-hadoop-shim:
 [echo] Project: shims
 [echo] Building shims 0.20S

build_shims:
 [echo] Project: shims
 [echo] Compiling 

 against hadoop 1.0.0 
(

ivy-init-settings:
 [echo] Project: shims

ivy-resolve-hadoop-shim:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 


ivy-retrieve-hadoop-shim:
 [echo] Project: shims
 [echo] Building shims 0.23

build_shims:
 [echo] Project: shims
 [echo] Compiling 

 against hadoop 0.23.3 
(

[jira] [Resolved] (HIVE-887) Allow SELECT without a mapreduce job

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-887.
---

   Resolution: Fixed
Fix Version/s: 0.10.0
 Release Note: Set hive-conf hive.fetch.task.conversion to more to make use 
of this feature. Turned-off by default.

Most of this got implemented in HIVE-2925

> Allow SELECT  without a mapreduce job
> --
>
> Key: HIVE-887
> URL: https://issues.apache.org/jira/browse/HIVE-887
> Project: Hive
>  Issue Type: New Feature
> Environment: All
>Reporter: Eric Sun
>Assignee: Ning Zhang
> Fix For: 0.10.0
>
>
> I often find myself needing to take a quick look at a particular column of a 
> Hive table.
> I usually do this by doing a 
> SELECT * from  LIMIT 20;
> from the CLI.  Doing this is pretty fast since it doesn't require a mapreduce 
> job.  However, it's tough to examine just 1 or 2 columns when the table is 
> very wide.
> So, I might do
> SELECT  from  LIMIT 20;
> but it's much slower since it requires a map-reduce.  It'd be really 
> convenient if a map-reduce wasn't necessary.
> Currently a good work around is to do
> hive -e "select * from table" | cut --key=n
> but it'd be more convenient if it were built in since it alleviates the need 
> for column counting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2935) Implement HiveServer2

[Rob Weltman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529113#comment-13529113
 ] 

Rob Weltman commented on HIVE-2935:
---

[~namit] - The JIRA for the core patch part is 
https://issues.apache.org/jira/browse/HIVE-3785


> Implement HiveServer2
> -
>
> Key: HIVE-2935
> URL: https://issues.apache.org/jira/browse/HIVE-2935
> Project: Hive
>  Issue Type: New Feature
>  Components: Server Infrastructure
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>  Labels: HiveServer2
> Attachments: beelinepositive.tar.gz, HIVE-2935.1.notest.patch.txt, 
> HIVE-2935.2.notest.patch.txt, HIVE-2935.2.nothrift.patch.txt, 
> HS2-changed-files-only.patch, HS2-with-thrift-patch-rebased.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3791) starting hive with --auxpath is not same as creating auxlib

[Sudhanshu Arora (JIRA)]

Sudhanshu Arora created HIVE-3791:
-

 Summary: starting hive with --auxpath is not same as creating 
auxlib
 Key: HIVE-3791
 URL: https://issues.apache.org/jira/browse/HIVE-3791
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 0.9.0
Reporter: Sudhanshu Arora


If I give two jars in hive class path say (jar x, jar y) as hive --auxpath 
jarx:jary and issue a query Select * from test. It works fine. 
However, if I say select x from test which does a map reduce job, I get an 
error saying 
ava.io.FileNotFoundException: File file:x.jar:y.jar does not exist.

If I create an auxlib directory under hive home and put the jars in that, then 
again everything works fine.

--auxpath and auxlib should work in exactly the same way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

RE: stats19.q is failing on the current trunk?

[Kevin Wilfong]

I'm looking into it

From: Navis류승우 [navis@nexr.com]
Sent: Monday, December 10, 2012 9:48 PM
To: dev@hive.apache.org
Subject: Re: stats19.q is failing on the current trunk?

It's booked on https://issues.apache.org/jira/browse/HIVE-3783 and I
also seen this.

2012/12/11 Zhenxiao Luo :
>
> Always get this diff:
>
> [junit] diff -a 
> /home/zhenxiao/Code/hive/build/ql/test/logs/clientpositive/stats19.q.out 
> /home/zhenxiao/Code/hive/ql/src/test/results/clientpositive/stats19.q.out
> [junit] 21,22c21,22
> [junit] < Stats prefix is hashed: false
> [junit] < Stats prefix is hashed: false
> [junit] ---
> [junit] > Stats prefix is hashed: true
> [junit] > Stats prefix is hashed: true
> [junit] 284,285c284,285
> [junit] < Stats prefix is hashed: false
> [junit] < Stats prefix is hashed: false
> [junit] ---
> [junit] > Stats prefix is hashed: true
> [junit] > Stats prefix is hashed: true
>
> Will file a Jira if other people found it, too.
>
> Thanks,
> Zhenxiao

[jira] [Assigned] (HIVE-3783) stats19.q is failing on trunk

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong reassigned HIVE-3783:
---

Assignee: Kevin Wilfong

> stats19.q is failing on trunk
> -
>
> Key: HIVE-3783
> URL: https://issues.apache.org/jira/browse/HIVE-3783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Ashutosh Chauhan
>Assignee: Kevin Wilfong
>
> This test-case was introduced in HIVE-3750 and is failing since as soon as it 
> was introduced. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3785) Core hive changes for HiveServer2 implementation

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529264#comment-13529264
 ] 

Prasad Mujumdar commented on HIVE-3785:
---

Code review request on https://reviews.facebook.net/D7281

> Core hive changes for HiveServer2 implementation
> 
>
> Key: HIVE-3785
> URL: https://issues.apache.org/jira/browse/HIVE-3785
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authentication, Build Infrastructure, Configuration, 
> Thrift API
>Affects Versions: 0.10.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HS2-changed-files-only.patch
>
>
> The subtask to track changes in the core hive components for HiveServer2 
> implementation

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3783) stats19.q is failing on trunk

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529310#comment-13529310
 ] 

Kevin Wilfong commented on HIVE-3783:
-

https://reviews.facebook.net/D7287

> stats19.q is failing on trunk
> -
>
> Key: HIVE-3783
> URL: https://issues.apache.org/jira/browse/HIVE-3783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Ashutosh Chauhan
>Assignee: Kevin Wilfong
>
> This test-case was introduced in HIVE-3750 and is failing since as soon as it 
> was introduced. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3783) stats19.q is failing on trunk

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3783:


Attachment: HIVE-3783.1.patch.txt

> stats19.q is failing on trunk
> -
>
> Key: HIVE-3783
> URL: https://issues.apache.org/jira/browse/HIVE-3783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Ashutosh Chauhan
>Assignee: Kevin Wilfong
> Attachments: HIVE-3783.1.patch.txt
>
>
> This test-case was introduced in HIVE-3750 and is failing since as soon as it 
> was introduced. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3783) stats19.q is failing on trunk

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3783:


Status: Patch Available  (was: Open)

> stats19.q is failing on trunk
> -
>
> Key: HIVE-3783
> URL: https://issues.apache.org/jira/browse/HIVE-3783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Ashutosh Chauhan
>Assignee: Kevin Wilfong
> Attachments: HIVE-3783.1.patch.txt
>
>
> This test-case was introduced in HIVE-3750 and is failing since as soon as it 
> was introduced. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3783) stats19.q is failing on trunk

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529316#comment-13529316
 ] 

Kevin Wilfong commented on HIVE-3783:
-

Found one reason for the test failing in some checkouts and not others.  I've 
run the test in a directory where it was previously failing >5 times, and so 
far it's been passing consistently.

> stats19.q is failing on trunk
> -
>
> Key: HIVE-3783
> URL: https://issues.apache.org/jira/browse/HIVE-3783
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11
>Reporter: Ashutosh Chauhan
>Assignee: Kevin Wilfong
> Attachments: HIVE-3783.1.patch.txt
>
>
> This test-case was introduced in HIVE-3750 and is failing since as soon as it 
> was introduced. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1850 - Still Failing

[Apache Jenkins Server]

Changes for Build #1844
[hashutosh] HIVE-3705 : Adding authorization capability to the metastore 
(Sushanth Sowmyan via Ashutosh Chauhan)


Changes for Build #1845
[hashutosh] HIVE-3231 : msck repair should find partitions already containing 
data files (Keegan Mosley via Ashutosh Chauhan)

[hashutosh] HIVE-2691 : Specify location of log4j configuration files via 
configuration properties (Zhenxiao Luo via Ashutosh Chauhan)

[hashutosh] HIVE-2794 : Aggregations without grouping should return NULL when 
applied to partitioning column of a partitionless table (Zhenxiao Luo via 
Ashutosh Chauhan)

[hashutosh] HIVE-3780 : RetryingMetaStoreClient Should Log the Caught Exception 
(Bhushan Mandhani via Ashutosh Chauhan)

[hashutosh] HIVE-3084 : Hive CI failing due to script_broken_pipe1.q (Gunther 
Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-3760 : TestNegativeMinimrCliDriver_mapreduce_stack_trace.q 
fails on hadoop-1 (Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #1846

Changes for Build #1847
[hashutosh] HIVE-3714 : Patch: Hive's ivy internal resolvers need to use 
sourceforge for sqlline (Gopal V via Ashutosh Chauhan)


Changes for Build #1848
[hashutosh] HIVE-3782 : testCliDriver_sample_islocalmode_hook fails on hadoop-1 
(Gunther Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-2288 : Adding the oracle nvl function to the UDF (Ed Capriolo, 
Guy Doulberg via Ashutosh Chauhan)

[hashutosh] HIVE-2689 : ObjectInspectorConverters cannot convert Void types to 
Array/Map/Struct types. (Jonathan Chang via Ashutosh Chauhan)


Changes for Build #1849
[hashutosh] HIVE-3622 : reflect udf cannot find method which has arguments of 
primitive types and String, Binary, Timestamp types mixed (Navis via Ashutosh 
Chauhan)

[namit] HIVE-3401 Diversify grammar for split sampling
(Navis via namit)


Changes for Build #1850



1 tests failed.
FAILED:  org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19

Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... 
-Dtest.silent=false" to get more logs.

Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
more logs.
at junit.framework.Assert.fail(Assert.java:47)
at 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19(TestCliDriver.java:41417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)




The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1850)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1850/ to 
view the results.

Jenkins build is back to normal : Hive-0.9.1-SNAPSHOT-h0.21 #226

[Apache Jenkins Server]

See 

[jira] [Updated] (HIVE-3724) Metastore tests use hardcoded ports

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3724:
---

Fix Version/s: (was: 0.11)
   0.10.0

> Metastore tests use hardcoded ports
> ---
>
> Key: HIVE-3724
> URL: https://issues.apache.org/jira/browse/HIVE-3724
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
>Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HIVE-3724.1.patch.txt, HIVE-3724.2.patch.txt, 
> hive-3724.svn-0.10.patch
>
>
> Several of the metastore tests use hardcoded ports for remote metastore 
> Thrift servers.  This is causing transient failures in Jenkins, e.g. 
> https://builds.apache.org/job/Hive-trunk-h0.21/1804/
> A few tests already dynamically determine free ports, and this logic can be 
> shared.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3724) Metastore tests use hardcoded ports

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529406#comment-13529406
 ] 

Ashutosh Chauhan commented on HIVE-3724:


Committed to 0.10 branch. Thanks, Sushanth for backport!

> Metastore tests use hardcoded ports
> ---
>
> Key: HIVE-3724
> URL: https://issues.apache.org/jira/browse/HIVE-3724
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
>Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HIVE-3724.1.patch.txt, HIVE-3724.2.patch.txt, 
> hive-3724.svn-0.10.patch
>
>
> Several of the metastore tests use hardcoded ports for remote metastore 
> Thrift servers.  This is causing transient failures in Jenkins, e.g. 
> https://builds.apache.org/job/Hive-trunk-h0.21/1804/
> A few tests already dynamically determine free ports, and this logic can be 
> shared.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3705) Adding authorization capability to the metastore

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3705:
---

Fix Version/s: (was: 0.11)
   0.10.0

> Adding authorization capability to the metastore
> 
>
> Key: HIVE-3705
> URL: https://issues.apache.org/jira/browse/HIVE-3705
> Project: Hive
>  Issue Type: New Feature
>  Components: Authorization, Metastore
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 0.10.0
>
> Attachments: HIVE-3705.D6681.1.patch, HIVE-3705.D6681.2.patch, 
> HIVE-3705.D6681.3.patch, HIVE-3705.D6681.4.patch, HIVE-3705.D6681.5.patch, 
> HIVE-3705.giant.svn-0.10.patch, HIVE-3705.giant.svn.patch, 
> hive-backend-auth.2.git.patch, hive-backend-auth.git.patch, 
> hive-backend-auth.post-review.git.patch, 
> hive-backend-auth.post-review-part2.git.patch, 
> hive-backend-auth.post-review-part3.git.patch, hivesec_investigation.pdf
>
>
> In an environment where multiple clients access a single metastore, and we 
> want to evolve hive security to a point where it's no longer simply 
> preventing users from shooting their own foot, we need to be able to 
> authorize metastore calls as well, instead of simply performing every 
> metastore api call that's made.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3705) Adding authorization capability to the metastore

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529478#comment-13529478
 ] 

Ashutosh Chauhan commented on HIVE-3705:


Committed to 0.10 branch. Thanks, Sushanth!

> Adding authorization capability to the metastore
> 
>
> Key: HIVE-3705
> URL: https://issues.apache.org/jira/browse/HIVE-3705
> Project: Hive
>  Issue Type: New Feature
>  Components: Authorization, Metastore
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 0.10.0
>
> Attachments: HIVE-3705.D6681.1.patch, HIVE-3705.D6681.2.patch, 
> HIVE-3705.D6681.3.patch, HIVE-3705.D6681.4.patch, HIVE-3705.D6681.5.patch, 
> HIVE-3705.giant.svn-0.10.patch, HIVE-3705.giant.svn.patch, 
> hive-backend-auth.2.git.patch, hive-backend-auth.git.patch, 
> hive-backend-auth.post-review.git.patch, 
> hive-backend-auth.post-review-part2.git.patch, 
> hive-backend-auth.post-review-part3.git.patch, hivesec_investigation.pdf
>
>
> In an environment where multiple clients access a single metastore, and we 
> want to evolve hive security to a point where it's no longer simply 
> preventing users from shooting their own foot, we need to be able to 
> authorize metastore calls as well, instead of simply performing every 
> metastore api call that's made.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3789) Patch HIVE-3648 causing the majority of unit tests to fail on branch 0.9

[Arup Malakar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529499#comment-13529499
 ] 

Arup Malakar commented on HIVE-3789:


Chris, I am seeing the errors too, I am investigating it. Meanwhile if someone 
has any clue on the following exception that I am seeing for the test failures:

{code}
[junit] Running org.apache.hadoop.hive.ql.parse.TestContribParse
[junit] Cleaning up TestContribParse
[junit] Exception: MetaException(message:Got exception: 
java.lang.IllegalArgumentException Wrong FS: 
pfile:/Users/malakar/code/oss/hive_09/hive/build/contrib/test/data/warehouse/src_json,
 expected: file:///)
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 4.433 sec
[junit] org.apache.hadoop.hive.ql.metadata.HiveException: 
MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong 
FS: 
pfile:/Users/malakar/code/oss/hive_09/hive/build/contrib/test/data/warehouse/src_json,
 expected: file:///)
[junit] at 
org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:813)
[junit] at 
org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:789)
[junit] at 
org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:421)
[junit] at 
org.apache.hadoop.hive.ql.QTestUtil.shutdown(QTestUtil.java:278)
[junit] at 
org.apache.hadoop.hive.ql.parse.TestContribParse.tearDown(TestContribParse.java:59)
[junit] at junit.framework.TestCase.runBare(TestCase.java:140)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:110)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:128)
[junit] at junit.framework.TestResult.run(TestResult.java:113)
[junit] at junit.framework.TestCase.run(TestCase.java:124)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:243)
[junit] at junit.framework.TestSuite.run(TestSuite.java:238)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
[junit] Caused by: MetaException(message:Got exception: 
java.lang.IllegalArgumentException Wrong FS: 
pfile:/Users/malakar/code/oss/hive_09/hive/build/contrib/test/data/warehouse/src_json,
 expected: file:///)
[junit] at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:785)
[junit] at 
org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:61)
[junit] at 
org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:200)
[junit] at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:929)
[junit] at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table(HiveMetaStore.java:944)
[junit] at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:553)
[junit] at 
org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:807)
[junit] ... 14 more
[junit] Test org.apache.hadoop.hive.ql.parse.TestContribParse FAILED
  [for] /Users/malakar/code/oss/hive_09/hive/contrib/build.xml: The 
following error occurred while executing this line:
  [for] /Users/malakar/code/oss/hive_09/hive/build.xml:321: The following 
error occurred while executing this line:
  [for] /Users/malakar/code/oss/hive_09/hive/build-common.xml:448: Tests 
failed!
{code}

> Patch HIVE-3648 causing the majority of unit tests to fail on branch 0.9
> 
>
> Key: HIVE-3789
> URL: https://issues.apache.org/jira/browse/HIVE-3789
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Tests
>Affects Versions: 0.9.0
> Environment: Hadooop 0.23.5, JDK 1.6.0_31
>Reporter: Chris Drome
>
> Rolling back to before this patch shows that the unit tests are passing, 
> after the patch, the majority of the unit tests are failing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3792) hive jars are not part of the share lib tar ball in oozie

[Ashish Singh (JIRA)]

Ashish Singh created HIVE-3792:
--

 Summary: hive jars are not part of the share lib tar ball in oozie
 Key: HIVE-3792
 URL: https://issues.apache.org/jira/browse/HIVE-3792
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 0.10.0
Reporter: Ashish Singh
Assignee: Ashish Singh
 Fix For: 0.10.0




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3792) hive jars are not part of the share lib tar ball in oozie

[Ashish Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singh updated HIVE-3792:
---

Description: hive-0.10.0 pom file has missing conf and scope mapping for 
compile configuration. 

> hive jars are not part of the share lib tar ball in oozie
> -
>
> Key: HIVE-3792
> URL: https://issues.apache.org/jira/browse/HIVE-3792
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.10.0
>Reporter: Ashish Singh
>Assignee: Ashish Singh
> Fix For: 0.10.0
>
>
> hive-0.10.0 pom file has missing conf and scope mapping for compile 
> configuration. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3780) RetryingMetaStoreClient Should Log the Caught Exception

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3780:
---

Fix Version/s: (was: 0.11)
   0.10.0

> RetryingMetaStoreClient Should Log the Caught Exception
> ---
>
> Key: HIVE-3780
> URL: https://issues.apache.org/jira/browse/HIVE-3780
> Project: Hive
>  Issue Type: Bug
>Reporter: Bhushan Mandhani
>Assignee: Bhushan Mandhani
>Priority: Trivial
> Fix For: 0.10.0
>
> Attachments: HIVE-3780.1.patch.txt
>
>
> Currently it logs the cause of the caught exception. It should log the caught 
> exception itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3780) RetryingMetaStoreClient Should Log the Caught Exception

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529530#comment-13529530
 ] 

Ashutosh Chauhan commented on HIVE-3780:


Applied to 0.10 branch as well.

> RetryingMetaStoreClient Should Log the Caught Exception
> ---
>
> Key: HIVE-3780
> URL: https://issues.apache.org/jira/browse/HIVE-3780
> Project: Hive
>  Issue Type: Bug
>Reporter: Bhushan Mandhani
>Assignee: Bhushan Mandhani
>Priority: Trivial
> Fix For: 0.10.0
>
> Attachments: HIVE-3780.1.patch.txt
>
>
> Currently it logs the cause of the caught exception. It should log the caught 
> exception itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3792) hive jars are not part of the share lib tar ball in oozie

[Ashish Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singh updated HIVE-3792:
---

Attachment: HIVE-3792.patch

> hive jars are not part of the share lib tar ball in oozie
> -
>
> Key: HIVE-3792
> URL: https://issues.apache.org/jira/browse/HIVE-3792
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.10.0
>Reporter: Ashish Singh
>Assignee: Ashish Singh
> Fix For: 0.10.0
>
> Attachments: HIVE-3792.patch
>
>
> hive-0.10.0 pom file has missing conf and scope mapping for compile 
> configuration. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3792) hive jars are not part of the share lib tar ball in oozie

[Ashish Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singh updated HIVE-3792:
---

Status: Patch Available  (was: Open)

Attaching the patch, that has the mapping of conf to scope for compile 
configuration.

> hive jars are not part of the share lib tar ball in oozie
> -
>
> Key: HIVE-3792
> URL: https://issues.apache.org/jira/browse/HIVE-3792
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.10.0
>Reporter: Ashish Singh
>Assignee: Ashish Singh
> Fix For: 0.10.0
>
> Attachments: HIVE-3792.patch
>
>
> hive-0.10.0 pom file has missing conf and scope mapping for compile 
> configuration. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3787) Regression introduced from HIVE-3401

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-3787:


Status: Patch Available  (was: Open)

> Regression introduced from HIVE-3401
> 
>
> Key: HIVE-3787
> URL: https://issues.apache.org/jira/browse/HIVE-3787
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-3787.D7275.1.patch
>
>
> By HIVE-3562, split_sample_out_of_range.q and split_sample_wrong_format.q are 
> not showing valid 'line:loc' information for error messages.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3045) Partition column values are not valid if any of virtual columns is selected

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-3045:


Resolution: Duplicate
Status: Resolved  (was: Patch Available)

Yes, it's fixed by HIVE-2925. 

> Partition column values are not valid if any of virtual columns is selected
> ---
>
> Key: HIVE-3045
> URL: https://issues.apache.org/jira/browse/HIVE-3045
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-3045.D3351.2.patch
>
>
> For example,
> {code}
> hive> select * from srcpart where key < 5;
> 0 val_0   2008-04-08  11
> 4 val_4   2008-04-08  11
> 0 val_0   2008-04-08  11
> 0 val_0   2008-04-08  11
> 2 val_2   2008-04-08  11
> 0 val_1   2008-04-09  12
> 4 val_5   2008-04-09  12
> 3 val_4   2008-04-09  12
> 2 val_3   2008-04-09  12
> 0 val_1   2008-04-09  12
> 1 val_2   2008-04-09  12
> hive> select *, BLOCK__OFFSET__INSIDE__FILE from srcpart where key < 5;
> 0 val_0   2008-04-09  11  968
> 4 val_4   2008-04-09  11  1218
> 0 val_0   2008-04-09  11  2088
> 0 val_0   2008-04-09  11  2632
> 2 val_2   2008-04-09  11  4004
> 0 val_1   2008-04-09  11  682
> 4 val_5   2008-04-09  11  1131
> 3 val_4   2008-04-09  11  1163
> 2 val_3   2008-04-09  11  2629
> 0 val_1   2008-04-09  11  4367
> 1 val_2   2008-04-09  11  5669
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3045) Partition column values are not valid if any of virtual columns is selected

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529559#comment-13529559
 ] 

Phabricator commented on HIVE-3045:
---

navis has abandoned the revision "HIVE-3045 [jira] Partition column values are 
not valid if any of virtual columns is selected".

  Fixed by HIVE-2925

REVISION DETAIL
  https://reviews.facebook.net/D3351

To: JIRA, navis


> Partition column values are not valid if any of virtual columns is selected
> ---
>
> Key: HIVE-3045
> URL: https://issues.apache.org/jira/browse/HIVE-3045
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-3045.D3351.2.patch
>
>
> For example,
> {code}
> hive> select * from srcpart where key < 5;
> 0 val_0   2008-04-08  11
> 4 val_4   2008-04-08  11
> 0 val_0   2008-04-08  11
> 0 val_0   2008-04-08  11
> 2 val_2   2008-04-08  11
> 0 val_1   2008-04-09  12
> 4 val_5   2008-04-09  12
> 3 val_4   2008-04-09  12
> 2 val_3   2008-04-09  12
> 0 val_1   2008-04-09  12
> 1 val_2   2008-04-09  12
> hive> select *, BLOCK__OFFSET__INSIDE__FILE from srcpart where key < 5;
> 0 val_0   2008-04-09  11  968
> 4 val_4   2008-04-09  11  1218
> 0 val_0   2008-04-09  11  2088
> 0 val_0   2008-04-09  11  2632
> 2 val_2   2008-04-09  11  4004
> 0 val_1   2008-04-09  11  682
> 4 val_5   2008-04-09  11  1131
> 3 val_4   2008-04-09  11  1163
> 2 val_3   2008-04-09  11  2629
> 0 val_1   2008-04-09  11  4367
> 1 val_2   2008-04-09  11  5669
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3045) Partition column values are not valid if any of virtual columns is selected

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3045:
---

Fix Version/s: 0.10.0

> Partition column values are not valid if any of virtual columns is selected
> ---
>
> Key: HIVE-3045
> URL: https://issues.apache.org/jira/browse/HIVE-3045
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Fix For: 0.10.0
>
> Attachments: HIVE-3045.D3351.2.patch
>
>
> For example,
> {code}
> hive> select * from srcpart where key < 5;
> 0 val_0   2008-04-08  11
> 4 val_4   2008-04-08  11
> 0 val_0   2008-04-08  11
> 0 val_0   2008-04-08  11
> 2 val_2   2008-04-08  11
> 0 val_1   2008-04-09  12
> 4 val_5   2008-04-09  12
> 3 val_4   2008-04-09  12
> 2 val_3   2008-04-09  12
> 0 val_1   2008-04-09  12
> 1 val_2   2008-04-09  12
> hive> select *, BLOCK__OFFSET__INSIDE__FILE from srcpart where key < 5;
> 0 val_0   2008-04-09  11  968
> 4 val_4   2008-04-09  11  1218
> 0 val_0   2008-04-09  11  2088
> 0 val_0   2008-04-09  11  2632
> 2 val_2   2008-04-09  11  4004
> 0 val_1   2008-04-09  11  682
> 4 val_5   2008-04-09  11  1131
> 3 val_4   2008-04-09  11  1163
> 2 val_3   2008-04-09  11  2629
> 0 val_1   2008-04-09  11  4367
> 1 val_2   2008-04-09  11  5669
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3766) Enable adding hooks to hive meta store init

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529581#comment-13529581
 ] 

Kevin Wilfong commented on HIVE-3766:
-

METASTORETHRIFTRETRIES has been changed to METASTORETHRIFTCONNECTIONRETRIES can 
you update the test?

> Enable adding hooks to hive meta store init
> ---
>
> Key: HIVE-3766
> URL: https://issues.apache.org/jira/browse/HIVE-3766
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Jean Xu
>Assignee: Jean Xu
> Attachments: jira3766.txt
>
>
> We will enable hooks to be added to init HMSHandler

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3300) LOAD DATA INPATH fails if a hdfs file with same name is added to table

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3300:
--

Attachment: HIVE-3300.D4383.3.patch

navis updated the revision "HIVE-3300 [jira] LOAD DATA INPATH fails if a hdfs 
file with same name is added to table".
Reviewers: JIRA

  1. Removed staging stage for source files before copy/move
  2. Rebase on trunk


REVISION DETAIL
  https://reviews.facebook.net/D4383

AFFECTED FILES
  build-common.xml
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
  ql/src/test/queries/clientpositive/load_fs2.q
  ql/src/test/results/clientpositive/load_fs2.q.out

To: JIRA, navis


> LOAD DATA INPATH fails if a hdfs file with same name is added to table
> --
>
> Key: HIVE-3300
> URL: https://issues.apache.org/jira/browse/HIVE-3300
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export
>Affects Versions: 0.10.0
> Environment: ubuntu linux, hadoop 1.0.3, hive 0.9
>Reporter: Bejoy KS
>Assignee: Navis
> Attachments: HIVE-3300.1.patch.txt, HIVE-3300.D4383.3.patch
>
>
> If we are loading data from local fs to hive tables using 'LOAD DATA LOCAL 
> INPATH' and if a file with the same name exists in the table's location then 
> the new file will be suffixed by *_copy_1.
> But if we do the 'LOAD DATA INPATH'  for a file in hdfs then there is no 
> rename happening but just a move task is getting triggered. Since a file with 
> same name exists in same hdfs location, hadoop fs move operation throws an 
> error.
> hive> LOAD DATA INPATH '/userdata/bejoy/site.txt' INTO TABLE test.site;
> Loading data to table test.site
> Failed with exception null
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask
> hive> 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3790) UDF to introduce an OFFSET(day,month or year) for a given date or timestamp

[Sun Rui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529609#comment-13529609
 ] 

Sun Rui commented on HIVE-3790:
---

Have you considered adding support to the SQL interval type? So, for example, 
if you want to add '1 year 3 month' to a date, you don't have to have two 
calls, first is to call date1=date_offset(date, 1, 'YEAR'), then to call 
date_offset (date1,3, 'MONTH').

> UDF to introduce an OFFSET(day,month or year) for a given date or timestamp 
> 
>
> Key: HIVE-3790
> URL: https://issues.apache.org/jira/browse/HIVE-3790
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Jithin John
>
> Current releases of Hive lacks a  generic function which would find the date 
> offset to a date / timestamp. Current releases have date_add (date) and 
> date_sub(date) which allows user to add or substract days only.But we could 
> not use year or month as a unit.
> 
> The Function DATE_OFFSET(date,offset,unit) returns the date offset value from 
> start_date according to the unit. Here the unit can be year , month and day.
> The function could be used for date range queries and is more flexible than 
> the existing functions.
> Functionality :-
> Function Name: DATE_OFFSET(date,offset,unit)
>
> Add a offset value to the unit part of the date/timestamp.
> Returns  the date in the format of "-MM-dd" .
> Example: hive> select date_offset('2009-07-29', -1 ,'MONTH' ) FROM src LIMIT 1
> -> 2009-06-29
> Usage :-
> Case  : To calculate the expiry date of a item from manufacturing date 
> Table :- ITEM_TAB
>  Manufacturing_date  |item id|store id|value|unit|price
>   2012-12-01|110001|00003|0.99|1.00|0.99
>   2012-12-02|110001|00008|0.99|0.00|0.00
>   2012-12-03|110001|00009|0.99|0.00|0.00
>   2012-12-04|110001|001112002|0.99|0.00|0.00
>   2012-12-05|110001|001112003|0.99|0.00|0.00
>   2012-12-06|110001|001112006|0.99|1.00|0.99
>   2012-12-07|110001|001112007|0.99|0.00|0.00
>   2012-12-08|110001|001112008|0.99|0.00|0.00
>   2012-12-09|110001|001112009|0.99|0.00|0.00
>   2012-12-10|110001|001112010|0.99|0.00|0.00
>   2012-12-11|110001|001113003|0.99|0.00|0.00
>   2012-12-12|110001|001113006|0.99|0.00|0.00
>   2012-12-13|110001|001113008|0.99|0.00|0.00
>   2012-12-14|110001|001113010|0.99|0.00|0.00
>   2012-12-15|110001|001114002|0.99|0.00|0.00
>   2012-12-16|110001|001114004|0.99|1.00|0.99
>   2012-12-17|110001|001114005|0.99|0.00|0.00
>   2012-12-18|110001|001121004|0.99|0.00|0.00 
> QUERY:
> select man_date , date_offset(man_date ,5 ,'year') as expiry_date from 
> item_tab;
> RESULT:
> 2012-12-01  2017-12-01
> 2012-12-02  2017-12-02
> 2012-12-03  2017-12-03
> 2012-12-04  2017-12-04
> 2012-12-05  2017-12-05
> 2012-12-06  2017-12-06
> 2012-12-07  2017-12-07
> 2012-12-08  2017-12-08
> 2012-12-09  2017-12-09
> 2012-12-10  2017-12-10
> 2012-12-11  2017-12-11
> 2012-12-12  2017-12-12
> 2012-12-13  2017-12-13
> 2012-12-14  2017-12-14
> 2012-12-15  2017-12-15
> 2012-12-16  2017-12-16
> 2012-12-17  2017-12-17
> 2012-12-18  2017-12-18

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3793) Print number of fetched rows after query in CliDriver

[Navis (JIRA)]

Navis created HIVE-3793:
---

 Summary: Print number of fetched rows after query in CliDriver
 Key: HIVE-3793
 URL: https://issues.apache.org/jira/browse/HIVE-3793
 Project: Hive
  Issue Type: Improvement
  Components: CLI
Reporter: Navis
Assignee: Navis
Priority: Trivial


Currently shows time taken only. But it would be useful showing number of rows 
fetched also. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3793) Print number of fetched rows after query in CliDriver

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3793:
--

Attachment: HIVE-3793.D7305.1.patch

navis requested code review of "HIVE-3793 [jira] Print number of fetched rows 
after query in CliDriver".
Reviewers: JIRA

  DPAL-1942 Print number of fetched rows after query in CliDriver

  Currently shows time taken only. But it would be useful showing number of 
rows fetched also.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D7305

AFFECTED FILES
  cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/17403/

To: JIRA, navis


> Print number of fetched rows after query in CliDriver
> -
>
> Key: HIVE-3793
> URL: https://issues.apache.org/jira/browse/HIVE-3793
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3793.D7305.1.patch
>
>
> Currently shows time taken only. But it would be useful showing number of 
> rows fetched also. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3793) Print number of fetched rows after query in CliDriver

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-3793:


Status: Patch Available  (was: Open)

> Print number of fetched rows after query in CliDriver
> -
>
> Key: HIVE-3793
> URL: https://issues.apache.org/jira/browse/HIVE-3793
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3793.D7305.1.patch
>
>
> Currently shows time taken only. But it would be useful showing number of 
> rows fetched also. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3420) Inefficiency in hbase handler when process query including rowkey range scan

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529641#comment-13529641
 ] 

Navis commented on HIVE-3420:
-

@Gang Deng 
This is pretty important issue. I'll make a patch for a review.

> Inefficiency in hbase handler when process query including rowkey range scan
> 
>
> Key: HIVE-3420
> URL: https://issues.apache.org/jira/browse/HIVE-3420
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.9.0
> Environment: Hive-0.9.0 + HBase-0.94.1
>Reporter: Gang Deng
>Priority: Critical
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> When query hive with hbase rowkey range, hive map tasks do not leverage 
> startrow, endrow information in tablesplit. For example, if the rowkeys fit 
> into 5 hbase files, then where will be 5 map tasks. Ideally, each task will 
> process 1 file. But in current implementation, each task processes 5 files 
> repeatedly. The behavior not only waste network bandwidth, but also worse the 
> lock contention in HBase block cache as each task have to access the same 
> block. The problem code is in HiveHBaseTableInputFormat.convertFilte as below:
> ……
> if (tableSplit != null) {
>   tableSplit = new TableSplit(
> tableSplit.getTableName(),
> startRow,
> stopRow,
> tableSplit.getRegionLocation());
> }
> scan.setStartRow(startRow);
> scan.setStopRow(stopRow);
> ……
> As tableSplit already include startRow, endRow information of file, the 
> better implementation will be:
> ……
> byte[] splitStart = startRow;
> byte[] splitStop = stopRow;
> if (tableSplit != null) {
> 
>if(tableSplit.getStartRow() != null){
> splitStart = startRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getStartRow(), startRow) >= 0 ?
> tableSplit.getStartRow() : startRow;
> }
> if(tableSplit.getEndRow() != null){
> splitStop = (stopRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getEndRow(), stopRow) <= 0) &&
>   tableSplit.getEndRow().length > 0 ?
> tableSplit.getEndRow() : stopRow;
> }   
>   tableSplit = new TableSplit(
> tableSplit.getTableName(),
> splitStart,
> splitStop,
> tableSplit.getRegionLocation());
> }
> scan.setStartRow(splitStart);
> scan.setStopRow(splitStop);
> ……
> In my test, the changed code will improve performance more than 30%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3672) Support altering partition column type in Hive

[Jingwei Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingwei Lu updated HIVE-3672:
-

Attachment: HIVE-3672.2.patch.txt

New patch fixed according to comments.

> Support altering partition column type in Hive
> --
>
> Key: HIVE-3672
> URL: https://issues.apache.org/jira/browse/HIVE-3672
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI, SQL
>Affects Versions: 0.10.0
>Reporter: Jingwei Lu
>Assignee: Jingwei Lu
> Fix For: 0.10.0
>
> Attachments: HIVE-3672.1.patch.txt, HIVE-3672.2.patch.txt
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Currently, Hive does not allow altering partition column types.  As we've 
> discouraged users from using non-string partition column types, this presents 
> a problem for users who want to change there partition columns to be strings, 
> they have to rename their table, create a new table, and copy all the data 
> over.
> To support this via the CLI, adding a command like ALTER TABLE  
> PARTITION COLUMN ( );

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3420) Inefficiency in hbase handler when process query including rowkey range scan

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-3420:


Affects Version/s: (was: 0.9.0)
   Status: Patch Available  (was: Open)

> Inefficiency in hbase handler when process query including rowkey range scan
> 
>
> Key: HIVE-3420
> URL: https://issues.apache.org/jira/browse/HIVE-3420
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
> Environment: Hive-0.9.0 + HBase-0.94.1
>Reporter: Gang Deng
>Priority: Critical
> Attachments: HIVE-3420.D7311.1.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> When query hive with hbase rowkey range, hive map tasks do not leverage 
> startrow, endrow information in tablesplit. For example, if the rowkeys fit 
> into 5 hbase files, then where will be 5 map tasks. Ideally, each task will 
> process 1 file. But in current implementation, each task processes 5 files 
> repeatedly. The behavior not only waste network bandwidth, but also worse the 
> lock contention in HBase block cache as each task have to access the same 
> block. The problem code is in HiveHBaseTableInputFormat.convertFilte as below:
> ……
> if (tableSplit != null) {
>   tableSplit = new TableSplit(
> tableSplit.getTableName(),
> startRow,
> stopRow,
> tableSplit.getRegionLocation());
> }
> scan.setStartRow(startRow);
> scan.setStopRow(stopRow);
> ……
> As tableSplit already include startRow, endRow information of file, the 
> better implementation will be:
> ……
> byte[] splitStart = startRow;
> byte[] splitStop = stopRow;
> if (tableSplit != null) {
> 
>if(tableSplit.getStartRow() != null){
> splitStart = startRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getStartRow(), startRow) >= 0 ?
> tableSplit.getStartRow() : startRow;
> }
> if(tableSplit.getEndRow() != null){
> splitStop = (stopRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getEndRow(), stopRow) <= 0) &&
>   tableSplit.getEndRow().length > 0 ?
> tableSplit.getEndRow() : stopRow;
> }   
>   tableSplit = new TableSplit(
> tableSplit.getTableName(),
> splitStart,
> splitStop,
> tableSplit.getRegionLocation());
> }
> scan.setStartRow(splitStart);
> scan.setStopRow(splitStop);
> ……
> In my test, the changed code will improve performance more than 30%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3420) Inefficiency in hbase handler when process query including rowkey range scan

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3420:
--

Attachment: HIVE-3420.D7311.1.patch

navis requested code review of "HIVE-3420 [jira] Inefficiency in hbase handler 
when process query including rowkey range scan".
Reviewers: JIRA

  DPAL-1943 Inefficiency in hbase handler when process query including rowkey 
range scan

  When query hive with hbase rowkey range, hive map tasks do not leverage 
startrow, endrow information in tablesplit. For example, if the rowkeys fit 
into 5 hbase files, then where will be 5 map tasks. Ideally, each task will 
process 1 file. But in current implementation, each task processes 5 files 
repeatedly. The behavior not only waste network bandwidth, but also worse the 
lock contention in HBase block cache as each task have to access the same 
block. The problem code is in HiveHBaseTableInputFormat.convertFilte as below:
  ……
  if (tableSplit != null)
  {
tableSplit = new TableSplit(
  tableSplit.getTableName(),
  startRow,
  stopRow,
  tableSplit.getRegionLocation());
  }
  scan.setStartRow(startRow);
  scan.setStopRow(stopRow);
  ……
  As tableSplit already include startRow, endRow information of file, the 
better implementation will be:

  ……
  byte[] splitStart = startRow;
  byte[] splitStop = stopRow;
  if (tableSplit != null) {

 if(tableSplit.getStartRow() != null)
  {
  splitStart = startRow.length == 0 ||
Bytes.compareTo(tableSplit.getStartRow(), startRow) >= 0 ?
  tableSplit.getStartRow() : startRow;
  }
  if(tableSplit.getEndRow() != null)
  {
  splitStop = (stopRow.length == 0 ||
Bytes.compareTo(tableSplit.getEndRow(), stopRow) <= 0) &&
tableSplit.getEndRow().length > 0 ?
  tableSplit.getEndRow() : stopRow;
  }

tableSplit = new TableSplit(
  tableSplit.getTableName(),
  splitStart,
  splitStop,
  tableSplit.getRegionLocation());
  }
  scan.setStartRow(splitStart);
  scan.setStopRow(splitStop);
  ……
  In my test, the changed code will improve performance more than 30%.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D7311

AFFECTED FILES
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/17415/

To: JIRA, navis


> Inefficiency in hbase handler when process query including rowkey range scan
> 
>
> Key: HIVE-3420
> URL: https://issues.apache.org/jira/browse/HIVE-3420
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
> Environment: Hive-0.9.0 + HBase-0.94.1
>Reporter: Gang Deng
>Priority: Critical
> Attachments: HIVE-3420.D7311.1.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> When query hive with hbase rowkey range, hive map tasks do not leverage 
> startrow, endrow information in tablesplit. For example, if the rowkeys fit 
> into 5 hbase files, then where will be 5 map tasks. Ideally, each task will 
> process 1 file. But in current implementation, each task processes 5 files 
> repeatedly. The behavior not only waste network bandwidth, but also worse the 
> lock contention in HBase block cache as each task have to access the same 
> block. The problem code is in HiveHBaseTableInputFormat.convertFilte as below:
> ……
> if (tableSplit != null) {
>   tableSplit = new TableSplit(
> tableSplit.getTableName(),
> startRow,
> stopRow,
> tableSplit.getRegionLocation());
> }
> scan.setStartRow(startRow);
> scan.setStopRow(stopRow);
> ……
> As tableSplit already include startRow, endRow information of file, the 
> better implementation will be:
> ……
> byte[] splitStart = startRow;
> byte[] splitStop = stopRow;
> if (tableSplit != null) {
> 
>if(tableSplit.getStartRow() != null){
> splitStart = startRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getStartRow(), startRow) >= 0 ?
> tableSplit.getStartRow() : startRow;
> }
> if(tableSplit.getEndRow() != null){
> splitStop = (stopRow.length == 0 ||
>   Bytes.compareTo(tableSplit.getEndRow(), stopRow) <= 0) &&
>   tableSplit.getEndRow().length > 0 ?
> tableSplit.getEndRow() : stopRow;
> }