How to convert Array to String in Hive0.7

2013-06-14 Thread lxw 
Hi, all :

How to convert Array to String in Hive0.7?

concat_ws(string SEP, array)   Like concat_ws() above, but taking 
an array of strings. (as of Hive 0.9.0) 

this function was not supported,who can help me?


thanks!

Re:working with HIVE VARIALBE: Pls suggest

2014-01-05 Thread lxw 
maybe you can see:

https://cwiki.apache.org/confluence/plugins/viewsource/viewpagesrc.action?pageId=30754722




-- Original --
From:  "yogesh dhari";;
Date:  Fri, Jan 3, 2014 01:13 AM
To:  "user"; 

Subject:  working with HIVE VARIALBE: Pls suggest



Hello Hive Champs,
 
 
 I have a case statement, where I need to check the date passed through 
parameter,
 
 If date is 1st date of the month then keep it as it as
else
set the parameter date to 1st date of the month.
  
and then later opretation are being performed on that date into hive quries,
  
 
I have wrote this Hive QL
 
 select case when as_of_dt = ${hivevar:as_of_dt} then ${hivevar:as_of_dt} else 
date_sub(${hivevar:as_of_dt} , (day(${hivevar:as_of_dt} )) -1 ) end as as_of_dt 
from TABLE group by as_of_dt ;
  
 O/P of this query is, lets say =  2012-08-01


 I want to store the value of this Query into a variable.
 
 like 

MY_VARIABLE = (select case when as_of_dt = ${hivevar:as_of_dt} then 
${hivevar:as_of_dt} else date_sub(${hivevar:as_of_dt} , 
(day(${hivevar:as_of_dt} )) -1 ) end as as_of_dt from TABLE group by as_of_dt; )
 

 



  
How to achieve that.
  
Pls suggest,
 Thanks in advance

Storage index table in HBase

2014-07-09 Thread lxw 
Hi,experts:

  I want to storage hive index table in HBase to improve index performance.
I do it like this :


  CREATE INDEX idx_number on table my_hive_table (number) as  
'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' with DEFERRED 
REBUILD 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
WITH 
SERDEPROPERTIES(??hbase.columns.mapping??=??:key,cf:bucketname,cf:offsets??)??

  but it fails, exception message is:
org.apache.hadoop.hive.ql.metadata.HiveException:must specify an InputFormat 
class.

 Who can help me to resolve this problem or tell me what should I do.

Thanks all!
Sean

Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException "Encountered unregistered class ID: 21"

2016-03-30 Thread lxw 
at 
java.security.AccessController.doPrivileged(Native Method)   at 
javax.security.auth.Subject.doAs(Subject.java:415)   at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused 
by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered 
unregistered class ID: 21 Serialization trace: outputFileFormatClass 
(org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo 
(org.apache.hadoop.hive.ql.plan.MapWork)at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238)
   at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226)
   at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745)  
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:333) 
... 13 more  Is any expert can help me?
 
Thanks!
lxw

?????? Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException "Encounteredunregistered class ID: 21"

2016-03-30 Thread lxw 
Thank you for reply!

I use default execution engine "mr" (hive.execution.engine=mr) to run HQL,
and Hive 0.13 works fine, it also deserializing MapWork plan via Kryo. 




--  --
??: "Mich Talebzadeh";;
: 2016??3??31??(??) 0:32
??: "user"; 

: Re: Hive2.0+Hadoop-2.3.0-cdh5.0 encounted KryoException 
"Encounteredunregistered class ID: 21"



Hive 2 works fine. I don't use any vendor specific build but it should work 
pretty easy.


What engine (mr, spark, TZ) have you specified in hive-site.xml. Kind of 
reminds me some error I saw in the past. Try to set hive.execution.engine=mr 
and run it again.




HTH


 
Dr Mich Talebzadeh
 
 
 
LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 
 
 
http://talebzadehmich.wordpress.com
 
 
 


 
On 30 March 2016 at 10:08, lxw  wrote:
Hi,experts:

 When I  use Hive 2.0 and Hadoop-2.3.0-cdh5.0 to run HQL like "SELECT COUNT(1) 
FROM myTable",
and encounted this exception below??(by the way,the configuration 
hive.plan.serialization.format was removed in Hive2.0)
2016-03-30 16:30:42,799 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: 
Deserializing MapWork via kryo 2016-03-30 16:30:42,880 ERROR [main] 
org.apache.hadoop.hive.ql.exec.Utilities: Failed to load plan: 
hdfs://cdh5/tmp/hive/lxw1234/c727552f-1f13-4f7d-897e-a7a737bd3683/hive_2016-03-30_16-30-12_861_5586529713231981443-1/-mr-10005/f57aeb75-94e0-4ae8-8ea3-55f5b5a1c1a9/map.xml
 org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered 
unregistered class ID: 21 Serialization trace: outputFileFormatClass 
(org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo 
(org.apache.hadoop.hive.ql.plan.MapWork) at 
org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)  at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238)
   at 
org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226)
   at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745)  
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
  at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at 
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
 at 
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
 at 
org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840)
 at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:333) at 
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:275)  at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)  at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
 at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
 at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
 at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:168)   
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)  at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)   at 
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at 
java.security.AccessController.doPrivileged(Native Method)   at 
javax.security.auth.Subject.doAs(Subject.java:415)   at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) 
2016-03-30 16:30:42,882 WARN [main] org.apache.hadoop.mapred.YarnChild: 
Exception running child : java.lang.RuntimeException: 
org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered 
unregistered class ID: 21 Serialization trace: outputFileFormatClass 
(org.apa