USE CASE:: Hierarchical Structure in Hive and Java

2014-10-21 Thread yogesh dhari
Hello All, We are having a use case where we need to create the hierarchical structure using Hive and Java For example Lets say in an organisation we need to create Org chart i.e. Senior director -> director -> associate director -> senior manager -> manager -> senior associate -> associate -> De

Maven build error while building hive 0.14

2014-10-21 Thread Niranjan Sarvi
Hello, I needed to use 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' as this is only available from the 0.14 onwords. I though to get the source and compile hive 0.14. I installed the cygwin on my windows laptop so that bash program can work and maven can run the bash as well. So, afte

Re: String to Timestamp conversion bug

2014-10-21 Thread Jason Dere
FYI it looks like this has to do with the representation of timestamps during vectorized execution - timestamps are represented as a long value representing nanoseconds since the epoch, thus the max timestamp value in that format is 2262-04-11 On Sep 22, 2014, at 4:13 PM, Peyman Mohajerian wr

Re: Migration of metastore tables from mysql to oracle.

2014-10-21 Thread Sanjay Subramanian
First question, Why are u migrating to Oracle ? Since u never store data on Hive Metastore MYSQL is a great choice.  I have done a MYSQL to MYSQL transfer From the source DB mysql dump, it should be possible  to mod any Oracle required syntax right ? From: hadoop hive To: user@hive.apa

Re: select * from table and select column from table in hive

2014-10-21 Thread Sanjay Subramanian
One way to debug is to put bash in action say you have a data file in hdfs (/data/rockers/rockers.csv) that looks like  cust_num,cust_name,instrument1,paul,bass2,john,rhythm3,ringo,drums4,george,lead to get the column=cust_num of data (in this case its column 1) hdfs dfs -cat /data/rockers/rockers.

Re: Optimize hive external tables with serde

2014-10-21 Thread Sanjay Subramanian
1. The gzip files are not splittable, so gzip itself will make the queries slower. 2. As a reference for JSON serdes , here is a example from my blog http://bigdatalatte.wordpress.com/2014/08/21/denormalizing-json-arrays-in-hive/ 3. Need to see your query first to try and optimize it 4. Even if y

Optimize hive external tables with serde

2014-10-21 Thread Ja Sam
*Part 1: my enviroment* I have following files uploaded to Hadoop: 1. The are plain text 2. Each line contains JSON like: {code:[int], customerId:[string], data:{[something more here]}} 1. code are numbers from 1 to 3000, 2. customerId are total up to 4 millions, daily up to 0.5 mil

Re: Possible bug with max() together with rank() and grouping sets

2014-10-21 Thread j.barrett Strausser
Perhaps related to https://issues.apache.org/jira/browse/HIVE-4663 I ran across similar issues in .11 not sure if that above ticket affects .13 On Tue, Oct 21, 2014 at 8:21 AM, Michal Krawczyk wrote: > Hi all, > > Recently I've run into a problem with incorrect results in one of the > queries o

Possible bug with max() together with rank() and grouping sets

2014-10-21 Thread Michal Krawczyk
Hi all, Recently I've run into a problem with incorrect results in one of the queries on our system after upgrade from Hive 0.8.1.4 to 0.13.1. We use Amazon Elastic Map Reduce servivce on Amazon. I tried to simplify the original query and replicate this issue on a small dataset. Please take a look

Migration of metastore tables from mysql to oracle.

2014-10-21 Thread hadoop hive
Hi folks, I have a requirement where I need to migrate metastore tables from mysql to oracle. Is there any who already performed this kind of activity. Any help will be appreciated Thanks Vikas Srivastava

Re: UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException

2014-10-21 Thread Navis류승우
The states in UDF(which is represented as "states SomeClass3") are needed? If it's not, you can try them mark as a transient field. Thanks, Navis 2014-10-20 23:59 GMT+09:00 Harel Gliksman : > Hello, > > I am experiencing inconsistent behavior when trying to use UDF on 0.13.1 > on Amazon's EMR (A