Re: Hive HWI ... request for your experience to be used Production

2013-01-04 Thread Qiang Wang
Hi Manish: Glad to receive your email because we are making efforts on HWI. We have improved the orignal and added some features and putted it on github: https://github.com/anjuke/hwi It's far from mature and standard, but it's improving and has already deployed for our company to use. After a

Re: HiveHistoryViewer concurrency problem

2013-01-04 Thread Qiang Wang
Maybe it's not. But this exception happens when I create an *HiveHistoryViewer* instance, in which case only reading, parsing file is invloved and it's not intended to be shared between threads. So the exception surprised me and I wonder why a static buffer was used instead of a local buffer whic

Re: HiveHistoryViewer concurrency problem

2013-01-04 Thread Qiang Wang
Hi Jie: As I know, hive history log is structured and class *HiveHistory* is used to write and read hive history log. *HiveHistoryViewer* serves as a listener to listen and store parsed log data. It has two members: HashMap *jobInfoMap*, which stores QueryInfo related with hive query and HashM

Re: HiveHistoryViewer concurrency problem

2013-01-04 Thread Edward Capriolo
It is likely an oversight. The Majority of hive code was not written to be multi-threaded. On Fri, Jan 4, 2013 at 10:41 PM, Jie Li wrote: > Hi Qiang, > > Could you describe how HiveHistoryViewer is used? I'm also looking for > a tool to understand the Hive log. > > Thanks, > Jie > > On Sat, Jan

Map-only aggregation

2013-01-04 Thread Jie Li
Hi all, Can Hive implement the aggregation as a Map-only job? As we know the data may be pre-partitioned via PARTITION-BY or CLUSTERED-BY, so we don't need the reduce phase to repartition the data. The Bucket Join seems to take advantage of the buckets for joins, so I wonder if there is some simi

Re: HiveHistoryViewer concurrency problem

2013-01-04 Thread Jie Li
Hi Qiang, Could you describe how HiveHistoryViewer is used? I'm also looking for a tool to understand the Hive log. Thanks, Jie On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang wrote: > Does Anybody have an idea about this? > > https://issues.apache.org/jira/browse/HIVE-3857 > > > 2013/1/4 Qiang Wang

Fwd: Hive HWI ... request for your experience to be used Production

2013-01-04 Thread Manish Malhotra
Hi All, We are exploring HWI to be used in PROD environment for adhoc queries etc. Want to check out in the hive community that can somebody share there experience while using the HWI in prod or any environment in terms of its stability and performance. Also evaluating to enhance to make it more u

Re: HiveHistoryViewer concurrency problem

2013-01-04 Thread Qiang Wang
Does Anybody have an idea about this? https://issues.apache.org/jira/browse/HIVE-3857 2013/1/4 Qiang Wang > new HiveHistoryViewer() throws ConcurrentModificationException when called > concurrently by several threads. > > According to the stack trace, HiveHistory.parseLine use *private static

Re: 0.8.0 -> 0.9.0 mysql schema upgrade

2013-01-04 Thread Sam William
Looks like this column is not even there in the 0.8/0.9 schema files . I have no idea, how I have it in my schema . I just set a default 'false' value and I m fine now. Sam On Jan 4, 2013, at 2:22 PM, Sam William wrote: > When I upgraded to 0.9.0, Im getting an exception when I try to creat

0.8.0 -> 0.9.0 mysql schema upgrade

2013-01-04 Thread Sam William
When I upgraded to 0.9.0, Im getting an exception when I try to create tables FAILED: Error in metadata: javax.jdo.JDODataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@4774e78a" using statement "INSERT INTO `SDS` (`SD_ID`,`NUM_BUCKETS`,`LOCATION`,

Re: Timestamp, Epoch Time, Functions and other Frustrations

2013-01-04 Thread John Omernik
So I read that JIRA, and also found this linked JIRA: https://issues.apache.org/jira/browse/HIVE-3454 So I decided to try the * 1.0 work around. select starttime, from_unixtime(starttime) as unixtime, cast((starttime * 1.0) as timestamp) as castts, from_utc_timestamp(starttime * 1.0, 'GMT') as

Re: Timestamp, Epoch Time, Functions and other Frustrations

2013-01-04 Thread Mark Grover
Brad is correct, there is a JIRA about this already: https://issues.apache.org/jira/browse/HIVE-3822 Sorry for the inconvenience. Mark On Fri, Jan 4, 2013 at 8:25 AM, Brad Cavanagh wrote: > Try multiplying your values by 1000, then running the conversions. I bet > they expect milliseconds since

Re: Timestamp, Epoch Time, Functions and other Frustrations

2013-01-04 Thread Brad Cavanagh
Try multiplying your values by 1000, then running the conversions. I bet they expect milliseconds since the epoch instead of seconds. Brad. On 2013-01-04, at 8:03 AM, John Omernik wrote: > Greetings all. I am getting frustrated with the documentation and lack of > intuitiveness in Hive relat

Re: Timestamp, Epoch Time, Functions and other Frustrations

2013-01-04 Thread John Omernik
One more test: to_utc_timestamp(from_unixtime(value), 'CST') as to_from, provided the proper timestamp for me, however, I still had to provide the timezone which I should NOT have to do. I know that this data coming in is in epoch time, therefore I should be able to create a timezone without knowi

Timestamp, Epoch Time, Functions and other Frustrations

2013-01-04 Thread John Omernik
Greetings all. I am getting frustrated with the documentation and lack of intuitiveness in Hive relating to timestamps and was hoping I could post here and get some clarification or other ideas. I have a field that is a string, but is actually a 10 digit int representation of epoch time, I am goin

Re: Thrift Hive client for CDH 4.1 HiveServer2?

2013-01-04 Thread Jov
they are in the src/service/if and src/metastore/if 在 2013-1-4 上午7:16,"David Morel" 写道: > Hi all (and happy New Year!) > > Is it possible to build a perl Thrift client for HiveServer2 (from > Cloudera's 4.1.x) ? > > I'm following the instructions found here: > http://stackoverflow.com/questions/52

Re: Job counters limit exceeded exception

2013-01-04 Thread Krishna Rao
I ended up increasing the counters limit to 130 which solved my issue. Do you know of any good sources to learn how to decipher hive's EXPLAIN? Cheers, Krishna On 2 January 2013 11:20, Alexander Alten-Lorenz wrote: > Hi, > > These happens when operators are used in queries (Hive Operators).