Thanks Jan for the suggestion.
*Raihan Jamal*
On Tue, Aug 7, 2012 at 10:01 PM, Jan Dolinár wrote:
> The shell will interpret the query in your command as SELECT
> ... explode(split(timestamps, *#*)) ... if you run it the way you wrote
> it, i.e. without the quotation. The way around this i
The shell will interpret the query in your command as SELECT
... explode(split(timestamps, *#*)) ... if you run it the way you wrote
it, i.e. without the quotation. The way around this is to either escape the
quotation marks or use single quotes:
hive -e *"*SELECT user_id
,product_id
,prod_and_ts
Since Hive (0.7.1) only supports equi-based join how are people using it for
joins employing the between clause?
Thanks,
Ranjith
Hi,
When I try to use Hive Indexing, I have the following questions.
1. Does Indexing have the same performance on both the partitioned and
non-partitioned tables? How about bucketed and un-bucked tables?
2. Is it possible for us to build index of function of indexed columns,
like
create
Hi Venkatesh,
So my shell script should be like this right?
*#!/bin/bash*
*HADOOP_HOME=/home/hadoop/latest*
*export HADOOP_HOME*
*JAVA_HOME=/usr/jdk/latest*
*export JAVA_HOME*
*HIVE_OPTS="$HIVE_OPTS -hiveconf mapred.job.queue.name=hdmi-technology"*
*export HIVE_OPTS*
*DATE_YEST=`TZ=GMT+48 date +%
Thanks Igor! It worked!
Date: Mon, 6 Aug 2012 23:26:04 -0700
Subject: Re: Changing table types from managed to external
From: i...@decide.com
To: user@hive.apache.org
CC: d...@hive.apache.org
TryALTER TABLE SET TBLPROPERTIES('EXTERNAL'='TRUE');
It worked for me.
igordecide.com
On Mon, Aug
you should be able to do this in hive using a group by on alpha and then
using a combination of the max and if statement... something on the
following lines
select alpha, max(abc), max(pqr), ...
(
select alpha, if (beta == 'abc', Gamma, NULL) as abc, if (beta == 'pqr',
Gamma, NUL) as pqr,
You can do this using case statements and summing the values. The only item to
remember here is that the values in the grid need to be numeric for the sum to
work.
Thanks,
Ranjith
From: richin.j...@nokia.com [mailto:richin.j...@nokia.com]
Sent: Tuesday, August 07, 2012 3:58 PM
To: user@hive.apa
Hi All,
One of my Query output looks like-
AlphaBeta Gamma
123 xyz 1.0
123 abc 0.5
123 pqr 1.3
123
Since the UDF unix_timestamp() is a non-deterministic function, Hive query
planner doesn't run partition pruning based the 'dt' column value. If your
table is partitioned by 'dt' column, the query would end up scanning entire
table.
It is ideal to compute the required date value dynamically in a
Is there a difference between the following two join sub queries in Hive?
A JOIN B ON (A.DUH1=B.DUH2) JOIN C ON (B.DUH2=C.DUH3)
vs.
A JOIN B JOIN C ON (A.DUH1=B.DUH2) AND (B.DUH2=C.DUH3)
?
And is it still necessary to keep the larger table at the very end? What if
tables B and C are selection
Right, no need for the function at all. Sorry it is getting late here and
my brain refuses to work any more :)
On Tue, Aug 7, 2012 at 8:39 PM, Techy Teck wrote:
> Then that means I don't need to create that userdefinedfunction right?
>
>
>
> On Tue, Aug 7, 2012 at 11:32 AM, Jan Dolinár wrote:
Let me try that and I will update on this thread.
*Raihan Jamal*
On Tue, Aug 7, 2012 at 11:39 AM, Techy Teck wrote:
> Then that means I don't need to create that userdefinedfunction right?
>
>
>
> On Tue, Aug 7, 2012 at 11:32 AM, Jan Dolinár wrote:
>
>> Hi Jamal,
>>
>> date is standard lin
Then that means I don't need to create that userdefinedfunction right?
On Tue, Aug 7, 2012 at 11:32 AM, Jan Dolinár wrote:
> Hi Jamal,
>
> date is standard linux/unix tool, see the manual page:
> http://linux.die.net/man/1/date.
>
> The $(...) tells the shell to execute the command and insert
Hi Jamal,
date is standard linux/unix tool, see the manual page:
http://linux.die.net/man/1/date.
The $(...) tells the shell to execute the command and insert it's output
into the string. So in this case it will execute command
date -d -1day +%Y%m%d
which returns yesterday date in the format you
Hi Vijay,
Thanks for the suggestion, If upgrading to Hive was under my control then
I would have done for sure, but I am working in a company and they are
running Hive 0.6 on all the cluster, And I told them to upgrade the Hive
version but they said it will take few months for them to do this. An
Let me try that and I will update on this thread If I found something
interesting.
Thanks for the help kulkarni.
On Tue, Aug 7, 2012 at 11:10 AM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:
> In that case you might want to try "count(1)" instead of "count(*)" and see
> if t
You actually don't need hive on the whole cluster. That's the beauty
of it. You only need it on the client machine where you're submitting
hive jobs. Of course the metadata store does need to be upgraded for
newer versions so that might still be a problem.
On Tue, Aug 7, 2012 at 11:26 AM, Raihan J
Yes it supports -e option, but in your query what is date?
hive -e "CREATE TEMPORARY FUNCTION yesterdaydate
AS 'com.example.hive.udf.YesterdayDate';
SELECT * FROM REALTIME where dt=$(*date* -d -1day +%Y%m%d) LIMIT 10;"
*Raihan Jamal*
On Tue, Aug 7, 2012 at 11:18 AM, Jan Dolinár wrote:
> By
By the way, even without hiveconf, you can run hive from shell like this to
achieve what you want using shell capabilities:
hive -e "CREATE TEMPORARY FUNCTION yesterdaydate
AS 'com.example.hive.udf.YesterdayDate';
SELECT * FROM REALTIME where dt=$(date -d -1day +%Y%m%d) LIMIT 10;"
At least if hiv
Given the implementation of the UDF, I don't think hive would be able
to use partition pruning. Especially the version you're using. I'd
really recommend upgrading to a later version that has the hiveconf
support. That can save a lot of trouble rather than trying to get
things working on 0.6
On Tu
In that case you might want to try "count(1)" instead of "count(*)" and see
if that makes any difference. [1]
[1] https://issues.apache.org/jira/browse/HIVE-287
On Tue, Aug 7, 2012 at 1:07 PM, Techy Teck wrote:
> I am running Hive 0.6.
>
>
>
>
>
> On Tue, Aug 7, 2012 at 11:04 AM, kulkarni.swar.
Hi Jan,
I have date in different format also, so that is the reason I was thinking
to do by this approach. How can I make sure this will work on the selected
partition only and it will not scan the entire table. I will add your
suggestion in my UDF as deterministic thing.
My simple question here
I am running Hive 0.6.
On Tue, Aug 7, 2012 at 11:04 AM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:
> What is the hive version that you are using?
>
>
> On Tue, Aug 7, 2012 at 12:57 PM, Techy Teck wrote:
>
>> I am not sure about the data, but when we do
>>
>> SELECT count(
What is the hive version that you are using?
On Tue, Aug 7, 2012 at 12:57 PM, Techy Teck wrote:
> I am not sure about the data, but when we do
>
> SELECT count(*) from data_realtime where dt='20120730' and uid is null
>
> I get the count
>
> but If I do-
>
> SELECT * from data_realtime where dt=
@kulkarni,
When I did explain on my query, I got these things, I am not sure how to
understand these thing. Any help will be appreciated whether my approach is
right or not?-
hive> EXPLAIN SELECT * FROM PDS_ATTRIBUTE_DATA_REALTIME where
dt=yesterdaydate('MMdd', 2) LIMIT 5;
OK
ABSTRACT S
I am not sure about the data, but when we do
SELECT count(*) from data_realtime where dt='20120730' and uid is null
I get the count
but If I do-
SELECT * from data_realtime where dt='20120730' and uid is null
I get zero record back. But if all the record is NULL then I should be
getting NULL r
Oops, sorry I made a copy&paste mistake :) The annotation should read
@*UDFType(deterministic=true*)
Jan
On Tue, Aug 7, 2012 at 7:37 PM, Jan Dolinár wrote:
> I'm afraid that he query
>
> SELECT * FROM REALTIME where dt= yesterdaydate('MMdd') LIMIT 10;
>
> will scan entire table, because th
I'm afraid that he query
SELECT * FROM REALTIME where dt= yesterdaydate('MMdd') LIMIT 10;
will scan entire table, because the functions is evaluated at runtime, so
Hive doesn't know what the value is when it decides which files to scan. I
am not 100% sure though, you should try it.
Also, yo
Have you tried using EXPLAIN[1] on your query? I usually like to use that
to get a better understanding of what my query is actually doing and
debugging at other times.
[1] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain
On Tue, Aug 7, 2012 at 12:20 PM, Raihan Jamal wrote
Just in case, all Record is null when uid is null?
On Tue, Aug 7, 2012 at 1:14 PM, Techy Teck wrote:
> SELECT count(*) from data_realtime where dt='20120730' and uid is null
>
>
>
> I get the count as 1509
>
>
>
> So that means If I will be doing
>
>
>
> SELECT * from data_realtime where dt='2012
Hi Jan,
I figured that out, it is working fine for me now. The only question I have
is, if I am doing like this-
SELECT * FROM REALTIME where dt= yesterdaydate('MMdd') LIMIT 10;
Then the above query will be evaluated as below right?
SELECT * FROM REALTIME where dt= ‘20120806’ LIMIT 1
SELECT count(*) from data_realtime where dt='20120730' and uid is null
I get the count as *1509*
So that means If I will be doing
SELECT * from data_realtime where dt='20120730' and uid is null
I should be seeing those records in which uid is null? right?
But I get zero record back wit
haven't tried this but - since your myoutputtable table is tab delimited,
and if this format suites your needs, you could create it as an external
table and specify its hadoop path then run the getmerge command off of that
location (without needing the 'insert overwrite directory ...' command, so
See
https://cwiki.apache.org/Hive/languagemanual-cli.html#LanguageManualCli-Logging
.
modify conf/hive-*-log4j.properties and set the logging level there!
On Tue, Aug 7, 2012 at 12:44 PM, Ravindra wrote:
> Hi ,
>
> I used following property in the hive-site.xml and I still see that
> logging i
Hi ,
I used following property in the hive-site.xml and I still see that logging
is happening. Can someone please help me to switch off the logging.
log4j.category.DataNucleus
off
--
Ravi.
*''We do not inherit the earth from our ancestors, we borrow it from our
36 matches
Mail list logo
