Hi, hive users,
We need help from the hive community!
We are now using very old hive version(0.13) due to historical reason, and
we often meet following issue:
Caused by: java.io.IOException: File already
exists:s3://smart-dmp/warehouse/uploaded/ad_dmp_pixel/dt=2021-06-21/key=259f3XXX
We ha
I have read the design doc of the Hive planner recently. I've found
the Hive only uses calcite as a SQL optimizer. Translate the optimized
relnode to hive operator tree, and then transfer it to the Tez execution
DAG.
Why not implement the calcite adaptor, this way is easier to use calcite?
The lin
I have read the design doc of the Hive planner recently. I've found
the Hive only uses calcite as a SQL optimizer. Translate the optimized
relnode to hive operator tree, and then transfer it to the Tez execution
DAG.
[image: image.png]
Why not implement the calcite adaptor, this way is easier to us
Hi,all
Did hive support execute history command like shell(use number like !10)?
And I find beeline not support this:
!
Executes a shell command from the Hive shell.
All of command with ! prefix will be SQLLine CLI command,I think it's a very
useful command.
I have some corrupt AVRO files in my table. And while I query them with
Hive, it will throw an exception and exit. Is there any way to ignore these
AVRO files and make Hive continue to run? Or is there a way to find all the
corrupt files on HDFS and delete it?
Hi, I followed the instructions in the link below to configure hivesever2
using PAM without success:
https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2
(1) download the JPAM native library
(2) unzip and copy libjpam.so and JPam-1.1.jar to a directory (/usr/lib/pam) on
t
riptor).
I am newer and don't know why this error is prompted still. The attachment is
my hive-site.xml.
Could any friend help me. I am expecting and appreciating your kindly reply.
Thanks.
Best regards,
Sean Wang
hive-site.xml
Description: Binary data
are hitting into :
> https://issues.apache.org/jira/browse/HIVE-9324
>
> On Thu, Jan 15, 2015 at 1:53 AM, Guodong Wang wrote:
>
>> Hi,
>>
>> I am using hive 0.13.1 and currently I am blocked by a bug when joining 2
>> tables. Here is the sample query.
>>
>>
Hi,
I am using hive 0.13.1 and currently I am blocked by a bug when joining 2
tables. Here is the sample query.
INSERT OVERWRITE TABLE test_archive PARTITION(data='2015-01-17', name, type)
SELECT COALESCE(b.resource_id, a.id) AS id,
a.timstamp,
a.payload,
a.name,
a.typ
Hi all,
I am looking for some tutorial about how to set up "SQL Standard Based Hive
Authorization" in hive 0.13.
I found this page
https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization
I have followed this page to setup the mentioned configuration. But
authorizatio
+ hive user group. :)
-- Forwarded message --
From: Gordon Wang
Date: Tue, Oct 28, 2014 at 11:22 AM
Subject: How to run TestCliDriver Unit Test
To: d...@hive.apache.org
Hi all,
I am working on a hive issue and I want to add a Unit test for hive query
parser.
Before hive 0.13
I am using Hive 0.12.0. And when putting multiple inserts for the same
table in one SQL, looks like hive queries plan analyzer fails to synthesis
the right plan.
Here is the issue.
create table T1(i int, j int);
create table T2(m int) partitioned by (n int);
explain from T1
insert into table T2 p
Hi,
We want to create Hive EXTERNAL TABLE to map to HBase tables.
Our questions is, what kind of character set "hbase.columns.mapping" can
support?
For example, for below clause.
CREATE EXTERNAL TABLE SomeTable (Default_Key STRING, `Hive Column Name`
STRING) STORED BY 'org.apache.had
Hello,
I've been looking for good ways to create and write to Hive tables from
Java code. So far, I've considered the following options:
1. Create Hive table using the JDBC client, write data to HDFS using bare
HDFS operations, and load that data into the Hive table using the JDBC
client. I didn'
effect )
I suggest that you execute HQL like "set fs.defaultFS=hdfs://hostA:8020;"
after an jdbc connection is established.
> Please reply
>
>
>
> Thanks
>
> Rinku Garg
>
>
>
> *From:* Gordon Wang [mailto:gw...@gopivotal.com]
> *Sent:* Friday, March 07,
48;
>
> set mapreduce.map.java.opts=-Xmx2048M;
>
> set yarn.app.mapreduce.am.command-opts=-Xmx2048m;
>
>
>
> Please suggest.
>
>
>
> Thanks
>
> Rinku Garg
>
>
>
> *From:* Gordon Wang [mailto:gw...@gopivotal.com]
> *Sent:* Friday, March 07, 2014 11:49 AM
&g
ion contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
--
Regards
Gordon Wang
value, and myapper simply read
from std.in, and convert to key,value.
Chen
On Mon, Feb 3, 2014 at 5:51 AM, Bogala, Chandra Reddy wrote:
> Hi Wang,
>
> I am first time trying MAP & Reduce inside hive query. Is it possible
> to share mymapper and myreducer code? So that I can u
it seems that hive.exec.reducers.bytes.per.reducer is still not big
enough: I added another 0, and now i only gets one file under each
partition.
On Sun, Feb 2, 2014 at 10:14 PM, Chen Wang wrote:
> Hi,
> I am using java reducer reading from a table, and then write to another
> one:
&
Hi,
I am using java reducer reading from a table, and then write to another one:
FROM (
FROM (
SELECT column1,...
FROM table1
WHERE ( partition > 6 and partition < 12 )
) A
MAP A.co
Hey guys,
I am using flume to directly sink data into my hive table. However, there
seems to be some schema inconsistency, and I am not sure how to
troubleshoot it.
I created a hive table 'targeting' in hive, it use sequence file, snappy
compression, partitioned by 'epoch'. After the table is crea
k.
>
> Alan.
>
> On Jan 6, 2014, at 6:26 PM, Chen Wang wrote:
>
> > Alan,
> > the problem is that the data is partitioned by epoch ten hourly, and i
> want all data belong to that partition to be written into one file named
> with that partition. How can i share the f
You can then close the files every 15 minutes (or whatever works for you)
> and have a separate job that creates a new partition in your Hive table
> with the files created by your bolts.
>
> Alan.
>
> On Jan 2, 2014, at 11:58 AM, Chen Wang wrote:
>
> > Guys,
> >
Guys,
I am using storm to read data stream from our socket server, entry by
entry, and then write them to file: one entry per file. At some point, i
need to import the data into my hive table. There are several approaches i
could think of:
1. directly write to hive hdfs file whenever I get the ent
elect distinct col1, col2, col3 works but if you want to select
> all 50 columns its tricky.
>
> Other option would be group by all those 9 keys and it should take care
> that you have combination of those 9 columns is unique.
>
>
> On Wed, Oct 2, 2013 at 12:34 PM, Phil
ust one row for a single combination of
> these 9 columns ?
>
>
> On Wed, Oct 2, 2013 at 6:07 AM, Philo Wang wrote:
>
>> Hi,
>>
>> I am using Hive 8.1.8 in EMR.
>>
>> We have an extremely large table (~50 columns) where the uniqueness key
>>
Hi,
I am using Hive 8.1.8 in EMR.
We have an extremely large table (~50 columns) where the uniqueness key is
a combination of 9 different columns. I want to filter out any duplicate
rows based on these 9 columns while retaining the ability to select other
columns on an ad hoc basis. I don’t expec
clause.**
> **
>
> ** **
>
> Thanks,
>
> Chalcy****
>
> *From:* Daning Wang [mailto:dan...@netseer.com]
> *Sent:* Wednesday, March 06, 2013 3:08 PM
> *To:* user@hive.apache.org
> *Subject:* Hadoop cluster hangs on big hive job
>
> ** **
>
> We have 5 nodes
We have 5 nodes cluster(Hadoop 1.0.4), It hung a couple of times while
running big hive jobs(hive-0.8.1). Basically all the nodes are dead, from
that trasktracker's log looks it went into some kinds of loop forever.
All the log entries like this when problem happened.
Any idea how to debug the is
the one
user that needs it, and you don't have to set up a special HDFS group for
it.
--Bobby
On 2/9/13 8:31 PM, "wang" wrote:
>Thank your 's response~
>In hive, user can directly execute load path command, if the dir is
>accessible by two user, then, one user can di
Hi all,
Idealy, in hive security, the client user should be transmitted to hdfs
and mr, just like this jira said.
I have some questions on how to implement this, mainly on one sql visit
multiple user's table.
For example , t1 belong to user1; t2 belong to user2, user1 want e
09
> How am I doing? Please feel free to provide feedback to my manager Rick
> Morris
> at r...@hortonworks.com
>
>
> On Feb 6, 2013, at 5:28 AM, Zhong Wang wrote:
>
> Hi all,
>
> I am running tests on Hive auto convert join. From the source code, it
> seems
Every hive query has a history file, and you can get these info from hive
history file
Following java code can be an example:
https://github.com/anjuke/hwi/blob/master/src/main/java/org/apache/hadoop/hive/hwi/util/QueryUtil.java
Regard,
Qiang
2013/1/30 Mathieu Despriee
> Hi folks,
>
> I would
class ObjectStore has a private member named 'classloader':
private ClassLoader classLoader;
{
classLoader = Thread.currentThread().getContextClassLoader();
if (classLoader == null) {
classLoader = QueryStore.class.getClassLoader();
}
}
But I can't find the pla
thing, or have I been mistaken this whole time?
>
> --Tom
>
>
> On Thursday, January 10, 2013, Qiang Wang wrote:
>
>> The HWI will create a cli session for each query through hive libs, so
>> several queries can run concurrently.
>>
>>
>> 2013/1/11
The HWI will create a cli session for each query through hive libs, so
several queries can run concurrently.
2013/1/11 Tom Brown
> How is concurrency achieved with this solution?
>
>
> On Thursday, January 10, 2013, Qiang Wang wrote:
>
>> I believe the HWI (Hive Web Inte
I believe the HWI (Hive Web Interface) can give you a hand.
https://github.com/anjuke/hwi
You can use the HWI to submit and run queries concurrently.
Partition management can be achieved by creating crontabs using the HWI.
It's simple and easy to use. Hope it helps.
Regards,
Qiang
2013/1/11 T
pen a jira on http://issues.apache.org/jira/hive and produce a
>>> patch that will apply to hive trunk. That will start the process.
>>>
>>>
>>>
>>>
>>> On Saturday, January 5, 2013, Qiang Wang wrote:
>>> > Hi Manish:
>>> >
;> Regards,
>> Manish
>>
>>
>> On Sun, Jan 6, 2013 at 9:10 AM, Edward Capriolo wrote:
>>
>>> The hive code is apache licensed. If you want to add you work to hive
>>> simple open a jira on http://issues.apache.org/jira/hive and produce a
>>&
ode of running queries or only scheduling /
> async way?
>
> Thanks for your reply and time,
>
> Regards,
> Manish
>
>
> On Fri, Jan 4, 2013 at 9:34 PM, Qiang Wang wrote:
>
>> Hi Manish:
>>
>> Glad to receive your email because we are making ef
Hi Manish:
Glad to receive your email because we are making efforts on HWI.
We have improved the orignal and added some features and putted it on
github:
https://github.com/anjuke/hwi
It's far from mature and standard, but it's improving and has already
deployed for our company to use.
After a
w HiveHistoryViewer is used? I'm also looking for
>> a tool to understand the Hive log.
>>
>> Thanks,
>> Jie
>>
>> On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang wrote:
>> > Does Anybody have an idea about this?
>> >
>> > https://i
tand the Hive log.
>
> Thanks,
> Jie
>
> On Sat, Jan 5, 2013 at 9:54 AM, Qiang Wang wrote:
> > Does Anybody have an idea about this?
> >
> > https://issues.apache.org/jira/browse/HIVE-3857
> >
> >
> > 2013/1/4 Qiang Wang
> >>
>
Does Anybody have an idea about this?
https://issues.apache.org/jira/browse/HIVE-3857
2013/1/4 Qiang Wang
> new HiveHistoryViewer() throws ConcurrentModificationException when called
> concurrently by several threads.
>
> According to the stack trace, HiveHistory.parseLine use *pr
a look, feedback welcome. The intent is the
> same as your suggestions but the approach is a little more
> conservative. If you feel strongly that it should be done according to
> one of your suggestions, let me know, I will take another look.
>
> Thanks!
> Mark
>
> On
anybody has an idea about this ?
https://issues.apache.org/jira/browse/HIVE-3810
2012/12/16 Qiang Wang
> glad to receive your reply!
>
> here is my point:
> Firstly, I think HiveHistoryViewer is inconsistent with HiveHistory.
> Secondly, hive server may be deloyed on linux, bu
rma.stanford.edu/~craig/utility/flip/> Hive
> is typically deployed on Linux.
> *
> *
> DOS & Windows: \r\n 0D0A (hex), 13,10 (decimal)
>
> Unix & Mac OS X: \n, 0A, 10
> Macintosh (OS 9): \r, 0D, 13
> *
> On Sun, Dec 16, 2012 at 11:23 AM, Qiang Wang wrote:
>
>> '\n', '\r',
>
>
>
>
>
rg/Hive/languagemanual-ddl.html#LanguageManualDDL-Create%252FDropTable
>
>
> 2012/7/10 Daning Wang
>
>> Hi all,
>>
>> New to here. I have sequence files and I want to read in Hive, I did
>> some search on the net. looks the way I need to do
>>
>> 1.
Hi all,
New to here. I have sequence files and I want to read in Hive, I did some
search on the net. looks the way I need to do
1. Write new FileInputFormat so Hive can read both key and value.
(ref http://osdir.com/ml/hive-user-hadoop-apache/2009-10/msg00034.html)
2. Write a SerDe to deseriali
Thanks a lot for Roberto Congiu and wil's help.
The problem has been solved with your assistance.
I think I should read the wiki guide more carefully!
Thank you very much!
Best regards!
2011-03-01
Jianhua Wang
t
seems that the tasktracker B and C can not find the script.
On hive wiki, I didn't find any instruction on how to place the user script.
What should I do to place my script in proper place?
Thanks in advance for any reply!
2011-03-01
Jianhua Wang
Hi Tony,
We're still working on the index implementation in Hive, so index
support is very limited. When you use CREATE INDEX in Hive, you must
specify the index type. Currently, the only built-in index type is the
Compact index, though we are working to add bitmap indexes and others.
Suppose you
Hi Carl,
With your assistance, I have checked the link to slide and got what I need.
As you have said, it consists of all the key diagrams, and is very helpful for
me.
Thank you very much!
Best regards!
2011-01-10
Jianhua Wang
发件人： Carl Steinbach
发送时间： 2011-01-10 14
page about map join optimization?
Thanks!
2011-01-10
Jianhua Wang
54 matches
Mail list logo
