For the time being I have added the create HDFS dir in my hive script…got to
keep moving on….cant wait for ideal solution :-) but would love to know the
ideal solution !
!hdfs dfs -mkdir
/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/record_counts
;
INSERT OVERWRITE DIRECTORY
'/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/record_counts'
SELECT
'outpdir_impressions_header',
'2013-07-01',
'record_counts',
'all_servers',
count(*)
FROM
outpdir_impressions_header
WHERE
header_date_partition='2013-07-01'
;
From: Sanjay Subramanian
<[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Wednesday, July 3, 2013 2:40 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: One query works the other does notŠany clues ?
THIS FAILS
=========
INSERT OVERWRITE DIRECTORY
'/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/record_counts'
select 'outpdir_impressions_header', '2013-07-01', 'record_counts',
'all_servers', count(*) from outpdir_impressions_header where
header_date_partition='2013-07-01'
13/07/03 14:28:31 ERROR exec.Task: Failed with exception Unable to rename:
hdfs://thv-nn1.pv.sv.nextag.com:8020/tmp/hive-nextag/hive_2013-07-03_14-25-39_324_3631745157890476605/-ext-10000
to:
/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/record_counts
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename:
hdfs://thv-nn1.pv.sv.nextag.com:8020/tmp/hive-nextag/hive_2013-07-03_14-25-39_324_3631745157890476605/-ext-10000
to:
/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/record_counts
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:95)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:159)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:695)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
THIS SUCCEEDS
=============
INSERT OVERWRITE DIRECTORY
'/user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/record_counts/2013-07-01'
select 'outpdir_impressions_header', '2013-07-01', 'record_counts',
'all_servers', count(*) from outpdir_impressions_header where
header_date_partition='2013-07-01'
TABLE DEFINITION
================
CREATE EXTERNAL TABLE IF NOT EXISTS impressions_hive_stats(table_name STRING,
aggregation_date STRING , metric_name STRING, metric_key STRING, metric_value
BIGINT) PARTITIONED BY (table_name_partition STRING, aggregation_date_partition
STRING , metric_name_partition STRING, metric_key_partition STRING) STORED AS
INPUTFORMAT \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\" OUTPUTFORMAT
\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\" ;
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the
intended recipient(s) and may contain confidential and privileged information.
Any unauthorized review, use, disclosure or distribution is prohibited. If you
are not the intended recipient, please contact the sender by reply email and
destroy all copies of the original message along with any attachments, from
your computer system. If you are the intended recipient, please be advised that
the content of this message is subject to access, review and disclosure by the
sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the
intended recipient(s) and may contain confidential and privileged information.
Any unauthorized review, use, disclosure or distribution is prohibited. If you
are not the intended recipient, please contact the sender by reply email and
destroy all copies of the original message along with any attachments, from
your computer system. If you are the intended recipient, please be advised that
the content of this message is subject to access, review and disclosure by the
sender's Email System Administrator.
