It is a hadoop limitation. hdfs move operation is inexpensive. I am
assuming that is not an option to you because you want to save the path
structure (for some backward compatibility sake).
Something like symbolic links (i think its not supported in 0.20, not sure)
or path filter might help. But,
Hi Vince,
Hive partitioning can only exist by issueing new directories in HDFS. There
is no way to partition the data in a Hive table without adding extra
filepaths/dirs in HDFS.
For an external table you have to redistribute the data yourself in
corresponding filepaths and add the new partition
pache.org>"
mailto:user@hive.apache.org>>
Subject: RE: Partitioning EXTERNAL TABLE without copying or moving files
Hi Vince,
External tables shouldn’t issue copy or move commands to your data files. You
should define the base table location to ‘/logs’, and issue alter table
commands to ad
Hi Vince,
External tables shouldn't issue copy or move commands to your data files. You
should define the base table location to '/logs', and issue alter table
commands to add partitions for each date.
Example:
CREATE EXTERNAL TABLE logs (
Data STRING
) PARTITIONED BY (cal_date STRING)
ROW FO
*Why is it looking for data file when my sequence file is located at
> /user/training/partitions/dt=1/engine, according to the partition
>
>
>
>
>
>
> > Date: Tue, 28 Dec 2010 11:25:50 -0500
> > Subject: Re: Partitioning External table
> > From: edlinuxg...@gmail.com
>
ns/dt=1/data)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.ExecDriver
Why is it looking for data file when my sequence file is located at
/user/training/partitions/dt=1/engine, according to the partition
> Date: Tue, 28 Dec 2010 11:25:50 -0500
>
On Tue, Dec 28, 2010 at 9:41 AM, David Ginzburg wrote:
> Hi,
> I am trying to test creation of an external table using partitions,
> my files on hdfs are:
>
> /user/training/partitions/dt=2/engine
> /user/training/partitions/dt=2/engine
>
> engine are sequence files which I have managed to creat
