Yes, if the only problem with my script solution is the variable number of
new days, you can just add all possibilities in advance, knowing that some
will be empty of content...
On Tue, Jan 29, 2013 at 7:49 PM, Sadananda Hegde wrote:
> Thanks, Edward. I can probably create all previous days part
Thanks, Edward. I can probably create all previous days partitions ahead
of time and then use Dean's logic to create new partitions on a daily
basis. I will probably end up having few empty partitions; need to make
sure it does not cause any confusions.
Thanks,
Sadu
On Tue, Jan 29, 2013 at 7:21
Thanks Dean,
But the M/R job creates the sub-folder based on the data and hence could
create the sub-folder for any of the previous days (and for multiple days
too). So I canot limit to today's folder only. I need to get the
year/month/day values from the HDFS location instead of from the system
d
You can also just create all your partitions ahead of time. They will not
do any harm if empty. (unless you have an older version and hit this...
http://issues.apache.org/jira/browse/HIVE-1007 )
On Tue, Jan 29, 2013 at 8:17 PM, Mark Grover wrote:
> Hi Sadananda,
> Sorry to hear that.
>
> It got c
Hi Sadananda,
Sorry to hear that.
It got committed, don't worry about the "ABORTED". Here is the commit on
the trunk:
https://github.com/apache/hive/commit/523f47c3b6e7cb7b6b7b7801c66406e116af6dbc
However, there is no Apache Hive release with that patch in it.
You have two options:
1. Download t
Thanks Mark,
Recover partition feature will satisfy my needs; but MSCK Repair Partition
< tablename> option is not working for me. It does not give any error; but
does not add any partitions either. It looks like it adds partitions only
when the sub-folder is empty; but not when the sub-folder ha
This is very easy to do with a shell script and you can have cron run it
nightly, say just after midnight so the year, month, and day calculation
below return the correct values:
#!/bin/bash
year=$(year +%Y)
month=$(year +%m)
day=$(year +%d)
hive -e "use mydb; ALTER TABLE foo ADD IF NOT EXISTS P
Sadananda,
See if this helps:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Recoverpartitions
On Mon, Jan 28, 2013 at 8:05 PM, Sadananda Hegde wrote:
> Hello,
>
> My hive table is partitioned by year, month and day. I have defined it as
> external table. Th
Sadananda,
Look at Oozie workflow.
Regards
Abhishek
On Jan 28, 2013, at 11:05 PM, Sadananda Hegde wrote:
> Hello,
>
> My hive table is partitioned by year, month and day. I have defined it as
> external table. The M/R job correctly loads the files into the daily
> subfolders. The hdfs file
Hello,
My hive table is partitioned by year, month and day. I have defined it as
external table. The M/R job correctly loads the files into the daily
subfolders. The hdfs files will be loaded to
/year=/month=mm/day=dd/ folders by the scheduled M/R jobs.
The M/R job has some business logic in d
10 matches
Mail list logo
