Re: Noob question on creating tables

2013-03-29 Thread Sanjay Subramanian
sanjay From: Mark mailto:static.void@gmail.com>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>> Date: Friday, March 29, 2013 1:25 PM To: "user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org&g

Re: Noob question on creating tables

2013-03-29 Thread Mark
-To: "user@hive.apache.org" > Date: Friday, March 29, 2013 11:37 AM > To: "user@hive.apache.org" > Subject: Re: Noob question on creating tables > > That's a drawback of external tables, but it's actually not as difficult as > it sounds. It'

Re: Noob question on creating tables

2013-03-29 Thread Sanjay Subramanian
uot;user@hive.apache.org<mailto:user@hive.apache.org>" mailto:user@hive.apache.org>> Subject: Re: Noob question on creating tables That's a drawback of external tables, but it's actually not as difficult as it sounds. It's easy to write a nightly "cron"

Re: Noob question on creating tables

2013-03-29 Thread Dean Wampler
That's a drawback of external tables, but it's actually not as difficult as it sounds. It's easy to write a nightly "cron" job that creates the partition for the next day (or a job per month...), if someone on your team has some bash experience. Other job scheduling tools should support this too. H

Re: Noob question on creating tables

2013-03-29 Thread Nitin Pawar
yes On Fri, Mar 29, 2013 at 11:46 PM, Mark wrote: > Thanks > > Does this mean I need to create a partition for each day manually? There > is no way to have infer that from my directory structure? > > On Mar 29, 2013, at 10:32 AM, Dean Wampler < > dean.wamp...@thinkbiganalytics.com> wrote: > > >

Re: Noob question on creating tables

2013-03-29 Thread Mark
Thanks Does this mean I need to create a partition for each day manually? There is no way to have infer that from my directory structure? On Mar 29, 2013, at 10:40 AM, Sanjay Subramanian wrote: > Hi > > CREATE EXTERNAL TABLE IF NOT EXISTS log_data(col1 datatype1, col2 > datatype2, . . . colN

Re: Noob question on creating tables

2013-03-29 Thread Mark
Thanks Does this mean I need to create a partition for each day manually? There is no way to have infer that from my directory structure? On Mar 29, 2013, at 10:32 AM, Dean Wampler wrote: > > > On Fri, Mar 29, 2013 at 12:19 PM, Mark wrote: > We have existing log data in directories in the

Re: Noob question on creating tables

2013-03-29 Thread Sanjay Subramanian
Hi CREATE EXTERNAL TABLE IF NOT EXISTS log_data(col1 datatype1, col2 datatype2, . . . colN datatypeN) PARTITIONED BY (YEAR INT, MONTH INT, DAY INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'; ALTER table log_data ADD PARTITION (YEAR=2013 , MONTH=2, DAY=27) LOCATION '/path/to/YEAR/MONTH/DAY/d

Re: Noob question on creating tables

2013-03-29 Thread Dean Wampler
On Fri, Mar 29, 2013 at 12:19 PM, Mark wrote: > We have existing log data in directories in the format of YEAR/MONTH/DAY. > > - How can we create a table over this table without hive modifying and/or > moving it? > create external table foo (...) partitioned by (year int, month int, day int); .

Noob question on creating tables

2013-03-29 Thread Mark
We have existing log data in directories in the format of YEAR/MONTH/DAY. - How can we create a table over this table without hive modifying and/or moving it? - How can we tell Hive to partition this data so it knows about each day of logs? - Does hive out of the box work with reading compresse