Re: Partitions

2011-07-01 Thread Aniket Mokashi
Thanks Prashanth, select Count(*) from segmentation_data where (dt='2011-07-01'); java.io.IOException: Not a file: hdfs://hadoop01:9000/data_feed/sophia/segmentation_data/1970-01-01 at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:206) at org.apache.hadoop.hive.ql.

RE: how to disable mapred.reduce.tasks

2011-07-01 Thread Steven Wong
Try -1, judging from this: mapred.reduce.tasks -1 The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas hive uses -1 as its default value. B

Re: Hive Loading Nulls When Using RegEx

2011-07-01 Thread Yichuan Hu
Use \\d instead of \d. On Jul 1, 2011, at 6:52 PM, Sal Scalisi wrote: > I'm new to hive and I'm having an issue loading a simple set of data via > regex. > > I have a data file called test.txt that contains the following: > > TESTONE-1 > TESTTWO-2 > TESTTHREE-3 > TESTFOUR-4 > TESTFIVE-5

Re: Partitions

2011-07-01 Thread Prashanth R
Pasting an example here: CREATE EXTERNAL TABLE IF NOT EXISTS tablename (...) partitioned by (insertdate string) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde'; alter table tablename add partition (insertdate='2008-01-01') LOCATION 's3n://' or 'hdfs:///abc/xyz/' - Prashant

Partitions

2011-07-01 Thread Aniket Mokashi
Hi, I have a data on HDFS that is already stored into directories as per date. for example- /abc/xyz/-mm-d1, /abc/xyz/-mm-d2. How do I create external table with partition key as date to point to data in this directory? Please advise. Thanks, Aniket

Hive Loading Nulls When Using RegEx

2011-07-01 Thread Sal Scalisi
I'm new to hive and I'm having an issue loading a simple set of data via regex. I have a data file called test.txt that contains the following: TESTONE-1 TESTTWO-2 TESTTHREE-3 TESTFOUR-4 TESTFIVE-5 I have this hive script: hive> CREATE TABLE test > ( > field_1 STRING > ) > ROW FORMAT SERDE '

Re: Loaded data all NULL

2011-07-01 Thread Yichuan (William) Hu
Thanks, Edward, I am just using this simple example to test regex serde. Ultimately I would like to use regex to parse various types of log files. On Fri, Jul 1, 2011 at 2:34 PM, Edward Capriolo wrote: > > > On Fri, Jul 1, 2011 at 2:16 PM, Yichuan (William) Hu > wrote: >> >> Hi, >> >> I am doing

Re: Loaded data all NULL

2011-07-01 Thread Edward Capriolo
On Fri, Jul 1, 2011 at 2:16 PM, Yichuan (William) Hu wrote: > Hi, > > I am doing some simple tests to create table, load data using Hive. I > am working on the VM provided by cloudera > (https://ccp.cloudera.com/display/SUPPORT/Cloudera%27s+Hadoop+Demo+VM). > > I have a text file with each line co

Re: Loaded data all NULL

2011-07-01 Thread Yichuan (William) Hu
I tried \\d instead of \d, problem solved. Thanks, Neerja! On Fri, Jul 1, 2011 at 2:28 PM, Neerja Bhatnagar wrote: > Hi, > When I was having this issue, it was due to 2 problems - 1. the slashes in > the regex need to be escaped Java-style so \d would be \\d and my regex's > were wrong. > Hope th

Loaded data all NULL

2011-07-01 Thread Yichuan (William) Hu
Hi, I am doing some simple tests to create table, load data using Hive. I am working on the VM provided by cloudera (https://ccp.cloudera.com/display/SUPPORT/Cloudera%27s+Hadoop+Demo+VM). I have a text file with each line containing an IP address and a name, e.g., 123.45.67.89 tom 123.45.67.92 m