Hi everyone,
I'd like to create a change request (or JIRA, not sure), do you think it's
feasible? And I search the document about how to contribute, but can't find a
way about how to create a request, could anyone point me to the document?
At 2011-08-14 17:08:26,"Daniel,Wu" wrote:
a simple u
DISTRIBUTE BY and CLUSTER BY didn't resolve all the issues I've seen
with very large data sets. I mean I'm loading a couple terabytes in a
dataset and running into some rather interesting problems. I noticed
however loading a couple months or two at a time (and making sure they
were from the
The wiki docs are incorrect here. CREATE INDEX does not yet supported a
PARTITIONED BY clause; that was added in the spec to support HIVE-1499, which
hasn't been implemented yet.
For now, the index partitioning always follows the table partitioning exactly.
JVS
On Aug 14, 2011, at 3:22 AM, Da
Ya I very much agree with you on those lines. Using the basic stuff would
literally run into memory issues with large datasets. I had some of those
resolved by using the DISTRIBUTE BY clause and so. In short a little work
around over your hive queries could help you out in some cases.
Regards
B
Something else I've noticed is when loading LOTS of historical data, if
you can try to say load a month of data at a time, try to just load THAT
month of data and only that month. I've been able to load several years
of data (depending on the data) at a single load however there have been
time
create table part (a int,b int) PARTITIONED by (c int);
create index part_idx on table part(b,c) AS
'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED
REBUILD
partitioned by (a) ;
hive> create index part_idx on table part(b,c) AS
'org.apache.hadoop.hive.ql.index.comp
a simple usage: for retailer data, which keep 10 years of data, that's 10 *
365 =3650 records in the calendar dimension, if there are 8000 stores and 8000
products, totally the sales will have 8000 * 8000 * 3650 =233,600,000,000
records if we has one record for each product/day/store combinati
