Only thing i was confused that mysql meta update may have lock and it might cause any data loss.
Now i am clear on this. Thanks a lot Nitin. On Fri, May 3, 2013 at 1:06 PM, Nitin Pawar <nitinpawar...@gmail.com> wrote: > Why dont you load all of your data into a temporary table and then from > there to your current tables. > > Hive will take care of adding dynic partitions and that will remove the > ocerhead from you. > > To answer your question, you can always load data in different partitions > parallely as long as you have resources available on hive cli machine > > On May 3, 2013 12:35 PM, "selva" <selvai...@gmail.com> wrote: > > > > Hi All, > > > > I need to load a month worth of processed data into a hive table. Table > have 10 partitions. Each day have many files to load and each file is > taking two seconds(constantly) and i have ~3000 files). So it will take > days to complete for 30 days worth of data. > > > > I planned to load every day data parellaly into respective partition so > that i can complete it short time. > > > > But i need clarrification before proceeding it. > > > > Question: > > > > 1. Will it cause data loss/corruption by loading parellely in different > partition of same hive table ? > > > > For example, Assume i am doing like below, > > > > Table : processedlogs > > Partition : logdate > > > > Running below commands parellely, > > LOAD DATA INPATH '/logs/processed/2013-04-01' OVERWRITE INTO TABLE > processedlogs PARTITION(logdate='2013-04-01'); > > LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE > processedlogs PARTITION(logdate='2013-04-02'); > > LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE > processedlogs PARTITION(logdate='2013-04-03'); > > LOAD DATA INPATH '/logs/processed/2013-04-02' OVERWRITE INTO TABLE > processedlogs PARTITION(logdate='2013-04-04'); > > ..... > > LOAD DATA INPATH '/logs/processed/2013-04-30' OVERWRITE INTO TABLE > processedlogs PARTITION(logdate='2013-04-30'); > > > > Thanks > > Selva > > > > > > > > > > > > > > > -- -- selva
