> On 2011-08-03 22:39:04, Paul Yang wrote: > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, lines > > 1240-1249 > > <https://reviews.apache.org/r/1259/diff/1/?file=30272#file30272line1240> > > > > Why are we saving unboundKey? Can we just break on the first instance > > when partSpec does not contain fs.getName()?
No, we want to support prefix specs, so at the and there can be unbound keys. The error is thrown only when there's a "hole" inside of specification. > On 2011-08-03 22:39:04, Paul Yang wrote: > > trunk/data/conf/hive-site.xml, line 142 > > <https://reviews.apache.org/r/1259/diff/1/?file=30262#file30262line142> > > > > Why is a hook needed to update this count? Count is updated in archive, unarchive and drop. It has been decided, that it will be more clear to keep it in hook than add to drop (and maybe in future into other?) tasks. > On 2011-08-03 22:39:04, Paul Yang wrote: > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MetaUtils.java, line 119 > > <https://reviews.apache.org/r/1259/diff/1/?file=30273#file30273line119> > > > > The max parameter for getPartitions is not yet implemented. It needs to > > be or otherwise this will hang for large tables. Well, you mean about class implementation or lower level one? I've added it to Hive class, but it works only under the assumption HiveMetaStoreClient.listPartitionsWithAuthInfo supports it (and it seems to depend on ThriftMetastoreIface.get_partitions_ps_with_auth). > On 2011-08-03 22:39:04, Paul Yang wrote: > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 3359 > > <https://reviews.apache.org/r/1259/diff/1/?file=30272#file30272line3359> > > > > This is going to be feature that we'll need to support. The script that > > drops based on retention can't unarchive as it will take too long. Yes, but it's quite hard to keep it atomic (many changes in metastore and then archive deletion) and it will need refactoring and I don't want to make patch even bigger. When we use single archives everything should be ok, so we don't drop backward compatibility. - Marcin ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1259/#review1278 ----------------------------------------------------------- On 2011-08-02 21:34:07, Marcin Kurczych wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/1259/ > ----------------------------------------------------------- > > (Updated 2011-08-02 21:34:07) > > > Review request for hive, Paul Yang and namit jain. > > > Summary > ------- > > Allowing archiving at chosen level. When table is partitioned by ds, hr, min > it allows archiving at ds level, hr level and min level. Corresponding > syntaxes are: > ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08'); > ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11'); > ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11', min='30'); > > You cannot do much to archived partitions. You can read them. You cannot > write to them / overwrite them. You can drop single archived partitions, but > not parts of bigger archives. > > > Diffs > ----- > > trunk/data/conf/hive-site.xml 1153271 > trunk/metastore/if/hive_metastore.thrift 1153271 > trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1153271 > trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1153271 > > trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java > 1153271 > > trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php > 1153271 > trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1153271 > trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1153271 > > trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java > 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MetaUtils.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1153271 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateArchivedCounter.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java > 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java > 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 1153271 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java > 1153271 > trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION > trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION > trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION > trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive1.q.out 1153271 > trunk/ql/src/test/results/clientnegative/archive2.q.out 1153271 > trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out > PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out > PRE-CREATION > trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION > > Diff: https://reviews.apache.org/r/1259/diff > > > Testing > ------- > > Added unittests for normal uses. > Tried recovery during manual tests (had to simulate exceptions in different > places in code, which shouldn't be committed, that's why it had to be manual > testing). > > > Thanks, > > Marcin > >
