> On 2010-09-16 13:28:48, John Sichi wrote: > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, > > line 499 > > <http://review.cloudera.org/r/826/diff/1/?file=11523#file11523line499> > > > > Doesn't this error message need to change?
Updated the comment to "' should be mapped to Map<? extends LazyPrimitive<?, ?>,?>, that is " + "the Key for the map should be of primitive type, but is ... " > On 2010-09-16 13:28:48, John Sichi wrote: > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, > > line 623 > > <http://review.cloudera.org/r/826/diff/1/?file=11523#file11523line623> > > > > I don't understand these TODO's. Removed/updated comment. > On 2010-09-16 13:28:48, John Sichi wrote: > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java, > > line 76 > > <http://review.cloudera.org/r/826/diff/1/?file=11523#file11523line76> > > > > We keep adding new List data members. Probably time to move to a > > single List<ColumnMapping>, with a new class ColumnMapping with fields for > > familyName, familyNameBytes, qualifierName, qualifierNameBytes, > > familyBinary, qualifierBinary. That will be a lot cleaner and also allow > > you to avoid the boolean [] here, which is a little clumsy. I have changed the code to use List<ColumnMapping> with the fields of interest as members of this data class. > On 2010-09-16 13:28:48, John Sichi wrote: > > trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java, > > line 480 > > <http://review.cloudera.org/r/826/diff/1/?file=11526#file11526line480> > > > > Why is this assertion commented out? I have removed this test. We do have coverage from the .q files for this case. This was failing due to small differences in the byte arrays from DataOutputStream/DataInputStream vs o.a.h.hbase.utils.Bytes. - bkm ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/826/#review1247 ----------------------------------------------------------- On 2010-10-21 20:11:06, bkm wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > http://review.cloudera.org/r/826/ > ----------------------------------------------------------- > > (Updated 2010-10-21 20:11:06) > > > Review request for Hive Developers and John Sichi. > > > Summary > ------- > > This addresses HIVE-1245 in part, for atomic or primitive types. > > The serde property "hbase.columns.storage.types" = "-,b,b,b,b,b,b,b,b" is a > specification of the storage option for the corresponding column in the serde > property "hbase.columns.mapping". Allowed values are '' for table default, > 's' for standard string storage, and 'b' for binary storage as would be > obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families > use a colon separated pair such as 's:b' for the key and value part > specifiers respectively. See the test cases and queries for HBase handler for > additional examples. > > There is also a table property "hbase.table.default.storage.type" = "string" > to specify a table level default storage type. The other valid specification > is "binary". The table level default is overridden by a column level > specification. > > This control is available for the boolean, tinyint, smallint, int, bigint, > float, and double primitive types. The attached patch also relaxes the > mapping of map types to HBase column families to allow any primitive type to > be the map key. > > > This addresses bug HIVE-1634. > http://issues.apache.org/jira/browse/HIVE-1634 > > > Diffs > ----- > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java > 1023967 > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java > 1023967 > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsPublisher.java > 1023967 > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java > 1023967 > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java > 1023967 > > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java > 1023967 > trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java > 1023967 > > trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestSetup.java > 1023967 > > trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java > 1023967 > > trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java > 1023967 > trunk/hbase-handler/src/test/queries/hbase_binary_external_table_queries.q > PRE-CREATION > trunk/hbase-handler/src/test/queries/hbase_binary_map_queries.q > PRE-CREATION > trunk/hbase-handler/src/test/queries/hbase_binary_storage_queries.q > PRE-CREATION > > trunk/hbase-handler/src/test/results/hbase_binary_external_table_queries.q.out > PRE-CREATION > trunk/hbase-handler/src/test/results/hbase_binary_map_queries.q.out > PRE-CREATION > trunk/hbase-handler/src/test/results/hbase_binary_storage_queries.q.out > PRE-CREATION > > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBooleanBinary.java > PRE-CREATION > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyByteBinary.java > PRE-CREATION > > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyDoubleBinary.java > PRE-CREATION > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java > 1023967 > > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFloatBinary.java > PRE-CREATION > > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyIntegerBinary.java > PRE-CREATION > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyLongBinary.java > PRE-CREATION > > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyShortBinary.java > PRE-CREATION > trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java > 1023967 > > Diff: http://review.cloudera.org/r/826/diff > > > Testing > ------- > > The HBase handler tests TestHBaseSerDe, TestLazyHBaseObject, > TestHBaseCliDriver, and TestHBaseMinimrCliDriver pass. > > New tests have been added to TestHBaseSerDe and TestLazyHBaseObject to test > this feature. > > New queries which exercise this feature have been added to query files > hbase_binary_map_queries.q and hbase_binary_storage_queries.q. > > > Thanks, > > bkm > >
