I know a solution with column family -> hive map, but HBase doesn't support more than 2 CF well. I need ~10 maps in row.
I think the idea with a flag is very well. I've prepared a patch to be able to use a flag. I am not sure where to include the possibility to define a flag and pass the flag to LazyHBaseCellMap? Let's move the discussion to https://issues.apache.org/jira/browse/HIVE-11329 and please look at my first patch and a unit test. Kindly regards Wojciech Indyk 2015-07-23 17:21 GMT+02:00 kulkarni.swar...@gmail.com <kulkarni.swar...@gmail.com>: > So let me ask you this. If we did not have the support for pulling data via > prefixes, there would be two options for us to pull this data. One, wither > we provide just the column family name like "fam:" and let hive pull > everything under that column family and stuff it in a map with key being > the column name. Or, the other option would be to provide the column names > individually. In either case, the column prefixes would end up in the hive > column name. My intend behind adding this support was to have a shortcut > way which was an extension of the existing support to pull all columns by > providing a "family_name:" to pulling just the columns that start with > given prefix. Everything else should stay same and consistent. That said, I > am ok with adding a flag to hide the prefix in the column name, IMO it > would be confusing for someone to understand why in this particular case > the prefix needs to be hidden vs not in any other case. > > Does that make sense? > > On Thu, Jul 23, 2015 at 9:46 AM, Wojciech Indyk <wojciechin...@gmail.com> > wrote: > >> Hello! >> >> Yes, but if I define a map prefix "tag_" I don't want to receive the >> prefix for each element of the map. I know what the prefix for the map >> is. It is hard to join such data with another structures which doesn't >> have prefixes. All in all it's easier to integrate data without >> prefixes. IMO Prefixes are artificial structure (like 'super-column') >> to optimize queries and be able to store a map in hbase. That's why i >> want to cut prefixes. >> >> What do you think about it? Does it make sense for you? Even if it's >> not a bug it would be nice to have option to hide prefixes in keys of >> map. >> >> Kindly regards >> Wojciech Indyk >> >> >> 2015-07-23 16:32 GMT+02:00 kulkarni.swar...@gmail.com >> <kulkarni.swar...@gmail.com>: >> > Hey, >> > >> > Just so that I understand your issue better, why do you think it should >> be >> > >> > key: one, value: 0.5 >> > key: two: value: 0.5 >> > >> > instead of >> > >> > key: tag_one, value: 0.5 >> > key: tag_two, value: 0.5 >> > >> > when you know that the prefixes for your columns are tag_. Hive won't >> > really do anything but simply pull all the columns that start with the >> > given prefix and add them to the key for your map which is exactly what >> you >> > are seeing here. >> > >> > >> > On Wed, Jul 22, 2015 at 10:03 AM, Wojciech Indyk < >> wojciechin...@gmail.com> >> > wrote: >> > >> >> Hi! >> >> I've created an issue https://issues.apache.org/jira/browse/HIVE-11329 >> >> and need an advice is it a bug or should it be a new feature, e.g. a >> >> flag to enable somewhere in a table definition? >> >> I am eager to create a patch, however I need some help with design a >> >> work to do (e.g. which modules affect this thing). >> >> >> >> Kindly regards >> >> Wojciech Indyk >> >> >> > >> > >> > >> > -- >> > Swarnim >> > > > > -- > Swarnim
