Thanks Alan for detailed explanation. Please bear in mind that any tool that needs to work with some repository (Oracle TimesTen IMDB has its metastore on Oracle classic), SAP Replication Server has its repository RSSD on SAP ASE and others First thing they do, they go and cache those tables and keep it in memory of the big brother database until they are shutdown. I reversed engineered and created Hive data model from physical schema (on Oracle). There are around 194 tables in total that can be easily cached.
For small medium enterprise (SME), they don't really have much data so anything will do and they are the ones that use open source databases. For bigger companies, they already pay bucks for Oracle and alike and they are the one that would not touch an open source database (not talking about big data), because in this new capital-sensitive risk-averse world, they do not want to expose themselves to unnecessary risk. So I am not sure whether they will take something like Hbase as a core product, unless it is going to be maintenance free. Going back to your point ".. but you have to pay for an expensive commercial license to make the metadata really work well is a non-starter" They already do and pay more if they have to. We will stick with Hive metadata on Oracle with schema on SSD . HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 24 October 2016 at 20:14, Alan Gates <alanfga...@gmail.com> wrote: > Some thoughts on this: > > First, there’s no plan to remove the option to use an RDBMS such as Oracle > as your backend. Hive’s RawStore interface is built such that various > implementations of the metadata storage can easily coexist. Obviously > different users will make different choices about what metadata store makes > sense for them. > > As to why HBase: > 1) We desperately need to get rid of the ORM layer. It’s causing us > performance problems, as evidenced by things like it taking several minutes > to fetch all of the partition data for queries that span many partitions. > HBase is a way to achieve this, not the only way. See in particular > Yahoo’s work on optimizing Oracle access https://issues.apache.org/ > jira/browse/HIVE-14870 The question around this is whether we can > optimize for Oracle, MySQL, Postgres, and SQLServer without creating a > maintenance and testing nightmare for ourselves. I’m skeptical, but others > think it’s possible. See comments on that JIRA. > > 2) We’d like to scale to much larger sizes, both in terms of data and > access from nodes. Not that we’re worried about the amount of metadata, > but we’d like to be able to cache more stats, file splits, etc. And we’d > like to allow nodes in the cluster to contact the metastore, which we do > not today since many RDBMSs don’t handle a thousand plus simultaneous > connections well. Obviously both data and connection scale can be met with > high end commercial stores. But saying that we have this great open source > database but you have to pay for an expensive commercial license to make > the metadata really work well is a non-starter. > > 3) By using tools within the Hadoop ecosystem like HBase we are helping to > drive improvement in the system > > To explain the HBase work a little more, it doesn’t use Phoenix, but works > directly against HBase, with the help of a transaction manager (Omid). In > performance tests we’ve done so far it’s faster than Hive 1 with the ORM > layer, but not yet to the 10x range that we’d like to see. We haven’t yet > done the work to put in co-processors and such that we expect would speed > it up further. > > Alan. > > > On Oct 23, 2016, at 15:46, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > > > > > A while back there was some notes on having Hive metastore on Hbase as > opposed to conventional RDBMSs > > > > I am currently involved with some hefty work with Hbase and Phoenix for > batch ingestion of trade data. As long as you define your Hbase table > through Phoenix and with secondary Phoenix indexes on Hbase, the speed is > impressive. > > > > I am not sure how much having Hbase as Hive metastore is going to add to > Hive performance. We use Oracle 12c as Hive metastore and the Hive > database/schema is built on solid state disks. Never had any issues with > lock and concurrency. > > > > Therefore I am not sure what one is going to gain by having Hbase as the > Hive metastore? I trust that we can still use our existing schemas on > Oracle. > > > > HTH > > > > > > > > Dr Mich Talebzadeh > > > > LinkedIn https://www.linkedin.com/profile/view?id= > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > http://talebzadehmich.wordpress.com > > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > >
