Re: how to extract metadata of hive tables in speed

Thanks for the info too! :) > On Apr 25, 2018, at 11:12 AM, 侯宗田 wrote: > > Hi, > > Thank you, I have looked up the source code of Hcatalog, it seems every time > when I run hcat -e “query”, it called hcatCli, then it make configuration, > create and start a session, then dump it after being u

Re: how to extract metadata of hive tables in speed

Hi, Thank you, I have looked up the source code of Hcatalog, it seems every time when I run hcat -e “query”, it called hcatCli, then it make configuration, create and start a session, then dump it after being used. It can’t keep a session or connection and don’t have a Cli. The initialization t

Re: how to extract metadata of hive tables in speed

Hi Hou, Kudu uses the Thrift HMS interface, and written in C. An example could be found here: https://github.com/apache/kudu/tree/master/src/kudu/hms As for parametrizing Hcatalog I have only found this: https://cwiki.apache.org/confluen

Re: how to extract metadata of hive tables in speed

Hi, Peter: I have started a standalone metastore server and it indeed short that part of time, it does connection instead of initialization. But I still have some questions, First, I believe the Hcatalog must be quick because it is a mature product and I have not seen others complaining about th

Re: how to extract metadata of hive tables in speed

Hi, This really helps me a lot, and I will try it soon, Thank you very much! Hou > 在 2018年4月23日，下午3:57，Peter Vary 写道： > > Hi, > > Disclaimer: I am not too familiar with the webhcat yet. > From the logs, I see, that: > - the first 3 seconds spent on starting a new session, and maybe a driver -

Re: how to extract metadata of hive tables in speed

Hi, Disclaimer: I am not too familiar with the webhcat yet. From the logs, I see, that: - the first 3 seconds spent on starting a new session, and maybe a driver - this can be reduced, if the session is already there, and the HiveServer2 is started (but do not know if webhcat could use HS2, or r

Re: how to extract metadata of hive tables in speed

Thank you very much for your reply, I am wondering whether I use the webhcat rightly, I don’t think it is normal to create all the directories and objects to get a table describ and take 8 seconds. The webhcat should not be so slow, Or it is because I forget to start some server which can respon

Re: how to extract metadata of hive tables in speed

Hi, Alexander Kolbasov has a project which might interest you (keeping in mind, that this is not production ready - more like a proof of concept): https://github.com/akolb1/gometastore/blob/master/hmstool/doc/hmstool.md Also you can use HMS thrift API directly to access the MetaStore, or if you c

Re: how to extract metadata of hive tables in speed

Can anyone give me some suggestions? I have been stuck in this problem for several days. Need help!! > 在 2018年4月22日，下午9:38，侯宗田 写道： > > > Hi, > > I am writing a application which needs the metastore about hive tables. I > have used webhcat to get the information about tables and process them.