Mich,
Thanks for clarifying. My words were probably misleading :-)
Udit,
Fallback (or HA) is different from load balancing. Having load issues
need not mean it is unreachable, so your second service. You want equal
load on both metastore services so a simple Round Robin load balancer
should do. We use http://www.haproxy.org/ for this. This LB picks one of N
metastore services which all look at the same DB.
That being said, I suggested this without really understanding if it will
actually help you. We use multiple instances coz the number of queries
being serviced at any time are higher than what a single instance could
handle. You need to first find out what's causing the load on your
metastore. It could be that a single service isn't your problem but
something else like :
- How are you querying the metastore? Do your INSERT queries create large
# partitions in production? The inserts could be locking up the mysql
tables for too long hence making them unavailable to others sporadically.
Use any standard mysql perf monitoring to check this or just speak to your
DB admin :-) We use Vividcortex for this which is a paid service.
- Number of partitons picked in your SELECT queries. Hive metastore
doesn't like it when a hive table has too many partitions (under 30k is
recommended I think?)
- Metastore service is JAVA so it can have the usual JVM latencies. What do
your GC pauses look like? Turn on gc logging or use a jvm profiler to
inspect. Solution could be picking better JVM tuning params.
.. there could be other reasons but these should give you a start.
-Gautam.
On Wed, Mar 30, 2016 at 3:33 PM, Udit Mehta <ume...@groupon.com> wrote:
> But dont the clients always pick the first URI for multiple instances
> mentioned in "*hive.metastore.uris" *config and fallback to the others
> only if the first is unreachable? This way, we would still have a
> bottleneck, right?
> Can you give a little more information on your setup and how you enable
> load balancing?
> I think i am missing something here.
>
> Thanks,
> Udit
>
> On Wed, Mar 30, 2016 at 3:20 PM, Gautam <gautamkows...@gmail.com> wrote:
>
>> The metastore service is a java process that is a thrift server .. so you
>> can point multiple such hive metastore instances with
>> "javax.jdo.option.ConnectionURL" poitning to the same mysql db.
>>
>> On Wed, Mar 30, 2016 at 3:11 PM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>>
>>>
>>> Can you clarify this please
>>>
>>> "Have you tried putting multiple metastores behind a load balancer"
>>>
>>> Are you implying that metastore and backend DB are different entities
>>> here.
>>>
>>> As far as I know $HIVE_HOME/bin/hive --service metastore & starts Hive
>>> threads to the backend database/metastore and Hive server2 acts a gateway
>>> for remote access to Hive metastore through beeline or other clients
>>>
>>> There is only one metastore here namely MySQL/Oracle or others.
>>>
>>> Thanks
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn *
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 30 March 2016 at 22:53, Gautam <gautamkows...@gmail.com> wrote:
>>>
>>>> Can you elaborate on where you see the bottleneck? A general overview
>>>> of your access path would be useful. For instance if you'r accessing Hive
>>>> metastore via HiveServer2 or from webhcat using embedded cli or something
>>>> else.
>>>>
>>>> Have you tried putting multiple metastores behind a load balancer? It's
>>>> just a thrift service over mysql so can have multiple instances pointing to
>>>> same backend db.
>>>>
>>>> On Wed, Mar 30, 2016 at 2:28 PM, Udit Mehta <ume...@groupon.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We are currently running Hive in production and staging with the
>>>>> metastore connecting to a MySql database in the backend. The traffic in
>>>>> production accessing the metastore is more than staging which is expected.
>>>>> We have had a sudden increase in traffic which has led to the metastore
>>>>> operation taking a lot longer than before. The same query on staging takes
>>>>> a lot less due to the lesser traffic on the staging cluster.
>>>>>
>>>>> We tried increasing the heap space for the metastore process as well
>>>>> as bumped up the memory for the mysql database. Both these changes did not
>>>>> seem to help much and we still see delays. Is there any other config we
>>>>> can
>>>>> increase to counter this increased traffic? I am looking at config for max
>>>>> threads as well but im not sure if this is the right path ahead.
>>>>>
>>>>> Im wondering if the metastore is a bottleneck here or im missing
>>>>> something.
>>>>>
>>>>> Looking forward to your reply,
>>>>> Udit
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> "If you really want something in this life, you have to work for it.
>>>> Now, quiet! They're about to announce the lottery numbers..."
>>>>
>>>
>>>
>>
>>
>> --
>> "If you really want something in this life, you have to work for it. Now,
>> quiet! They're about to announce the lottery numbers..."
>>
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
