Re: threading with hive client

2013-08-16 Thread Brock Noland
https://issues.apache.org/jira/browse/HIVE-4239 https://issues.apache.org/jira/browse/HIVE-80 I would guess this comment on HIVE-80 is still applicable: "there may still be some thread-unsafe code, but no one knows for sure. Given that, the only approach may be to do as much review as possible (e

Re: threading with hive client

2013-08-16 Thread Kristopher Glover
One more question. If the SemanticAnalyzer isn't fully thread safe could you provide any pointers as to why it may not be thread safe? It's a 9000 line file so any hints as to where to get started would be much appreciated. I don't see anything very obvious like globally shared member variables so

Re: threading with hive client

2013-08-15 Thread Kristopher Glover
Thanks for all the great insight. I'll poke around a little more to see if I could at least start documenting the changes required to make everything thread safe as well as remove the synchronization. @Xuefu- I completely understand your points, I was just trying to figure out if there was a sp

Re: threading with hive client

2013-08-15 Thread Xuefu Zhang
To add, 1. Being public doesn't necessarily guarantee thread-safety. Of course, this is no excuse for not documenting thread-safety. 2. Sometimes a method is made public for testing, which is bad in my opnion, but I saw many instances like this before. --Xuefu On Thu, Aug 15, 2013 at 1:11 PM,

Re: threading with hive client

2013-08-15 Thread Kristopher Glover
Interesting, I didn't realize that. If that's the case then I suppose it'd be really bad for me to circumvent the lock by reproducing the Driver#run method by calling Driver#compile and Driver#execute directly from within my app. If that is the case why make Driver#compile and Driver#execute publ

Re: threading with hive client

2013-08-15 Thread Brock Noland
Well you would have probably found the areas we need to fix! :) The hive source is is not strict about methods and member visibility. The good news is that we have been making significant improvements in this aspect. Brock On Thu, Aug 15, 2013 at 2:55 PM, Kristopher Glover wrote: > Interesting,

Re: threading with hive client

2013-08-15 Thread Brock Noland
The hive semantic analyzer is not fully thread safe. We'd like to remove that lock but it will be a large project. Brock On Thu, Aug 15, 2013 at 11:12 AM, Kristopher Glover wrote: > Hi Everyone, > > I'm experiencing a threading issue with the Hive client where I want to > run multiple queries

threading with hive client

2013-08-15 Thread Kristopher Glover
Hi Everyone, I'm experiencing a threading issue with the Hive client where I want to run multiple queries on the same JVM. The problem I'm having is that org.apache.hadoop.hive.ql.Driver#run (line 907) has the following few lines of code : synchronized (compileMonitor) { ret = compil