Thanks for the inputs. It turned out that the scalability and performance issues that I ran into are due to the driver. It was a JDBC driver from a vendor.
I am reasonably convinced that JDBC might be a better approach. How about the support for the complex data types? I do not see any JDBC drivers that support any of the complex data types such as Maps, arrays, structs etc...Is it in plan for the Apache JDBC drivers? Regards On 5 Jan 2016 02:42, "Prasad Mujumdar" <pras...@apache.org> wrote: > I would strong second Thejas's points. A custom thrift client application > won't help you with SQL compatibility and perf/scalability would also be > doubtful .. It would be a bit of work to build the client and more > importantly to maintain it. Note that the RPC and authentication mechanisms > could change in future and you'll have to handle those changes yourself for > next upgrade. > > thanks > Prasad > > > On Mon, Jan 4, 2016 at 11:32 AM, Thejas Nair <thejas.n...@gmail.com> > wrote: > > > Can you please elaborate on the scalability/performance issues you > > faced ? What kind of scale do you need and what did you see in your > > results ? > > Why do you feel switching to direct thrift api is likely to help with > > performance ? The jdbc driver is also using that api. You seem to be > > effectively reinventing the wheel by writing another stripped down > > jdbc/odbc driver. It would be better to channel your efforts in fixing > > the existing jdbc driver, so that you don't end up duplicating a lot > > of work. > > > > > > Regarding SQL standards compliance, hive 1.2.1 has made lot of > > progress in regarding that aspect. Just like any other SQL database, > > hive happens to have many extensions to the standard. > > > > > > On Wed, Dec 30, 2015 at 6:52 PM, Srinivas M <smudigo...@gmail.com> > wrote: > > > Thanks Thejas for prompt response. > > > > > > I should have elaborated the question a bit more. When I said ODBC / > JDBC > > > is not an option, what I mean was, we have existing ODBC / JDBC > > > applications and they generate SQL statements adhering to the SQL92 > > > standard. Since HiveQL is different from it and supports syntax for > Load > > > etc, reuse of the existing ODBC / JDBC applications may not be > feasible. > > > But, for writing a new module specific to the Hive, we can still use > any > > > interface, ODBC / JDBC / Hive Thrift and currently weighing the best > > > options. > > > > > > Here are the reasons for considering Thrift Hive. > > > > > > 1. While evaluating different JDBC drivers, we have ran into > scalability > > > and performance issues with the JDBC Drivers. > > > 2. During the evaluation of the JDBC drivers, ran into situations > where, > > > not all the features that we were looking for are supported by any > > > particular driver. > > > 3. The expectation was, thrift API would provide better performance > over > > > the JDBC / ODBC. > > > > > > Need to implement the following capabilities > > > 1. Kerberos authentication > > > 2. Support to generate native HiveQL and partitioning capabilities. > > > 3. Performance is one of the key criteria. > > > > > > Based on the above, after evaluating some of the ODBC / JDBC drivers, > > since > > > I have seen some performance issues and functionality issues, I am > > > exploring on the Thrift Hive client API with an assumption that some of > > the > > > current shortfalls with the ODBC / JDBC drivers can be addressed by > using > > > the Client API. The initial hiccup that I ran into is with the > > > authentication. > > > > > > Hope this gives enough background. > > > > > > In case if I can get a better throughput using the JDBC option itself, > > then > > > I can still consider using the JDBC as opposed to the Thrift Hive. > Right > > > now, evaluating the available options. > > > > > > Regards > > > > > > > > > > > > > > > On Wed, Dec 30, 2015 at 11:45 PM, Thejas Nair <thejas.n...@gmail.com> > > wrote: > > > > > >> Srinivas, > > >> Can you please elaborate on why ODBC/JDBC is not an option ? > > >> I didn't understand what you meant by "unable to generate the HiveQL" > > >> with those options. How does using thrift api directly help in that > > >> case ? > > >> > > >> ODBC/JDBC is the preferred API for users. There are many features > > >> implemented in those layers, including the security and also high > > >> availability features. > > >> Incorrect use of the thrift api can potentially lead to other issues > > >> like memory leaks in HiveServer2. > > >> > > >> > > >> > > >> > > >> > > >> > > >> On Wed, Dec 30, 2015 at 8:37 AM, Srinivas M <smudigo...@gmail.com> > > wrote: > > >> > Hi > > >> > > > >> > I am trying to develop a custom application using the Thrift Hive > > client > > >> > interface to access the Hive and read and write into the hive > tables. > > >> ODBC > > >> > and JDBC are not option because of the inherent limitations with > those > > >> > interfaces (i.e unable to generate the HiveQL etc). > > >> > > > >> > While using the Thrift Hive interface, the first challenge that I > ran > > >> into > > >> > is with the authentication. I could able to connect to hive only > when > > I > > >> set > > >> > the authentication of the hiveserver2 as NOSASL. > > >> > My application should be able to provide Kerberos authentication and > > all > > >> > other authentications supported by Hive. > > >> > > > >> > I had tried to look for some samples around implementing > > authentication > > >> in > > >> > the Thrift Hive client, but I could not find much details on that. > > >> > Can someone help me understand the options available for > implementing > > >> > authentication while using the Thrift Hive client interfaces ? > > >> > > > >> > -- > > >> > Srinivas > > >> > (*-*) > > >> > > > >> > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > >> > You have to grow from the inside out. None can teach you, none can > > make > > >> you > > >> > spiritual. > > >> > -Narendra Nath Dutta(Swamy Vivekananda) > > >> > > > >> > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > >> > > > > > > > > > > > > -- > > > Srinivas > > > (*-*) > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > > You have to grow from the inside out. None can teach you, none can make > > you > > > spiritual. > > > -Narendra Nath Dutta(Swamy Vivekananda) > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > >
