+1 On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <cloud0...@gmail.com> wrote:
> +1 for the new thrift server to get rid of the Hive dependencies! > > On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <wgy...@gmail.com> wrote: > >> I'm +1 for this SPIP for these two reasons: >> >> 1. The current thriftserver has some issues that are not easy to solve, >> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>. >> 2. The difference between the version of ORC we are using and the >> built-in Hive is using is getting bigger and bigger. We can't ensure that >> there will be no compatibility issues in the future. If thriftserver does >> not depend on Hive, it will be much easier to upgrade the built-in Hive in >> the future. >> >> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <angers....@gmail.com> wrote: >> >>> Hi all, >>> >>> I have complete a Design doc about how to use and config this new thrift >>> server, and some design detail about change and impersonation. >>> >>> Hope for your suggestions and ideas. >>> >>> SPIP DOC : >>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 >>> Design DOC : >>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14 >>> Thrift server about configurations >>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web&ouid=110963191229426834922 >>> >>> Best Regards >>> angers.zhu >>> angers....@gmail.com >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>> --------- Forwarded Message --------- >>> From： angers.zhu <angers....@gmail.com> <angers....@gmail.com> >>> Date： 12/18/2019 22:29 >>> To： dev-ow...@spark.apache.org <dev-ow...@spark.apache.org> >>> <dev-ow...@spark.apache.org> >>> Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based >>> on protocol v11 >>> >>> Add spark-dev group access privilege to google. >>> angers.zhu >>> angers....@gmail.com >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>> On 12/18/2019 22:02，Sandeep Katta<sandeep0102.opensou...@gmail.com> >>> <sandeep0102.opensou...@gmail.com> wrote： >>> >>> I couldn't access the doc, please give permission to the spark-dev group >>> >>> On Wed, 18 Dec 2019 at 18:05, angers.zhu <angers....@gmail.com> wrote: >>> >>>> With the development of Spark and Hive，in current sql/hive-thriftserver >>>> module, >>>> >>>> we need to do a lot of work to solve code conflicts for different >>>> built-in hive versions. >>>> >>>> It's an annoying and unending work in current ways. And these issues >>>> have limited >>>> >>>> our ability and convenience to develop new features for Spark’s thrift >>>> server. >>>> >>>> We suppose to implement a new thrift server and JDBC driver based >>>> on Hive’s latest v11 >>>> >>>> TCLService.thrift thrift protocol. Finally, the new thrift server have >>>> below feature: >>>> >>>> 1. >>>> >>>> Build new module spark-service as spark’s thrift server >>>> 2. >>>> >>>> Don't need as much reflection and inherited code as >>>> `hive-thriftser` modules >>>> 3. >>>> >>>> Support all functions current `sql/hive-thriftserver` support >>>> 4. >>>> >>>> Use all code maintained by spark itself, won’t depend on Hive >>>> 5. >>>> >>>> Support origin functions use spark’s own way, won't limited by >>>> Hive's code >>>> 6. >>>> >>>> Support running without hive metastore or with hive metastore >>>> 7. >>>> >>>> Support user impersonation by Multi-tenant splited hive >>>> authentication and DFS authentication >>>> 8. >>>> >>>> Support session hook for with spark’s own code >>>> 9. >>>> >>>> Add a new jdbc driver spark-jdbc, with spark’s own connection url >>>> “jdbc:spark:<host>:<port>/<db>” >>>> 10. >>>> >>>> Support both hive-jdbc and spark-jdbc client, then we can support >>>> most clients and BI platform >>>> >>>> >>>> >>>> https://issues.apache.org/jira/browse/SPARK-29018 >>>> >>>> Google Doc: >>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit# >>>> >>>> [ ] +1: Accept the proposal as an official SPIP >>>> [ ] +0 >>>> [ ] -1: I don't think this is a good idea because ... >>>> >>>> I'll start with my +1 >>>> angers.zhu >>>> angers....@gmail.com >>>> >>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D> >>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>>> >>>>
