Hi Martijn. > So this implementation would not rely in any way on Hive, only on Thrift?
Yes. The dependency is light. We also can just copy the iface file from the Hive repo and maintain by ourselves. Best, Shengkai Martijn Visser <martijnvis...@apache.org> 于2022年5月4日周三 21:44写道： > Hi Shengkai, > > > Actually we will only rely on the API in the Hive, which only contains > the thrift file and the generated code > > So this implementation would not rely in any way on Hive, only on Thrift? > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi, Jark and Martijn > > > > Thanks for your feedback. > > > > > Kyuubi provides three ways to configure Hive metastore [1]. Could we > > provide similar abilities? > > > > Yes. I have updated the FLIP about this and it takes some time to figure > > out how the jdbc driver works. I added the section about how to use the > > hive JDBC to configure the session-level catalog. > > > > > I think we can improve the "HiveServer2 Compatibility" section. > > > > Yes. I have updated the FLIP and added more details about the > > compatibility. > > > > > Prefer to first complete the discussion and vote on FLIP-91 then > discuss > > FLIP-223 > > > > Of course. We can wait until the discussion of the FLIP-91 finishes. > > > > > Maintenance concerns about the hive > > > > Actually we will only rely on the API in the Hive, which only contains > the > > thrift file and the generated code[1]. I think it will not influence us > to > > upgrade the java version. > > > > [1] https://github.com/apache/hive/tree/master/service-rpc > > > > Best, > > Shengkai > > > > Martijn Visser <martijnvis...@apache.org> 于2022年4月26日周二 20:44写道： > > > > > Hi all, > > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a couple > of > > > questions/concerns: > > > > > > 1. What is the relationship between this FLIP and FLIP-91? My > assumption > > > would be that this FLIP (and therefore the HiveServer2) implementation > > > would need to be integrated in the REST Gateway, is that correct? If > so, > > I > > > would prefer to first complete the discussion and vote on FLIP-91, else > > > we'll have two moving FLIPs who have a direct relationship with each > > other. > > > > > > 2. While I understand that Hive is important (in the Chinese ecosystem, > > not > > > so much in Europe and the US), I still have maintenance concerns on > this > > > topic. We know that the current Hive integration isn't exactly ideal > and > > > requires a lot of work to get in better shape. At the same time, Hive > > still > > > doesn't support Java 11 while we need (and should, given the premier > > > support has ended already) to move away from Java 8. > > > > > > Best regards, > > > > > > Martijn Visser > > > https://twitter.com/MartijnVisser82 > > > https://github.com/MartijnVisser > > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> wrote: > > > > > > > Thank Shengkai for driving this effort, > > > > I think this is an essential addition to Flink Batch. > > > > > > > > I have some small suggestions: > > > > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could > we > > > > provide similar abilities? > > > > Especially with the JDBC Connection URL, users can visit different > Hive > > > > metastore server instances. > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility" section. > > > > We need to figure out two compatibility matrices. One is SQL Gateway > > with > > > > different versions of Hive metastore, > > > > and the other is different versions of Hive client (e.g., Hive JDBC) > > with > > > > SQL Gateway. We need to clarify > > > > what metastore and client versions we support and how users configure > > the > > > > versions. > > > > > > > > Best, > > > > Jark > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > > > > > Hi, Jiang. > > > > > > > > > > Thanks for your feedback! > > > > > > > > > > > Integrating the Hive ecosystem should not require changing the > > > service > > > > > interface > > > > > > > > > > I move the API change to the FLIP-91. But I think it's possible we > > add > > > > more > > > > > interfaces to intergrate the new endpoints in the future because > > every > > > > > endpoints's functionality is different. For example, the REST > > endpoint > > > > > doen't support to fetch operation-level logs but the hiveserver2 > > > endpoint > > > > > supports. In this case, we need to modify the shared GatewayService > > to > > > > > support the functionality exposed by the new endpint. > > > > > > > > > > > How to support different Hive versions? > > > > > > > > > > Do you means to support the different HiveServer2 version? The > > > > HiveServer2 > > > > > uses the version to guarantee the compatibility. During the > > > openSession, > > > > > the client and server will determine the protocol > > > version(minimun(client > > > > > version, hiveendpoint version)). After that the client and the > server > > > > uses > > > > > the determined version to communicate. In the HiveServer2 endpoint, > > it > > > > > determines how the endpoint deserialize the results and the result > > > > schema. > > > > > I add a section about HiveServer2 compatiblity. > > > > > > > > > > > Could you please fully provide its definition including input > > > > parameters > > > > > and the corresponding return value schema? > > > > > > > > > > Because we implements the interface exposed by the Hive. So I add > the > > > > file > > > > > link to the HiveServer2 interfaces[1], which contains all input > > > > parameters > > > > > and the results. Considering the file doesn't contain the output > for > > > the > > > > > Operation, I add the output schema for all the supported Operation > in > > > the > > > > > FLIP, which is not covered in the link. Hope these can address your > > > > > question. > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:43写道： > > > > > > > > > > > Hi Shengkai. > > > > > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint support. > > For > > > > the > > > > > > "GatewayService API Change", I don't think the motivation for > > > > supporting > > > > > > HiveServer2 endpoint need to change the GatewayService API, in > > other > > > > > words, > > > > > > integrating the Hive ecosystem should not require changing the > > > service > > > > > > interface. If you confirm to change GatewayService interface, > IMO, > > > the > > > > > > proposal could be discussed in FLIP-91 because the public > > interfaces > > > > are > > > > > > defined in FLIP-91. > > > > > > > > > > > > In addtion, how to support different Hive versions and how to > > > guarantee > > > > > > compatibility is not mentioned in the design. What's the behavior > > of > > > > the > > > > > > compatibility? > > > > > > > > > > > > Finally, for the public interfaces, could you please fully > provide > > > its > > > > > > definition including input parameters and the corresponding > return > > > > value > > > > > > schema? > > > > > > > > > > > > Thanks, > > > > > > Nicholas Jiang > > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > > > > Hi, Flink developers. > > > > > > > > > > > > > > I want to start a discussion about the FLIP-223: Support > > > HiveServer2 > > > > > > > Endpoint[1]. The Endpoint will implement the thrift interface > > > exposed > > > > > by > > > > > > > the HiveServer2, and users' BI, CLI and other tools based on > the > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL > > > Gateway. > > > > > > After > > > > > > > the FLIP finishes, the users can have almost the same > experience > > in > > > > the > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the > > > > HiveServer2. > > > > > > > > > > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > > > > > > > Best, > > > > > > > Shengkai > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > > > > > > > > > > > > > > > > > > > >
