Hi, Jark and Martijn Thanks for your feedback.
> Kyuubi provides three ways to configure Hive metastore [1]. Could we provide similar abilities? Yes. I have updated the FLIP about this and it takes some time to figure out how the jdbc driver works. I added the section about how to use the hive JDBC to configure the session-level catalog. > I think we can improve the "HiveServer2 Compatibility" section. Yes. I have updated the FLIP and added more details about the compatibility. > Prefer to first complete the discussion and vote on FLIP-91 then discuss FLIP-223 Of course. We can wait until the discussion of the FLIP-91 finishes. > Maintenance concerns about the hive Actually we will only rely on the API in the Hive, which only contains the thrift file and the generated code[1]. I think it will not influence us to upgrade the java version. [1] https://github.com/apache/hive/tree/master/service-rpc Best, Shengkai Martijn Visser <martijnvis...@apache.org> 于2022年4月26日周二 20:44写道： > Hi all, > > I'm not too familiar with Hive and HiveServer2, but I do have a couple of > questions/concerns: > > 1. What is the relationship between this FLIP and FLIP-91? My assumption > would be that this FLIP (and therefore the HiveServer2) implementation > would need to be integrated in the REST Gateway, is that correct? If so, I > would prefer to first complete the discussion and vote on FLIP-91, else > we'll have two moving FLIPs who have a direct relationship with each other. > > 2. While I understand that Hive is important (in the Chinese ecosystem, not > so much in Europe and the US), I still have maintenance concerns on this > topic. We know that the current Hive integration isn't exactly ideal and > requires a lot of work to get in better shape. At the same time, Hive still > doesn't support Java 11 while we need (and should, given the premier > support has ended already) to move away from Java 8. > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> wrote: > > > Thank Shengkai for driving this effort, > > I think this is an essential addition to Flink Batch. > > > > I have some small suggestions: > > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could we > > provide similar abilities? > > Especially with the JDBC Connection URL, users can visit different Hive > > metastore server instances. > > > > 2) I think we can improve the "HiveServer2 Compatibility" section. > > We need to figure out two compatibility matrices. One is SQL Gateway with > > different versions of Hive metastore, > > and the other is different versions of Hive client (e.g., Hive JDBC) with > > SQL Gateway. We need to clarify > > what metastore and client versions we support and how users configure the > > versions. > > > > Best, > > Jark > > > > > > [1]: > > > > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com> wrote: > > > > > Hi, Jiang. > > > > > > Thanks for your feedback! > > > > > > > Integrating the Hive ecosystem should not require changing the > service > > > interface > > > > > > I move the API change to the FLIP-91. But I think it's possible we add > > more > > > interfaces to intergrate the new endpoints in the future because every > > > endpoints's functionality is different. For example, the REST endpoint > > > doen't support to fetch operation-level logs but the hiveserver2 > endpoint > > > supports. In this case, we need to modify the shared GatewayService to > > > support the functionality exposed by the new endpint. > > > > > > > How to support different Hive versions? > > > > > > Do you means to support the different HiveServer2 version? The > > HiveServer2 > > > uses the version to guarantee the compatibility. During the > openSession, > > > the client and server will determine the protocol > version(minimun(client > > > version, hiveendpoint version)). After that the client and the server > > uses > > > the determined version to communicate. In the HiveServer2 endpoint, it > > > determines how the endpoint deserialize the results and the result > > schema. > > > I add a section about HiveServer2 compatiblity. > > > > > > > Could you please fully provide its definition including input > > parameters > > > and the corresponding return value schema? > > > > > > Because we implements the interface exposed by the Hive. So I add the > > file > > > link to the HiveServer2 interfaces[1], which contains all input > > parameters > > > and the results. Considering the file doesn't contain the output for > the > > > Operation, I add the output schema for all the supported Operation in > the > > > FLIP, which is not covered in the link. Hope these can address your > > > question. > > > > > > Best, > > > Shengkai > > > > > > [1] > > > > > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:43写道： > > > > > > > Hi Shengkai. > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint support. For > > the > > > > "GatewayService API Change", I don't think the motivation for > > supporting > > > > HiveServer2 endpoint need to change the GatewayService API, in other > > > words, > > > > integrating the Hive ecosystem should not require changing the > service > > > > interface. If you confirm to change GatewayService interface, IMO, > the > > > > proposal could be discussed in FLIP-91 because the public interfaces > > are > > > > defined in FLIP-91. > > > > > > > > In addtion, how to support different Hive versions and how to > guarantee > > > > compatibility is not mentioned in the design. What's the behavior of > > the > > > > compatibility? > > > > > > > > Finally, for the public interfaces, could you please fully provide > its > > > > definition including input parameters and the corresponding return > > value > > > > schema? > > > > > > > > Thanks, > > > > Nicholas Jiang > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > > Hi, Flink developers. > > > > > > > > > > I want to start a discussion about the FLIP-223: Support > HiveServer2 > > > > > Endpoint[1]. The Endpoint will implement the thrift interface > exposed > > > by > > > > > the HiveServer2, and users' BI, CLI and other tools based on the > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL > Gateway. > > > > After > > > > > the FLIP finishes, the users can have almost the same experience in > > the > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the > > HiveServer2. > > > > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > > > > > > >
