Hi Shengkai, Thanks for driving.
I have a few comments: ## Unsupported features I've read the FLIP and it's not quite clear what the specific unsupported items are? - For example, security related, is it not supported. - For example, is there a loss of precision for types - For example, the FetchResults are not the same ## Support multiple metastore clients in one gateway? > During the setup, the HiveServer2 tires to load the config in the hive-site.xml to initialize the Hive metastore client. In the Flink, we use the Catalog interface to connect to the Hive Metastore, which is allowed to communicate with different Hive Metastore[1]. Therefore, we allows the user to specify the path of the hive-site.xml as the endpoint parameters, which will used to create the default HiveCatalog in the Flink. Considering the different users may have different requirements to connect to different meta stores, they can use the DDL to register the HiveCatalog that satisfies their requirements. I understand it is difficult. You really want to support? ## Hive versions and setup I saw jark also commented, but FLIP does not seem to have been modified, how should the user setup, which jar to add, which hive metastore version to support? How to setup to support? Best, Jingsong On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fskm...@gmail.com> wrote: > Hi, all. > > Considering we start to vote for FLIP-91 for a while, I think we can > restart the discussion about the FLIP-223. > > I am glad that you can give some feedback about FLIP-223. > > Best, > Shengkai > > > Martijn Visser <mart...@ververica.com> 于2022年5月6日周五 19:10写道： > > > Hi Shengkai, > > > > Thanks for clarifying. > > > > Best regards, > > > > Martijn > > > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fskm...@gmail.com> wrote: > > > > > Hi Martijn. > > > > > > > So this implementation would not rely in any way on Hive, only on > > Thrift? > > > > > > Yes. The dependency is light. We also can just copy the iface file > from > > > the Hive repo and maintain by ourselves. > > > > > > Best, > > > Shengkai > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年5月4日周三 21:44写道： > > > > > > > Hi Shengkai, > > > > > > > > > Actually we will only rely on the API in the Hive, which only > > contains > > > > the thrift file and the generated code > > > > > > > > So this implementation would not rely in any way on Hive, only on > > Thrift? > > > > > > > > Best regards, > > > > > > > > Martijn Visser > > > > https://twitter.com/MartijnVisser82 > > > > https://github.com/MartijnVisser > > > > > > > > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > > > > > Hi, Jark and Martijn > > > > > > > > > > Thanks for your feedback. > > > > > > > > > > > Kyuubi provides three ways to configure Hive metastore [1]. Could > > we > > > > > provide similar abilities? > > > > > > > > > > Yes. I have updated the FLIP about this and it takes some time to > > > figure > > > > > out how the jdbc driver works. I added the section about how to use > > the > > > > > hive JDBC to configure the session-level catalog. > > > > > > > > > > > I think we can improve the "HiveServer2 Compatibility" section. > > > > > > > > > > Yes. I have updated the FLIP and added more details about the > > > > > compatibility. > > > > > > > > > > > Prefer to first complete the discussion and vote on FLIP-91 then > > > > discuss > > > > > FLIP-223 > > > > > > > > > > Of course. We can wait until the discussion of the FLIP-91 > finishes. > > > > > > > > > > > Maintenance concerns about the hive > > > > > > > > > > Actually we will only rely on the API in the Hive, which only > > contains > > > > the > > > > > thrift file and the generated code[1]. I think it will not > influence > > us > > > > to > > > > > upgrade the java version. > > > > > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年4月26日周二 20:44写道： > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a > > > couple > > > > of > > > > > > questions/concerns: > > > > > > > > > > > > 1. What is the relationship between this FLIP and FLIP-91? My > > > > assumption > > > > > > would be that this FLIP (and therefore the HiveServer2) > > > implementation > > > > > > would need to be integrated in the REST Gateway, is that correct? > > If > > > > so, > > > > > I > > > > > > would prefer to first complete the discussion and vote on > FLIP-91, > > > else > > > > > > we'll have two moving FLIPs who have a direct relationship with > > each > > > > > other. > > > > > > > > > > > > 2. While I understand that Hive is important (in the Chinese > > > ecosystem, > > > > > not > > > > > > so much in Europe and the US), I still have maintenance concerns > on > > > > this > > > > > > topic. We know that the current Hive integration isn't exactly > > ideal > > > > and > > > > > > requires a lot of work to get in better shape. At the same time, > > Hive > > > > > still > > > > > > doesn't support Java 11 while we need (and should, given the > > premier > > > > > > support has ended already) to move away from Java 8. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn Visser > > > > > > https://twitter.com/MartijnVisser82 > > > > > > https://github.com/MartijnVisser > > > > > > > > > > > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> wrote: > > > > > > > > > > > > > Thank Shengkai for driving this effort, > > > > > > > I think this is an essential addition to Flink Batch. > > > > > > > > > > > > > > I have some small suggestions: > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore [1]. > > > Could > > > > we > > > > > > > provide similar abilities? > > > > > > > Especially with the JDBC Connection URL, users can visit > > different > > > > Hive > > > > > > > metastore server instances. > > > > > > > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility" > > section. > > > > > > > We need to figure out two compatibility matrices. One is SQL > > > Gateway > > > > > with > > > > > > > different versions of Hive metastore, > > > > > > > and the other is different versions of Hive client (e.g., Hive > > > JDBC) > > > > > with > > > > > > > SQL Gateway. We need to clarify > > > > > > > what metastore and client versions we support and how users > > > configure > > > > > the > > > > > > > versions. > > > > > > > > > > > > > > Best, > > > > > > > Jark > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > > > > > > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > Hi, Jiang. > > > > > > > > > > > > > > > > Thanks for your feedback! > > > > > > > > > > > > > > > > > Integrating the Hive ecosystem should not require changing > > the > > > > > > service > > > > > > > > interface > > > > > > > > > > > > > > > > I move the API change to the FLIP-91. But I think it's > possible > > > we > > > > > add > > > > > > > more > > > > > > > > interfaces to intergrate the new endpoints in the future > > because > > > > > every > > > > > > > > endpoints's functionality is different. For example, the REST > > > > > endpoint > > > > > > > > doen't support to fetch operation-level logs but the > > hiveserver2 > > > > > > endpoint > > > > > > > > supports. In this case, we need to modify the shared > > > GatewayService > > > > > to > > > > > > > > support the functionality exposed by the new endpint. > > > > > > > > > > > > > > > > > How to support different Hive versions? > > > > > > > > > > > > > > > > Do you means to support the different HiveServer2 version? > The > > > > > > > HiveServer2 > > > > > > > > uses the version to guarantee the compatibility. During the > > > > > > openSession, > > > > > > > > the client and server will determine the protocol > > > > > > version(minimun(client > > > > > > > > version, hiveendpoint version)). After that the client and > the > > > > server > > > > > > > uses > > > > > > > > the determined version to communicate. In the HiveServer2 > > > endpoint, > > > > > it > > > > > > > > determines how the endpoint deserialize the results and the > > > result > > > > > > > schema. > > > > > > > > I add a section about HiveServer2 compatiblity. > > > > > > > > > > > > > > > > > Could you please fully provide its definition including > input > > > > > > > parameters > > > > > > > > and the corresponding return value schema? > > > > > > > > > > > > > > > > Because we implements the interface exposed by the Hive. So I > > add > > > > the > > > > > > > file > > > > > > > > link to the HiveServer2 interfaces[1], which contains all > input > > > > > > > parameters > > > > > > > > and the results. Considering the file doesn't contain the > > output > > > > for > > > > > > the > > > > > > > > Operation, I add the output schema for all the supported > > > Operation > > > > in > > > > > > the > > > > > > > > FLIP, which is not covered in the link. Hope these can > address > > > your > > > > > > > > question. > > > > > > > > > > > > > > > > Best, > > > > > > > > Shengkai > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 > > 16:43写道： > > > > > > > > > > > > > > > > > Hi Shengkai. > > > > > > > > > > > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint > > > support. > > > > > For > > > > > > > the > > > > > > > > > "GatewayService API Change", I don't think the motivation > for > > > > > > > supporting > > > > > > > > > HiveServer2 endpoint need to change the GatewayService API, > > in > > > > > other > > > > > > > > words, > > > > > > > > > integrating the Hive ecosystem should not require changing > > the > > > > > > service > > > > > > > > > interface. If you confirm to change GatewayService > interface, > > > > IMO, > > > > > > the > > > > > > > > > proposal could be discussed in FLIP-91 because the public > > > > > interfaces > > > > > > > are > > > > > > > > > defined in FLIP-91. > > > > > > > > > > > > > > > > > > In addtion, how to support different Hive versions and how > to > > > > > > guarantee > > > > > > > > > compatibility is not mentioned in the design. What's the > > > behavior > > > > > of > > > > > > > the > > > > > > > > > compatibility? > > > > > > > > > > > > > > > > > > Finally, for the public interfaces, could you please fully > > > > provide > > > > > > its > > > > > > > > > definition including input parameters and the corresponding > > > > return > > > > > > > value > > > > > > > > > schema? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Nicholas Jiang > > > > > > > > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > > > > > > > Hi, Flink developers. > > > > > > > > > > > > > > > > > > > > I want to start a discussion about the FLIP-223: Support > > > > > > HiveServer2 > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift > > interface > > > > > > exposed > > > > > > > > by > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools based > > on > > > > the > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink > > SQL > > > > > > Gateway. > > > > > > > > > After > > > > > > > > > > the FLIP finishes, the users can have almost the same > > > > experience > > > > > in > > > > > > > the > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the > > > > > > > HiveServer2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
