Hi Shengkai, Thanks for clarifying.
Best regards, Martijn On Fri, 6 May 2022 at 08:40, Shengkai Fang <fskm...@gmail.com> wrote: > Hi Martijn. > > > So this implementation would not rely in any way on Hive, only on Thrift? > > Yes. The dependency is light. We also can just copy the iface file from > the Hive repo and maintain by ourselves. > > Best, > Shengkai > > Martijn Visser <martijnvis...@apache.org> 于2022年5月4日周三 21:44写道： > > > Hi Shengkai, > > > > > Actually we will only rely on the API in the Hive, which only contains > > the thrift file and the generated code > > > > So this implementation would not rely in any way on Hive, only on Thrift? > > > > Best regards, > > > > Martijn Visser > > https://twitter.com/MartijnVisser82 > > https://github.com/MartijnVisser > > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskm...@gmail.com> wrote: > > > > > Hi, Jark and Martijn > > > > > > Thanks for your feedback. > > > > > > > Kyuubi provides three ways to configure Hive metastore [1]. Could we > > > provide similar abilities? > > > > > > Yes. I have updated the FLIP about this and it takes some time to > figure > > > out how the jdbc driver works. I added the section about how to use the > > > hive JDBC to configure the session-level catalog. > > > > > > > I think we can improve the "HiveServer2 Compatibility" section. > > > > > > Yes. I have updated the FLIP and added more details about the > > > compatibility. > > > > > > > Prefer to first complete the discussion and vote on FLIP-91 then > > discuss > > > FLIP-223 > > > > > > Of course. We can wait until the discussion of the FLIP-91 finishes. > > > > > > > Maintenance concerns about the hive > > > > > > Actually we will only rely on the API in the Hive, which only contains > > the > > > thrift file and the generated code[1]. I think it will not influence us > > to > > > upgrade the java version. > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc > > > > > > Best, > > > Shengkai > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年4月26日周二 20:44写道： > > > > > > > Hi all, > > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a > couple > > of > > > > questions/concerns: > > > > > > > > 1. What is the relationship between this FLIP and FLIP-91? My > > assumption > > > > would be that this FLIP (and therefore the HiveServer2) > implementation > > > > would need to be integrated in the REST Gateway, is that correct? If > > so, > > > I > > > > would prefer to first complete the discussion and vote on FLIP-91, > else > > > > we'll have two moving FLIPs who have a direct relationship with each > > > other. > > > > > > > > 2. While I understand that Hive is important (in the Chinese > ecosystem, > > > not > > > > so much in Europe and the US), I still have maintenance concerns on > > this > > > > topic. We know that the current Hive integration isn't exactly ideal > > and > > > > requires a lot of work to get in better shape. At the same time, Hive > > > still > > > > doesn't support Java 11 while we need (and should, given the premier > > > > support has ended already) to move away from Java 8. > > > > > > > > Best regards, > > > > > > > > Martijn Visser > > > > https://twitter.com/MartijnVisser82 > > > > https://github.com/MartijnVisser > > > > > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <imj...@gmail.com> wrote: > > > > > > > > > Thank Shengkai for driving this effort, > > > > > I think this is an essential addition to Flink Batch. > > > > > > > > > > I have some small suggestions: > > > > > 1) Kyuubi provides three ways to configure Hive metastore [1]. > Could > > we > > > > > provide similar abilities? > > > > > Especially with the JDBC Connection URL, users can visit different > > Hive > > > > > metastore server instances. > > > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility" section. > > > > > We need to figure out two compatibility matrices. One is SQL > Gateway > > > with > > > > > different versions of Hive metastore, > > > > > and the other is different versions of Hive client (e.g., Hive > JDBC) > > > with > > > > > SQL Gateway. We need to clarify > > > > > what metastore and client versions we support and how users > configure > > > the > > > > > versions. > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations > > > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskm...@gmail.com> > > wrote: > > > > > > > > > > > Hi, Jiang. > > > > > > > > > > > > Thanks for your feedback! > > > > > > > > > > > > > Integrating the Hive ecosystem should not require changing the > > > > service > > > > > > interface > > > > > > > > > > > > I move the API change to the FLIP-91. But I think it's possible > we > > > add > > > > > more > > > > > > interfaces to intergrate the new endpoints in the future because > > > every > > > > > > endpoints's functionality is different. For example, the REST > > > endpoint > > > > > > doen't support to fetch operation-level logs but the hiveserver2 > > > > endpoint > > > > > > supports. In this case, we need to modify the shared > GatewayService > > > to > > > > > > support the functionality exposed by the new endpint. > > > > > > > > > > > > > How to support different Hive versions? > > > > > > > > > > > > Do you means to support the different HiveServer2 version? The > > > > > HiveServer2 > > > > > > uses the version to guarantee the compatibility. During the > > > > openSession, > > > > > > the client and server will determine the protocol > > > > version(minimun(client > > > > > > version, hiveendpoint version)). After that the client and the > > server > > > > > uses > > > > > > the determined version to communicate. In the HiveServer2 > endpoint, > > > it > > > > > > determines how the endpoint deserialize the results and the > result > > > > > schema. > > > > > > I add a section about HiveServer2 compatiblity. > > > > > > > > > > > > > Could you please fully provide its definition including input > > > > > parameters > > > > > > and the corresponding return value schema? > > > > > > > > > > > > Because we implements the interface exposed by the Hive. So I add > > the > > > > > file > > > > > > link to the HiveServer2 interfaces[1], which contains all input > > > > > parameters > > > > > > and the results. Considering the file doesn't contain the output > > for > > > > the > > > > > > Operation, I add the output schema for all the supported > Operation > > in > > > > the > > > > > > FLIP, which is not covered in the link. Hope these can address > your > > > > > > question. > > > > > > > > > > > > Best, > > > > > > Shengkai > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nicholas Jiang <nicholasji...@apache.org> 于2022年4月22日周五 16:43写道： > > > > > > > > > > > > > Hi Shengkai. > > > > > > > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint > support. > > > For > > > > > the > > > > > > > "GatewayService API Change", I don't think the motivation for > > > > > supporting > > > > > > > HiveServer2 endpoint need to change the GatewayService API, in > > > other > > > > > > words, > > > > > > > integrating the Hive ecosystem should not require changing the > > > > service > > > > > > > interface. If you confirm to change GatewayService interface, > > IMO, > > > > the > > > > > > > proposal could be discussed in FLIP-91 because the public > > > interfaces > > > > > are > > > > > > > defined in FLIP-91. > > > > > > > > > > > > > > In addtion, how to support different Hive versions and how to > > > > guarantee > > > > > > > compatibility is not mentioned in the design. What's the > behavior > > > of > > > > > the > > > > > > > compatibility? > > > > > > > > > > > > > > Finally, for the public interfaces, could you please fully > > provide > > > > its > > > > > > > definition including input parameters and the corresponding > > return > > > > > value > > > > > > > schema? > > > > > > > > > > > > > > Thanks, > > > > > > > Nicholas Jiang > > > > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote: > > > > > > > > Hi, Flink developers. > > > > > > > > > > > > > > > > I want to start a discussion about the FLIP-223: Support > > > > HiveServer2 > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift interface > > > > exposed > > > > > > by > > > > > > > > the HiveServer2, and users' BI, CLI and other tools based on > > the > > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL > > > > Gateway. > > > > > > > After > > > > > > > > the FLIP finishes, the users can have almost the same > > experience > > > in > > > > > the > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the > > > > > HiveServer2. > > > > > > > > > > > > > > > > > > > > > > > > I am glad that you can give some feedback about FLIP-223. > > > > > > > > > > > > > > > > Best, > > > > > > > > Shengkai > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
