Hi Simhadri, Thanks for presenting your understanding of this feature. Your insight in this area is always helpful.
> This sounds like a really useful feature with a lot of potential I couldn't agree more. I would say Apache Hive has always proposed and shaped the cutting-edge landscape of Open Data Warehouses. To enforce FGAC or complex storage requirements for external clients with near-zero overhead, llap-ext-client is a promising option. Actually, the Apache Iceberg community hit the same problem and is trying to solve it with an HWC-like solution[1] and Credential Vending, if I understand correctly. I'd say we will have a chance to propose our solution again in the future, which would help not only Hive users but also more users in the Lakehouse era, even if we decommission the current Gen-1 external connector. Regards, Okumin - [1] https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L625-L673 On Sat, Aug 22, 2026 at 11:21 PM Simhadri G <[email protected]> wrote: > > Hi, > > Just my thoughts: llap-ext-client allows an external process (like Spark) to > submit executable tasks to an LLAP daemon and consume its output. This sounds > like a really useful feature with a lot of potential, and I would be sad to > see it go. > > It is also true that no one is using llap-ext-client today. The only place I > have seen it being used was for the Arrow integration with Spark-LLAP [1], > which later became HWC. Even there, they stopped using llap-ext-client a few > years ago. Later, in April this year, we removed Arrow from Hive [2]. So this > is definitely a maintenance burden as well. > > So I would cast my vote as 0 for this (non-binding). > > Thanks, > Simhadri G > > > [1] > https://github.com/hortonworks-spark/spark-llap/blob/26d164e62b45cfa1420d5d43cdef13d1d29bb877/assembly/pom.xml#L44 > [2] https://lists.apache.org/[email protected]:2025-4:arrow > > On Sat, Aug 22, 2026, 11:07 AM Shohei Okumiya <[email protected]> wrote: >> >> Hi, >> >> Thanks for your feedback! >> >> I am testing what we can drop for reference[1]. In short, I currently >> expect the following to be in scope. >> >> - LlapStorageHandler, LlapInputFormat, LlapOutputFormat, LlapInputSplit, etc. >> - LlapDump: a test tool for this feature >> - get_splits, get_llap_splits UDFs, which are the entry points of this >> feature >> - Port 15003, 30004, and 30005 from LLAP daemons >> - Related fields and messages from Protobuf IDL >> >> I plan to break down HIVE-28932 into a few steps so that we can >> confidently remove the feature and review each PR. More modules and >> files are involved than I expected. #6714 is already beyond my >> cognitive abilities. >> >> I will start working on the first PR if we don't see any objections. >> I'd appreciate it if some of us familiar with this feature could share >> their opinions. >> >> Best, >> Okumin >> >> - [1] https://github.com/apache/hive/pull/6714/changes >> >> On Thu, Aug 20, 2026 at 7:12 PM Attila Turoczy <[email protected]> wrote: >> > >> > +1 purge >> > >> > On 2026. Aug 20., Thu at 11:46, Stamatis Zampetakis <[email protected]> >> > wrote: >> >> >> >> Thanks for kicking off the discussion Okumin! I am always in favor of >> >> dropping redundant/unused code so +1 for the removal. >> >> >> >> I am not very familiar with the module but since HIVE-28904 was released >> >> as part of 4.1.0 and nobody complained I guess it's safe to continue the >> >> cleanup and remove the rest. >> >> >> >> Best, >> >> Stamatis >> >> >> >> On Thu, Aug 20, 2026 at 10:05 AM Shohei Okumiya <[email protected]> wrote: >> >>> >> >>> Hi Hive users and developers, >> >>> >> >>> I propose removing the llap-ext-client module[1] and the related >> >>> parameters and code. >> >>> >> >>> If I understand correctly, the module includes some classes to >> >>> integrate an external client, such as Apache Spark, with Hive LLAP >> >>> daemons. However, HIVE-28904 removed the direct interface, and a test >> >>> tool[2] remains as the only real use case. >> >>> >> >>> Therefore, I suspect we can safely remove the module and >> >>> `hive.llap.external.*` in HiveConf, along with all configurations and >> >>> code used only by the feature. >> >>> >> >>> Since llap-ext-client supports unusual workloads, I believe >> >>> decommissioning it would meaningfully reduce our maintenance burden. >> >>> >> >>> Please share your thoughts about this proposal. >> >>> >> >>> Best, >> >>> Okumin >> >>> >> >>> - [1] https://github.com/apache/hive/tree/master/llap-ext-client >> >>> - [2] >> >>> https://github.com/apache/hive/blob/master/llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapDump.java
