Regarding JDK-11, we didn't actively chased that, so its status remains uncertain. With lower compile-time support like JDK-8, we can certainly ensure runtime support for both JDK-11 and JDK-17, which I believe should work for Hive 4.0.1. As far as I know, this approach works downstream for us atleast.
However, maintaining compile-time compatibility across JDK-17, JDK-11, and JDK-8 simultaneously may not be feasible. JDK-17 introduces significant changes, and I don’t believe it’s naive—or even possible—to compile Hive using both JDK-8 and JDK-17 & expect things to work properly. The initial plan appeared to be supporting JDK-8 for Hive 4.0.x release line, with potential tweaks if required to enable runtime support for JDK-17. Starting with Hive 4.1.x, we could shift to JDK-17 for compile-time support, and later evaluate the feasibility of supporting JDK-21 or newer versions. Once we see a state when nobody want JDK-8 compile time support & everyone has moved to JDK-17 or say using 4.1.x release line, We can then drop having bug fix releases for 4.0.x -Ayush On Mon, 9 Dec 2024 at 15:11, Butao Zhang <butaozha...@163.com> wrote: > IMO, I think there two challenges if we want to maintain JDK8 & JDK11 & > JDK17 at the same time. > > - Different jdk syntactic sugar between multiple jdks, and advanced > feats in jdk17&jdk21, such as jdk vector api. If we want the multiple jdks, > we can not use the advanced feats in high jdk in the future. Such as the > discussion in https://issues.apache.org/jira/browse/HIVE-28614 > - We need to run the jenkins CI process multiple times to run the > multiple jdk env. I'm not sure we have enough jenkins resources to do this. > However, i want to say if we can use github actions to run the whole CI? I > have seen Apache Spark have done the good job which using the forked > repository github actions to run the CI. FRY > https://lists.apache.org/thread/dmv1h2snqc7qtdjlzj4k8qxookclp720 > > > Thanks, > Butao Zhang > ---- Replied Message ---- > From Shohei Okumiya<oku...@apache.org> <undefined> > Date 12/9/2024 17:06 > To <dev@hive.apache.org> <dev@hive.apache.org> > Subject Re: Move to JDK-11 > Hi, > > I appreciate everyone's work on the JDK upgrade. It would be ideal if we > could migrate JDK like this. That will give us an > incremental migration path. > > 1. (now) Hive 4.0 compiled by JDK 8, and run by JDK 8 > 2. Hive 4.x compiled by JDK 8, and run by JDK 17 > 3. Hive 4.x compiled by JDK 17, and run by JDK 17 > > I guess it is technically possible that Hive supports JDK 8 and 17 at > runtime. Apache Spark supports JDK 8, 11, and 17. What is the most > challenging point? > > Regards, > Okumin > > On Thu, Aug 29, 2024 at 10:48 PM Butao Zhang <butaozha...@163.com> wrote: > >> Hi Akshat, >> >> Thanks for your reply! I agree that supporting both JDK 8 and JDK 17 is >> not easy. I think we can safely to move to JDK 17 if we can ensure *hive >> application* can access to HiveServer2/Metastore without forcing to use >> the JDK 17. >> For example, if a hive application is developed by JDK 8, it can import >> the *hive-exec.jar & hive-jdbc.jar* which is compiled by JDK 17 and can >> access to hs2/hms successfully. If this case can be supported, I think we >> can move to JDK 17 as soon as possible. >> >> BTW, If Apache Hive , Apache Tez and Apache Hadoop are all compiled with >> JDK17 or higer version, i think the hive job will run more faster. Users >> will enjoy significant performance improvements due to JDK version >> upgrades. >> I'm looking forward to the performance improvement. :) >> >> Thanks, >> Butao Zhang >> ---- Replied Message ---- >> From Akshat m<akshatats...@gmail.com> <undefined> >> Date 8/29/2024 18:44 >> To <dev@hive.apache.org> <dev@hive.apache.org> >> Subject Re: Move to JDK-11 >> Hi Butao, >> >> It won’t be feasible to support both JDK 8 and JDK 17 in the same Hive >> version due to compatibility issues and behavior changes, as detailed in >> the PR. The internal changes in JDK implementations have made this >> challenging. Additionally, many other libraries maintain separate versions >> for JDK 17+ and JDK 8 support. >> >> To align with this approach, I suggest we continue with Hive 4 supporting >> JDK 8. Once the JDK 17 PR is merged, we could plan a separate release that >> specifically targets JDK 17+ support. This would ensure that Hive stays >> up-to-date while maintaining stability across different environments. >> >> Would love to hear your thoughts or any additional suggestions the >> community might have! >> >> Regards >> >> Akshat >> >> >> >> On Wed, Aug 28, 2024 at 8:00 AM Butao Zhang <butaozha...@163.com> wrote: >> >>> Good job! Thanks everyone who put effort into the JDK upgrading. That's >>> a big good news for Apache Hive community! >>> Just a question: Given that many people are still using the lower jdk8 >>> version. Do we consider to keeping compatibility with lower jdk8 version? >>> >>> >>> Thanks, >>> Butao Zhang >>> ---- Replied Message ---- >>> From Akshat m<akshatats...@gmail.com> <undefined> >>> Date 8/27/2024 23:12 >>> To dev<dev@hive.apache.org> <dev@hive.apache.org> >>> Cc kokila narayanan<kokilanarayana...@gmail.com>, >>> <kokilanarayana...@gmail.com>Raghav >>> Aggarwal<raghavaggarwal03...@gmail.com>, >>> <raghavaggarwal03...@gmail.com>Ayush Saxena<ayush...@gmail.com>, >>> <ayush...@gmail.com>tanishq chugh<tanishqchugh4...@gmail.com>, >>> <tanishqchugh4...@gmail.com><aturo...@cloudera.com> >>> <aturo...@cloudera.com> >>> Subject Re: Move to JDK-11 >>> >>> Hi everyone, >>> >>> I’m excited to share that after months of dedicated work, we’ve finally >>> achieved a passing PR[1] for running *Apache Hive on JDK 17*! >>> >>> This was a challenging task, requiring to address various compatibility >>> issues and ensure our comprehensive test suite passed without a hitch. >>> >>> Thanks Kokila and Tanishq for joining me in the initiative and Raghav >>> for contributing fix for the errorProne issue, and thanks to Attila, Ayush >>> and other Hive PMC members for their guidance in the initiative >>> >>> It’s been a real team effort, and I’m grateful for the collaboration and >>> support from everyone involved. >>> >>> I’d appreciate it if you could take some time to review the PR. >>> Community feedback is really important to make this happen. >>> >>> The PR #5404 addresses the integration of Apache Hive with JDK 17, >>> introducing a range of changes aimed at ensuring compatibility and >>> improving the codebase. Here’s a detailed breakdown: >>> >>> 1. Core Code Adjustments: >>> >>> - Compatibility Fixes: Various parts of the Hive codebase have been >>> modified to ensure they work seamlessly with JDK 17. This includes >>> addressing language or API changes that JDK 17 introduces. >>> >>> - Reflection and Modules: Adjustments related to how Hive interacts >>> with Java's reflection API, which has stricter module system rules in JDK >>> 17. >>> >>> 2. Dependency Management: >>> >>> - Version Upgrades: Several third-party dependencies have been >>> updated to versions that are compatible with JDK 17. This is critical >>> because older versions of some libraries may not work properly or might use >>> deprecated features removed in JDK 17. >>> >>> - Data nucleus upgrade: To ensure compatibility with Java 17, we >>> upgraded DataNucleus to version 6.x. This upgrade, however, introduced an >>> issue with Hive schema creation. The issue stems from a commit in >>> DataNucleus[2] that alters the handling of primary key column names by >>> enclosing them in double quotes. This change caused conflicts during Hive >>> schema generation. As an interim solution, we implemented a patched >>> DataNucleus-rdbms module to address the issue in Hive. The necessary fix is >>> now included in the DataNucleus repository[3]. Once DataNucleus 6.0.8 is >>> officially released, we will upgrade to this version and remove the >>> temporary patched module. >>> >>> - Dependency Cleanup: Unused or outdated dependencies were removed, >>> reducing potential conflicts and simplifying the build process. >>> >>> 3. Testing Enhancements: >>> >>> - Test Suite Updates: The PR includes updates to Hive’s test suite to >>> ensure all tests pass under JDK 17. This likely involved modifying tests >>> that needed adjustments to align with JDK 17’s behaviour. >>> >>> - New Tests: New tests may have been added to cover areas where JDK 17 >>> introduced new features or changes that could impact Hive. >>> >>> 4. Compatibility Checks: >>> >>> - JDK Version Verification: A new compatibility check has been >>> introduced to verify the JDK version at runtime. This ensures that Hive is >>> running on an appropriate JDK version, preventing unexpected errors or >>> behaviour due to version mismatches. >>> >>> 5. Build Updates: >>> >>> - Build System Tweaks: The PR likely includes changes to the Maven >>> build configuration to accommodate JDK 17, ensuring that the build process >>> is smooth and compatible. >>> >>> Overall, this PR is a comprehensive effort to bring Apache Hive up to >>> date with the latest Java platform, ensuring long-term maintainability and >>> performance improvements. >>> >>> As the next steps, we’ll be focusing on further sanity checks and >>> extensive testing to make sure everything functions smoothly. Community >>> involvement in this process will be crucial, so please do share any issues >>> or feedback as you test the changes. >>> >>> I’m looking forward to seeing the benefits this will bring to our >>> community and users. >>> >>> >>> [1] https://github.com/apache/hive/pull/5404 >>> >>> [2] >>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 >>> >>> [3] >>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 >>> >>> >>> Best regards, >>> >>> Akshat Mathur >>> >>> >>> On Tue, 11 Jul 2023 at 2:25 PM, Attila Turoczy < >>> aturo...@cloudera.com.invalid> wrote: >>> >>>> Returning to this topic, I kindly request those who would like to >>>> advocate >>>> for the continued support of JDK8 to please share their reasoning and >>>> insights with us. Your input and perspective are greatly appreciated! >>>> Thank you. >>>> >>>> -Attila >>>> >>>> On Fri, Jun 2, 2023 at 12:43 PM Attila Turoczy <aturo...@cloudera.com> >>>> wrote: >>>> >>>> > Hi All, >>>> > >>>> > I know my opinion might not be the most popular, but I advocate for >>>> using* >>>> > JDK 17*. Here's why: >>>> > >>>> > Let's consider a scenario where a customer wants to use the latest >>>> version >>>> > of Apache Hive. They would typically install it locally or on a small >>>> > cluster. In 2023, is it realistic to assume that this customer won't >>>> be >>>> > able to install JDK 17 on their cluster? Even in large enterprises, it >>>> > should be feasible to install an LTS JDK, especially considering the >>>> > widespread adoption of cloud computing. Sungwoo Park's measurements >>>> also >>>> > support this recommendation to go with JDK 17. It outperforms JDK 11 >>>> by 8% >>>> > in terms of runtime speed, and JDK 11 itself is 10+% faster than JDK >>>> 8. >>>> > This is a significant value proposition. Who would be the customer >>>> that >>>> > says, "I don't want faster query execution! I'd rather use JDK 8 and >>>> pay >>>> > more for cloud or data center resources instead of using JDK 17!" It >>>> > doesn't make sense to me. >>>> > >>>> > The tech industry has been evolving at an incredible pace, with >>>> > improvements in serialization, IPC mechanisms, and parallelized >>>> frameworks >>>> > since the release of JDK 8 ten years ago. We should leverage these >>>> > advancements! Couple years ago, we invested a lot to improve 1-2% of >>>> the >>>> > execution. We prayed for 3 gods, sacrificed 2 ships and traveled >>>> around the >>>> > world to make it happen. :-) Now, the JDK itself provides a >>>> substantial >>>> > amount of improvement. So, why would we resist progress just because >>>> there >>>> > are a few lazy or conservative admins who don't want to spend two >>>> minutes >>>> > installing a JDK? >>>> > >>>> > A platform needs to be modern and incorporate the latest technologies >>>> to >>>> > attract developers and users. I understand that some may prefer to >>>> stay >>>> > with JDK 8 as it seems like the safest position, but I believe in >>>> taking >>>> > bold bets to achieve big wins. Even if we decide to stick with JDK 8, >>>> I >>>> > would still be happy since we are moving forward and not dwelling on >>>> a JDK >>>> > that is a decade old. Personally, I think focusing on one thing that >>>> brings >>>> > more value to us and our users is the idealistic path forward. >>>> > >>>> > -Attila >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > On Thu, Jun 1, 2023 at 11:23 AM Stamatis Zampetakis < >>>> zabe...@gmail.com> >>>> > wrote: >>>> > >>>> >> Hey everyone, >>>> >> >>>> >> If we claim that Hive supports a certain JDK then we should compile >>>> and >>>> >> run >>>> >> tests with it. >>>> >> >>>> >> The more JDKs we can support the better for everyone but this comes >>>> at a >>>> >> cost (resources mostly). We should have a precommit run for every >>>> >> supported >>>> >> JDK (frequency to be determined once per day/week) that compiles and >>>> run >>>> >> all tests. >>>> >> >>>> >> From my perspective, I would be pretty happy if we could cover the >>>> two >>>> >> edge >>>> >> LTS releases at every point in time. >>>> >> >>>> >> Then we have to decide also which JDK shall we use for the pull >>>> requests >>>> >> and local dev environment. I think it makes sense to use the latest. >>>> >> People >>>> >> like working on modern stuff and also it makes sense that newer >>>> releases >>>> >> will also use newer versions. It would be pretty awkward if someone >>>> wants >>>> >> to use the latest Hive version and it turns out that it can only run >>>> on >>>> >> JDK8. >>>> >> >>>> >> Best, >>>> >> Stamatis >>>> >> >>>> >> On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park <glap...@gmail.com> wrote: >>>> >> >>>> >> > Hi, everyone. >>>> >> > >>>> >> > I have not tested the master branch with Java 11/17 yet, but I >>>> would >>>> >> like >>>> >> > to share my experience with testing a fork of branch-3.1 with Java >>>> 11/17 >>>> >> > (as part of developing Hive-MR3), in case that it can be useful >>>> for the >>>> >> > discussion. I merged the patches listed in [1] HIVE-22415 and >>>> updated >>>> >> the >>>> >> > Maven configuration for Java 11. >>>> >> > >>>> >> > 1. Building Hive was fine and I was able to run it with Java 11 as >>>> well >>>> >> as >>>> >> > Java 17. So, it seems that the work reported in [1] is indeed >>>> complete >>>> >> for >>>> >> > upgrading to Java 11 (and Java 17) and getting Hive to work. >>>> >> > >>>> >> > 2. However, there was a problem with running tests, so this can be >>>> >> > additional work for upgrading to Java 11. >>>> >> > >>>> >> > 3. For performance, Java 17 gives about 8 percent of (free) >>>> performance >>>> >> > improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 >>>> seconds, >>>> >> > whereas Java 17 takes 7415 seconds. Considering the maturity of >>>> Hive, I >>>> >> > think this is not a small improvement because almost every query >>>> gets >>>> >> some >>>> >> > speedup. >>>> >> > >>>> >> > Thanks, >>>> >> > >>>> >> > --- Sungwoo >>>> >> > >>>> >> > [1] https://issues.apache.org/jira/browse/HIVE-22415 >>>> >> > >>>> >> > >>>> >> > On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala >>>> >> > <saihema...@cloudera.com.invalid> wrote: >>>> >> > >>>> >> > > Hi All, >>>> >> > > >>>> >> > > I would strongly advocate keeping support for JDK8. >>>> >> > > Between JDK11 and JDK17, Depending on the amount of effort on the >>>> >> upgrade >>>> >> > > I'm inclined towards JDK17 (JDK21 LTS will be released in Sep >>>> 2023). >>>> >> > > >>>> >> > > Thanks, >>>> >> > > Sai. >>>> >> > > >>>> >> > > On Wed, May 31, 2023 at 5:39 AM László Bodor < >>>> >> bodorlaszlo0...@gmail.com> >>>> >> > > wrote: >>>> >> > > >>>> >> > > > *Hi!* >>>> >> > > > >>>> >> > > > >>>> >> > > > *Should we support both JDK-11 & JDK-8?* >>>> >> > > > IMO absolutely yes, let's not break up with JDK-8: according >>>> to its >>>> >> > > > lifecycle, it's going to stay with us for a long time. >>>> >> > > > >>>> >> > > > I believe >>>> >> > > > a) we should be able to compile on JDK8, JDK11, and JDK17 >>>> (github >>>> >> > actions >>>> >> > > > can cover this conveniently in precommit time, like tez >>>> >> > > > < >>>> >> >>>> https://github.com/apache/tez/blob/master/.github/workflows/build.yml >>>> >> > >) >>>> >> > > > b) the release artifacts should be compatible with JDK8 as >>>> long as >>>> >> it >>>> >> > is >>>> >> > > > with us. >>>> >> > > > >>>> >> > > > Regards, >>>> >> > > > Laszlo Bodor >>>> >> > > > >>>> >> > > > >>>> >> > > > Butao Zhang <butaozha...@163.com> ezt írta (időpont: 2023. >>>> máj. >>>> >> 31., >>>> >> > > Sze, >>>> >> > > > 14:33): >>>> >> > > > >>>> >> > > > > Thanks Ayush for driving this! Good to know that Hive is >>>> getting >>>> >> > ready >>>> >> > > > for >>>> >> > > > > newer JDK. >>>> >> > > > > From my opinon, if we have more community energy to put into >>>> it, >>>> >> we >>>> >> > can >>>> >> > > > > support both JDK-11 and JDK-17 like Spark[1]. If we have to >>>> make >>>> >> a >>>> >> > > > choice >>>> >> > > > > between a JDK-11 and JDK-17, i would like to choose the >>>> relatively >>>> >> > new >>>> >> > > > > version JDK-17, meanwhile, we should maintain compatibility >>>> with >>>> >> > jdk8, >>>> >> > > as >>>> >> > > > > JDK-8 is still widely used in most big data platforms. >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > Thanks, >>>> >> > > > > Butao Zhang >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > [1]https://issues.apache.org/jira/browse/SPARK-33772 >>>> >> > > > > ---- Replied Message ---- >>>> >> > > > > | From | Ayush Saxena<ayush...@gmail.com> | >>>> >> > > > > | Date | 5/31/2023 18:39 | >>>> >> > > > > | To | dev<dev@hive.apache.org> | >>>> >> > > > > | Subject | Move to JDK-11 | >>>> >> > > > > Hi Everyone, >>>> >> > > > > Want to pull in the attention of folks towards moving to >>>> JDK-11 >>>> >> > compile >>>> >> > > > > time support in Hive. There was a ticket in the past [1] >>>> which >>>> >> talks >>>> >> > > > about >>>> >> > > > > it and If I could decode it right, it was blocked because the >>>> >> Hadoop >>>> >> > > > > version used by Hive didn't had JDK-11 runtime support, But >>>> with >>>> >> [2] >>>> >> > in >>>> >> > > > we >>>> >> > > > > have upgraded the Hadoop version, so that problem is sorted >>>> out. I >>>> >> > > > couldn't >>>> >> > > > > even see any unresolved tickets in the blocked state either. >>>> >> > > > > >>>> >> > > > > I quickly tried* a mvn clean install -DskipTests -Piceberg >>>> >> -Pitests >>>> >> > > > > -Dmaven.javadoc.skip=true >>>> >> > > > > >>>> >> > > > > And no surprises it failed with some weird exceptions >>>> towards the >>>> >> > end. >>>> >> > > > But >>>> >> > > > > I think that should be solvable. >>>> >> > > > > >>>> >> > > > > So, Questions? >>>> >> > > > > >>>> >> > > > > - What do folks think about this? Should we put in some >>>> effort >>>> >> > towards >>>> >> > > > > JDK-11 >>>> >> > > > > - Should we support both JDK-11 & JDK-8? >>>> >> > > > > - Ditch JDK-11 and directly shoot for JDK-17? >>>> >> > > > > >>>> >> > > > > Let me know your thoughts, In case anyone has some >>>> experience in >>>> >> this >>>> >> > > > area >>>> >> > > > > and have tried something in the context, feel free to share >>>> or >>>> >> may be >>>> >> > > if >>>> >> > > > > someone has any potential action plan or so >>>> >> > > > > >>>> >> > > > > -Ayush >>>> >> > > > > >>>> >> > > > > [1] https://issues.apache.org/jira/browse/HIVE-22415 >>>> >> > > > > [2] https://issues.apache.org/jira/browse/HIVE-24484 >>>> >> > > > > >>>> >> > > > > * changed the maven.compiler.source & maven.compiler.target >>>> to 11 >>>> >> > > > > >>>> >> > > > >>>> >> > > >>>> >> > >>>> >> >>>> > >>>> >>>
