Hello Sir. I have applied for a wiki account.
Account name: lisoda Email: lis...@yeah.net Please help me to activate the access of HIVE-wiki. Thank you. lisoda 在 2024-10-24 21:25:28，"Ayush Saxena" <ayush...@gmail.com> 写道： You can submit a request for account here: https://selfserve.apache.org/confluence-account.html Please add some relevant information while submitting the request so that I can identify it is you & approve. Once approved let us know, anyone amongst us will grant you write access to the hive space for that account. -Ayush On Thu, 24 Oct 2024 at 12:16, lisoda <lis...@yeah.net> wrote: Hello Ayush. It looks like I don't have access to the wiki(I tried to log in using jira's account), and I can't find an entry point on the page to request an account.Can you tell me how to apply for an account? Also, if I am unable to apply for an account, how do I provide the relevant information? Best Lisoda 在 2024-10-22 13:01:41，"Ayush Saxena" <ayush...@gmail.com> 写道： Sorry for coming back late, I don’t think there should be any problem in this approach if things are working fine.. I think it doesn’t require any code changes, Do you plan to contribute the steps or details around the approach via wiki or so, If yes you can share the details of your wiki id if you already have one & I can give you the permissions for the Hive space. I started some initiative [1] for documentation for 4.0.x, though I couldn’t spend enough time on that, but a page under Installing Hive could be a good place to keep this or let me know if you or other folks following have any other idea or plans -Ayush [1] https://cwiki.apache.org/confluence/display/Hive/Apache+Hive+4.0.0 On 12 Oct 2024, at 8:57 PM, lisoda <lis...@yeah.net> wrote: Hello Sir. I agree with your comments related to hadoop2, and I don't actually intend to support it.We just need to support hadoop 3.x and we're good to go. On a low version of hadoop3, this is what we do: 1. Download the hadoop binaries separately(high version,example:3.3.6), and set hadoop_home in hive to be the directory where the higher version of hadoop is stored. 2. Package tez with all the dependencies and native lib (including the required ones for hadoop). 3. In tez-site.xml.Specify that tez will only use all the jar packages in its own lib folder, and not any hadoop related dependencies in the cluster. With the above steps, we are currently running hive4.0.1+tez0.10.4 on hdp 3.1.0(hadoop 3.1.1). They work fine. This is the solution we are currently using, do you see any problems with this solution? If there are no problems with this solution, can we extend it to all hive's users? Tks. LiSoDa. 在 2024-10-12 14:27:32，"Ayush Saxena" <ayush...@gmail.com> 写道： If you already have a solution in place, feel free to create a Jira & PR with it. However, third-party dependencies present significant challenges. Different versions of Hadoop bring their own set of third-party libraries, which can cause compatibility issues with the versions used by Hive. A prime example is Guava: while Hadoop upgraded Guava in versions post-3.1.x, Hive couldn’t follow suit. Hadoop eventually shaded Guava in 3.3.x, which is why we aligned with that version. One potential improvement could be to switch to using hadoop-client-api, hadoop-client-runtime, and hadoop-client-minicluster instead of directly specifying the Hadoop dependencies. These artifacts shade most of the third-party libraries, which may help minimize conflicts. Spark, for example, already uses them [1]. As for releasing separate binaries for different Hadoop versions, I don't think that’s feasible. However, users are free to build their own versions from the source tarball we provide, using -Dhadoop.version=X. The actual release is the source code; the binaries are just convenience binaries That said, I don’t believe supporting the 2.x Hadoop line would be easy, or even possible, at this point, but we can attempt for 3.x maybe -Ayush [1] https://github.com/apache/spark/blob/6734d4883e76b82249df5c151d42bc83173f4122/pom.xml#L1401-L1424 On Wed, 9 Oct 2024 at 17:32, lisoda <lis...@yeah.net> wrote: HI TEAM. I would like to discuss with everyone the issue of running Hive4 in Hadoop environments below version 3.3.6. Currently, a large number of Hive users are still using low-version environments such as Hadoop 2.6/2.7/3.1.1. To be honest, upgrading Hadoop is a challenging task. We cannot force users to upgrade their Hadoop cluster versions just to use Hive4. In order to encourage these potential users to adopt and use Hive4, we need to provide a general solution that allows Hive4 to run on low-version Hadoop (at least we need to address the compatibility issues with Hadoop version 3.1.0). The general plan is as follows: In both the Hive and Tez projects, in addition to providing the existing tar packages, we should also provide tar packages that include high-version Hadoop dependencies. By defining configuration files, users can avoid using any jar package dependencies from the Hadoop cluster. In this way, users can initiate Tez tasks on low-version Hadoop clusters using only the built-in Hadoop dependencies. This is how Spark does it, which is also the main reason why users are more likely to adopt Spark as a SQL engine. Spark not only provides tar packages without Hadoop dependencies but also provides tar packages with built-in Hadoop 3 and Hadoop 2. Users can upgrade to a new version of Spark without upgrading the Hadoop version. We have implemented such a plan in our production environment, and we have successfully run Hive4.0.0 and Hive4.0.1 in the HDP 3.1.0 environment. They are currently working well. Based on our successful experience, I believe it is necessary for us to provide tar packages with all Hadoop dependencies built in. At the very least, we should document that users can successfully run Hive4 on low-version Hadoop in this way. However, my idea may not be mature enough, so I would like to know what others think. It would be great if someone could participate in this topic and discuss it. TKS. LISODA.
