Re: How to migrate transaction tables across clusters（Hive-3.1.0）

Hi, Please check this thread: https://lists.apache.org/thread/0dg7k57p1jsdsv063jdv3tw29nz57roj. Note, MSCK REPAIR support for transactional tables was added in Hive 4. For additional details, refer to HIVE-23671. Regards, Denys

Re: Re: Migrating Hive 3 to Hive 4

> > By the way, I have one other question, how to migrate transaction table to > normal orc table(with no acid but stored as orc format) > > Thanks for your help. > Regards, > Zhao Wenbing > > -- > 243776...@qq.com > > > *From:* Denys Kuzmen

Re: Re: Migrating Hive 3 to Hive 4

Hive 3 cannot read ACID tables created by Hive 4, as Hive 4 is forward compatible only, not backward compatible.

Re: Migrating Hive 3 to Hive 4

Hive 3 cannot read ACID tables created by Hive 4, as Hive 4 is forward compatible only, not backward compatible.

Re: Migrating Hive 3 to Hive 4

Hi Ratnesh, For 1, I think you might need to upgrade Metastore database schema. Here are some results from my experiments. In my experiments, Hive 4 merges HIVE-26537 (Deprecate older APIs in the HMS thrift interface), and the results might be different without merging HIVE-26537. Exp #1. Can H

Re: Question: Hive String Concatenation with NULLs - Behavior and Rationale

Hi Sadegh, If I recall well the behavior of the string concatenation is defined by the SQL standard (ISO/IEC 9075). Section 6.29: General rules 2.b.i: If at least one of S1 and S2 is the null value, then the result of the is the null value. Hive as well as many other DBMS systems strive to rema

Re: Migrating Hive 3 to Hive 4

Hi Ratnesh, 1. Hive 4.x is expected to support reading ACID tables created with Hive 3.x. Could you please share the exact error message or exception you are encountering? Make sure 'hive' user has access to the underlying data. 2. Are you using MR or query-based compaction? Have you checked th

Re: Question on Apache Hive + AWS Glue Data Catalog

Unsubscribe David Novogrodsky david.novogrod...@gmail.com http://www.linkedin.com/in/davidnovogrodsky On Fri, Apr 25, 2025 at 9:34 AM Sungwoo Park wrote: > Hello, > > I am wondering if anyone uses Apache Hive 3 or 4 with AWS Glue Data > Catalog. There is a git repository for this purpose: > >

Re: Performance evaluation of Trino 468, Spark 4.0.0-RC2, and Hive 4 on Tez/MR3

can your next testing have kudu + impala included? Thanks. Sungwoo Park: For Hive-Tez, DAGAppMaster is reused across queries. Only worker containers are not reused across queries.

Re: Re: Performance evaluation of Trino 468, Spark 4.0.0-RC2, and Hive 4 on Tez/MR3

Evaluating Hive4-LLAP can be of interest to many users in this mailing list, but it's a lot of work and we are not sure if we can finish parameter tuning to achieve the best performance. For Hive-Tez, DAGAppMaster is reused across queries. Only worker containers are not reused across queries. ---

Re: Performance evaluation of Trino 468, Spark 4.0.0-RC2, and Hive 4 on Tez/MR3

>From average response time analysis: For Spark, it performs better than its total execution time suggests, with an average response time significantly lower than Hive on Tez. For long-running complex queries (like query 24) on large datasets, Hive on Tez can be a better choice than Spark, even w

Re: Performance evaluation of Trino 468, Spark 4.0.0-RC2, and Hive 4 on Tez/MR3

Thanks for the doc. I am surprised to see spark 4 is even slower than hive on Tez. [Total Execution Time (Sequential). Trino is the fastest, followed closely by Hive on MR3, which significantly outperformed Hive on Tez. Spark is the slowest, skewed by a few outlier queries.] Sungwoo Park:

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Thank you for the PR, Owen! I've added a few comments, please review them when you have time. Regards, Denys

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Hi, I have submitted the bug-fix PR we discussed. It seems that the GitHub workflow needs approval. Is there anything else I need to do? Regards, Owen 2025년 3월 27일 (목) 오후 6:11, Denys Kuzmenko 님이 작성: > Great, thank you! >

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Great, thank you!

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Note: HiveIcebergMetaHook acquires an HMS lock regardless of `iceberg.engine.hive.lock-enabled` config - needs to be refactored.

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Yes, I'd like to submit a bug-fix PR. Thank you for your proposal. 2025년 3월 27일 목요일, Denys Kuzmenko 님이 작성: > Sure, go ahead. > > To confirm, you'd like to submit a bug-fix PR as well, right? I started > looking into it, and I think a quick fix might look like: > > private HiveLock lockObject(or

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Sure, go ahead. To confirm, you'd like to submit a bug-fix PR as well, right? I started looking into it, and I think a quick fix might look like: private HiveLock lockObject(org.apache.hadoop.hive.metastore.api.Table hmsTable) { if (hiveLockEnabled(hmsTable, conf)) { return new Meta

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Dear Denys, Thank you for your confirmation. Im considering reporting a bug on JIRA and creating a PR. Please advise. Regards, Owen 2025년 3월 27일 (목) 오후 4:52, Denys Kuzmenko 님이 작성: > Note: HiveIcebergMetaHook acquires an HMS lock regardless of > `iceberg.engine.hive.lock-enabled` config - needs

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Hi Owen, It seems like a bug, but I need to check it more closely. There is no need for a stats task to take an EXCL_WRITE lock in a HiveIcebergMetaHook. Thanks, Denys

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

veServer2-Background-Pool: Thread-204] ql.Driver: Completed executing command(queryId=hive_20250327125045_22d5e40b-e2c5-49c8-a731-3acf6750edc4); Time taken: 194.688 seconds 2025-03-27T12:54:00,342 INFO [HiveServer2-Background-Pool: Thread-204] lockmgr.DbTxnManager: Stopped heartbeat for query: hive_2

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Hi, What do you mean by "possible deadlock issue"? - IOW is stuck trying to acquire a write lock (could you check this in HS2 logs)? - What is the value of `hive.txn.ext.locking.enabled` config? If 'false', pessimistic locking won't be enabled on iceberg tables. - Are you using any Hive-mana

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

I have not checked the detail about HVIE-28366. Maybe Denys can give more explanations. But for your case: For iceberg table, IOW in Hive and Insert in Trino at the same time. If you want to ensure data correctness, You should ensure that Hive/Hs2 and Trino use the same Iceberg catalog. Howev

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Thank you for your response. Based on HIVE-28366, it appears that data correctness can be ensured through HMS Lock when 'Insert' and 'IOW' operations are executed concurrently. If both operations are performed by Hive, my understanding is that the problem is resolved regardless of the 'hive.txn.man

Re: Inquiry about Possible Deadlock Problem in Hive 4.1.0 with Insert overwrite Query on Iceberg Table

Hi, IMO, 'org.apache.hadoop.hive.ql.lockmgr.DbTxnManager' is used for hive native acid table, not for Iceberg table. The lock mechanism in Iceberg table is controlled by HMS Catalog or other Iceberg Catalog(JDBC catalog, etc). So 'org.apache.hadoop.hive.ql.lockmgr.DbTxnManager' should not be us

Re: upgrade to the latest versions

Hello, I just wanted to highlight that Hive 3.x line is EOL. It has various known security vulnerabilities, many serious bugs (including wrong results and data corruption), and lacks lots of improvements and major features that are available in Hive 4. Upgrading is the right path forward. Best, S

Re: upgrade to the latest versions

well I have this stack Hive 3.1.1 Hadoop 3.1.1 Source code repository https://github.com/apache/hadoop -r 2b9a8c1d3a2caf1e733d57f346af3ff0d5ba529c Compiled by leftnoteasy on 2018-08-02T04:26Z Compiled with protoc 2.5.0 spark 3.4.4 and all work fine. What are you going to get through this upgrade

Re: Hive-standalone-metastore : where is the latest "bin" ?

Hi, I'm cutting in to share some more information. A Trino maintainer said Trino supports Hive 4, and another said there are two known issues. - https://trinodb.slack.com/archives/C0305TQ05KL/p1738231818533559 - https://github.com/trinodb/trino/issues/24453 - https://github.com/trinodb/trino/iss

Re: Hive-standalone-metastore : where is the latest "bin" ?

On 2025-02-19 22:42, Denys Kuzmenko wrote: Hi Vivien, In Hive-4.x we didn't release HMS binaries separately, so you could call it a miss. The Apache Hive community is actively preparing for the upcoming Hive 4.1 release, anticipated in the coming months. Glad to hear that. wait for your co

Re: Hive-standalone-metastore : where is the latest "bin" ?

Hi Vivien, In Hive-4.x we didn't release HMS binaries separately, so you could call it a miss. The Apache Hive community is actively preparing for the upcoming Hive 4.1 release, anticipated in the coming months. We'll discuss this internally and in the dev thread if we could build and releas

Re: Hive-standalone-metastore : where is the latest "bin" ?

You can build the standalone metastore from the 4.0.1 source but you may have compatibility problems if Trino is expecting to communicate with a <4.0 version of the metastore On Wed, 2025-02-19 at 10:07 +0100, Vivien Brissat wrote: Hello dear community, As far as i know, the latest standalone m

Re: Question: Disabling HMS S3 access when running as Spark sidecar

Check your hive-site.xml What is this set to hive.metastore.uris thrift://rhes75:9083 Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. if you find any S3-related configurations (like fs.defaultFS pointing to S3) in hive-site.xml, tha

Re: which one is better to use in hive when storage date like text type ,string or varchar?

For the most part, I think just using the string type is sufficient. Replied Message | From | liubin_w...@yeah.net | | Date | 01/16/2025 15:01 | | To | user@hive.apache.org | | Cc | | | Subject | which one is better to use in hive when storage date like text type ,string or varchar? |

Re: Re: Blog article 'Performance Tuning for Single-table Queries'

laq-uxsLA> > > -Ayush > > On 28 Dec 2024, at 4:43 PM, lisoda wrote: > > see again > > > Replied Message > From lisoda > Date 12/24/2023 00:28 > To user > Cc > Subject Re: Blog article 'Performance Tuning for Single-table Queries' &

Re: Blog article 'Performance Tuning for Single-table Queries'

Cc Subject Re: Blog article 'Performance Tuning for Single-table Queries' 🎉🚀 Replied Message From Sungwoo Park

Re: Blog article 'Performance Tuning for Single-table Queries'

see again Replied Message | From | lisoda | | Date | 12/24/2023 00:28 | | To | user | | Cc | | | Subject | Re: Blog article 'Performance Tuning for Single-table Queries' | 🎉🚀 Replied Message | From | Sungwoo Park | | Date | 12/24/2023 00:06 | | To | user@hive.

Re: Blog article 'Performance Tuning for Single-table Queries'

1 Replied Message | From | Sungwoo Park | | Date | 12/24/2023 00:06 | | To | user@hive.apache.org | | Cc | | | Subject | Blog article 'Performance Tuning for Single-table Queries' | Hello Hive users, I have published a new blog article 'Performance Tuning for Single-table Queries'.

Re: Building standalone-metastore binary for Hive 4.x

Hi Andor, I tried to build it on my machine and managed without any problems. I am not sure what is the issue when building it with docker. Try with JDK8, since 17 support is on it's way and not yet finished. Regards, Zoltán On Mon, Dec 2, 2024 at 2:26 PM Márkus Andor Rudolf wrote: > Hi Stama

Re: Building standalone-metastore binary for Hive 4.x

Hi Zoltán, Thank you for the suggestion! Switching to the maven:3-eclipse-temurin-8-focal image resolved the compilation error. The build is now completing successfully. Best regards, Andor On Mon, 2 Dec 2024 at 15:47, Zoltán Rátkai wrote: > Hi Andor, > > I tried to build it on my machine and

Re: Building standalone-metastore binary for Hive 4.x

Hi Stamatis, Thank you for the detailed build instructions. I understand the reasoning behind discontinuing separate standalone builds, and I appreciate the guidance on building it myself. Following your instructions, I attempted to create a Dockerfile ( attached below ) for the build process. Ho

Re: Building standalone-metastore binary for Hive 4.x

Hi Andor, The Hive community is not doing separate releases for standalone metastore anymore because it adds overhead to the release process. Since now we never had requests to provide these individual binaries. The functionality is still there so you may be able to construct the standalone metas

Re: Building standalone-metastore binary for Hive 4.x

Hi Zoltán, Thank you for your response. Let me provide additional context about our specific use case. We're running Spark on Kubernetes where we need Hive Metastore Standalone as a sidecar container in our Spark driver pod. This setup enables secure communication between Spark and our MySQL meta

Re: Building standalone-metastore binary for Hive 4.x

Hi Andor Rudolf, you can start metastore with hive --service metastore Since Hive 4 there is a docker image you can experience with: https://hub.docker.com/r/apache/hive https://hive.apache.org/developement/quickstart/ It show you how you can start standalon Metastore with different DBs. Rega

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hi Sungwoo, Thanks for reporting the impact of those patches. I'm happy to see the decreasing number of seconds. This is a great thread to learn the mechanism of `tez.runtime.pipelined-shuffle.enabled`. I also want to withdraw the mention of priority. If a user is willing to use it, it has use ca

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hello, We've merged all three pull requests. Thanks for your contributions. > The updated version of HIVE-28489 additionally reduces the total running time of 10TB TPC-DS by about 100 seconds. So, the total running time now decreases from around 5700s to 5200s. Considering the maturity of the Hiv

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hi, We've merged all three pull requests. Thanks for your contributions. > 1. The query plan is identical, but Trino is much faster. This is due to the > architectural difference between Trino and Hive (on shuffle-intensive > queries): Trino is based on MPP and thus uses the push model, while H

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hello, Thanks a lot for reviewing HIVE-28489 in detail. So, finally it looks like all the three patches will be merged to Hive! While analyzing query plans generated by Hive and Trino, we identified a few more categories of queries on which Trino was much faster than Hive, such as: 1. The query

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hi, Thanks for submitting the patches and writing the post with fantastic illustrations. The impressive documentation linking each ticket made it easy for me to review. I gave +1 to the last pull request. Unless anyone finds another issue, I will merge it in 1 day. https://github.com/apache/hive/

Re: [ANNOUNCE] New Committer: Shohei Okumiya

Hi, everyone in the Hive community! I am honored to have the excellent opportunity to maintain the leading enterprise data warehouse software, and I appreciate all the community members' help. Following the guidance, I created my first pull request as a committer. I am also pleased to include my

Re: [ANNOUNCE] New Committer: Shohei Okumiya

Congratulations -AyushOn 12 Nov 2024, at 2:53 PM, Butao Zhang wrote:This is definitely a good news! Congratulations Shohei!!! Look forward to your further interactions with Apache Hive community! Thanks,Butao Zhang Replied Message FromStamatis ZampetakisDate11/12/2024 17:17To

Re: [ANNOUNCE] New Committer: Shohei Okumiya

This is definitely a good news! Congratulations Shohei!!! Look forward to your further interactions with Apache Hive community! Thanks, Butao Zhang Replied Message | From | Stamatis Zampetakis | | Date | 11/12/2024 17:17 | | To | dev, | | Subject | [ANNOUNCE] New Committer: Shohei O

Re: [ANNOUNCE] New Committer: Dmitriy Fingerman

Congratulations Dmitriy Fingerman on the commitership. Very well deserved!! Thanks, Sai. On Wed, Nov 6, 2024 at 7:57 AM Dmitriy Fingerman < dmitriy.fingerman@gmail.com> wrote: > Thanks a lot Denys, Butao and the entire Hive community! I am very happy > to become a committer! Looking forward

Re: [ANNOUNCE] New Committer: Dmitriy Fingerman

Thanks a lot Denys, Butao and the entire Hive community! I am very happy to become a committer! Looking forward to continue working with you all! On Wed, Nov 6, 2024, 9:48 a.m. Butao Zhang wrote: > Congratulation Dmitriy Fingerman! You did lots of good jobs about iceberg > performance improvemen

Re: [ANNOUNCE] New Committer: Dmitriy Fingerman

Congratulation Dmitriy Fingerman! You did lots of good jobs about iceberg performance improvement, especially table optimization&compaction. Looking forward your further contribution!!! Thanks, Butao Zhang Replied Message | From | Denys Kuzmenko | | Date | 11/6/2024 22:11 | | To |

Re: hive:4.0.1 docker with s3a managed location

Hi Chris, I wanted to try the same with the official docker image, however, looks like /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar is not even added to classpath: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found I'll create a ticket for that. But to answer your original quest

Re: [ANNOUNCE] New Apache Hive PMC member : Zhihua Deng

Congratulations Zhihua Deng! Well deserved! Thanks , Simhadri G On Sat, Oct 12, 2024, 6:48 PM Butao Zhang wrote: > Congratulations, Zhihua!!! > > Thanks. > > -- > *From:* dev-return-161390-butaozhang1=163@hive.apache.org > on behalf of > Ayush Saxena > *Sent:*

Re: [ANNOUNCE] New Apache Hive PMC member : Zhihua Deng

Congratulations, Zhihua!!! Thanks.  From: dev-return-161390-butaozhang1=163@hive.apache.org on behalf of Ayush Saxena Sent: Saturday, October 12, 2024 8:55 PMTo: dev ; user@hive.apache.org Subject: [ANNOUNCE] New Apache Hive PMC member : Zhihua Deng Hi All, It gives

Re: HIVE-28488/28489/28490 and the performance of Hive 4.0.1 on MR3 1.12 (vs Trino 453)

Hello, For your query, pre-partitioning is performed on if() expressions. Since the output of if() expressions is skewed, the resultant query plan is inefficient. We think that the pre-partitioning columns should be restricted to those found in grouping sets (which I think is the case in Trino).

Re: Hive4 compatibility issues with MariaDb

Hi Denys, Thanks so much for looking into this . I have filed below JIRA for this issue https://issues.apache.org/jira/browse/HIVE-28567 Thanks, Ratnesh On Wed, Oct 9, 2024 at 1:45 PM Denys Kuzmenko wrote: > Hi Ratnesh, > > Thanks for reporting the issue. Would you be able to create a JIRA tick

Re: Hive4 compatibility issues with MariaDb

Hi Ratnesh, Thanks for reporting the issue. Would you be able to create a JIRA ticket with the same? I did a quick search and found escape chars being added only in case of direct SQL [1] {code} // The following syntax is required for using LIKE clause wildcards '_' and '%' as literals.

Re: [ANNOUNCE] Hive 3.x EOL

Hi, First of all, thanks for all the work you are putting in this project! However, for me it feels a bit rushed :/ Hive 4.0 was just released in April 30th, and in less than 6 months we have seen the EOL of 3 major versions of Hive :S I assume that the most pressing issue is the lack of manpower

Re: Question related to reuse of BytesColumnVector.vector[][].

Hello, The error looks like a bug and is data/query specific thus I assume reproducible. I would suggest filing a JIRA ticket with as many details as possible (query, DDLs, logs, plans, data) to reproduce the issue. Best, Stamatis On Sun, Sep 29, 2024 at 7:51 AM lisoda wrote: > > Currently, whe

Re: HIVE-22392 appears to have failed to be handled correctly

In fact, HIVE-22392 was merged to master branch. But this feature(writing data) can not be used. see the unresolved ticket HIVE-25717. Thanks, Butao Zhang Replied Message | From | lisoda | | Date | 9/24/2024 16:51 | | To | user@hive.apache.org | | Subject | HIVE-22392 appears to h

RE: Hive 4 integration to store table on S3 and ADLS gen2

: RE: Hive 4 integration to store table on S3 and ADLS gen2 Thanks, Raghav, for the detailed explanation. I will go through your details and follow the instructions. Below is my few findings. I have done docker images setup for hive 4.0.0 and able store table on S3 but while inserting its

RE: Hive 4 integration to store table on S3 and ADLS gen2

=08S01,code=1) 0: jdbc:hive2://localhost:1/> Thanks, Somesh From: Raghav Aggarwal Sent: Thursday, September 19, 2024 10:41 PM To: d...@hive.apache.org Cc: Ayush Saxena ; user@hive.apache.org; d...@iceberg.apache.org; Awasthi, Somesh Subject: Re: Hive 4 integration to store table on S3 and A

Re: IMPORTANT: Hive date parsing issue, input needed

, so as people can get >> >> it. >> >> >> >> I will vote for (2) considering it was returning `null` earlier as >> >> well & the new behaviour came up recently, it would be a behaviour >> >> change for folks migrating from earlier ver

Re: IMPORTANT: Hive date parsing issue, input needed

ns of hive to 4.x & the > >> new behaviour ain't that fancy either... > >> > >> -Ayush > >> > >> -- Forwarded message - > >> From: Zoltán Rátkai > >> Date: Wed, 18 Sept 2024 at 12:55 > >> Subject: Re: IMPOR

Re: IMPORTANT: Hive date parsing issue, input needed

rating from earlier versions of hive to 4.x & the > >> new behaviour ain't that fancy either... > >> > >> -Ayush > >> > >> -- Forwarded message - > >> From: Zoltán Rátkai > >> Date: Wed, 18 Sept 2024 at 1

RE: Hive 4 integration to store table on S3 and ADLS gen2

Anyone can help here what is wrong with setup From: Awasthi, Somesh Sent: Wednesday, September 18, 2024 1:34 PM To: Ayush Saxena ; d...@hive.apache.org Cc: user@hive.apache.org; d...@iceberg.apache.org Subject: RE: Hive 4 integration to store table on S3 and ADLS gen2 Any idea plz suggest. From

Re: Planning Hive 4.0.1

Hi Zhihua, Thanks for driving the release. I’ve added labels to a few more tickets that were already resolved but seem like good candidates for inclusion in the 4.0.1 release. -Ayush On Wed, 18 Sept 2024 at 15:51, Zhihua Deng wrote: > > Hi Team, > > As most of the issues in > https://issues.apa

Re: IMPORTANT: Hive date parsing issue, input needed

turning `null` earlier as >> well & the new behaviour came up recently, it would be a behaviour >> change for folks migrating from earlier versions of hive to 4.x & the >> new behaviour ain't that fancy either... >> >> -Ayush >> >> -

Re: IMPORTANT: Hive date parsing issue, input needed

up recently, it would be a behaviour > change for folks migrating from earlier versions of hive to 4.x & the > new behaviour ain't that fancy either... > > -Ayush > > -- Forwarded message - > From: Zoltán Rátkai > Date: Wed, 18 Sept 2024 at 12:55 &

RE: Hive 4 integration to store table on S3 and ADLS gen2

Any idea plz suggest. From: Awasthi, Somesh Sent: Wednesday, September 18, 2024 11:52 AM To: Ayush Saxena ; d...@hive.apache.org Cc: user@hive.apache.org; d...@iceberg.apache.org Subject: RE: Hive 4 integration to store table on S3 and ADLS gen2 Hi Aayush thanks for your quick response . Hadoop

Re: IMPORTANT: Hive date parsing issue, input needed

Hi Ayush, thanks for your thoughts! My answers: - On the official Hive page about types this is not documented at all: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=82706456#LanguageManualTypes-date "Dates: DATE values describe a particular year/month/day, in the form *-

Re: IMPORTANT: Hive date parsing issue, input needed

++ Adding the dev mailing list, as this might be relevant to them as well. I have a few additional questions: * Has this behavior been present since the inception of Hive, or did it start occurring more recently? If this has been the behavior for over a decade, it might be best to leave it unchan

RE: Hive 4 integration to store table on S3 and ADLS gen2

it seamlessly Thanks for u support. Thanks, Somesh From: Ayush Saxena Sent: Wednesday, September 18, 2024 11:41 AM To: d...@hive.apache.org Cc: user@hive.apache.org; d...@iceberg.apache.org; Awasthi, Somesh Subject: Re: Hive 4 integration to store table on S3 and ADLS gen2 Caution: This email

Re: Hive 4 integration to store table on S3 and ADLS gen2

Hi Somesh, But while trying so we are seeing following exception : > hadoop fs -ls s3a://somesh.qa.bucket/ -: This has nothing to do with Hive as such, You have configured Hadoop S3 client wrong, you are missing configs, your hadoop ls command itself is failing, there is no Hive involved here.

Re: Re: Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Hi Clinton, I'd like you to start a new thread about the problem with MV. It is far from the original syntax problem, and MV is a fairly advanced feature. A new thread with a proper title would attract more appropriate people. You can also present the minimal SQLs that reproduce the problem on th

Re:Re: Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

hello Okumin. After porting the patch HIVE-28428, I observed that the slow query problem disappeared.Currently the query efficiency for orc+zstd table is basically the same as orc+snappy. Therefore I strongly recommend addingHIVE-28428to version 4.0.1. Regards, lisoda At 2024-09-02 23:28

Re: Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Thanks Lisoda for those insights. @Okumin , this is what I observed when checking the log files. Attached is a log file and the hive-site.xml file configuration. I have observed this error comes when the execution engine is set to Tez , the moment i switch to MR the issue does not come up. This

Re: Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Hi Clinton, Thanks for sharing your problem. If you provide more information, such as a dataset or queries, we can reproduce it and file the problem. Hi Lisoda, Thanks for giving us real examples. Interesting. Can I understand the first problem that happens when there is a big data file in an Ic

Re: Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Hello Lisoda, Thanks for this information. On Sun, Sep 1, 2024 at 4:04 PM lisoda wrote: > Hello Clinton: > > We have actually encountered the same issue where, in many cases, querying > Iceberg does not meet expected efficiency, falling short of regular > ORC/Parquet tables in speed. Since the

Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Thanks Okumin. I am new to Hive and Tez and I have struggled to deploy a high-performance Dockerized Hive setup. I followed the documentation for setting up a remote Metastore. I have a single node with 32 GB of RAM and 8 cores, but I have a dataset of about 2 GB (Iceberg table partitioned on on

Re: Merge Operation Failing Results in this SQL Error [40000] [42000]

Hi Clinton, I tested MERGE INTO with minimal reproduction. I saw the same error. ``` CREATE TABLE src (col1 INT, col2 INT); CREATE TABLE dst (id BIGINT DEFAULT SURROGATE_KEY(), col1 INT, col2 INT, PRIMARY KEY (id) DISABLE NOVALIDATE) STORED BY ICEBERG; MERGE INTO dst d USING src s ON s.col1 = d.

Re: Hive 4.0.0 on MR3 released

Hi all, We have written a blog article that compares Hive 4 on MR3 and Trino 453 (the latest release of Trino) using the 10TB TPC-DS benchmark. While it does not evaluate vanilla Hive, the results might be useful to those interested in the overall speed of Hive 4 in LLAP mode. https://www.datamon

Re: Hive 4.0.0 on MR3 released

Congratulations, Sungwoo. I look forward to trying out Hive 4.0.0 on MR3 when my time allows. David On Tue, Jul 30, 2024 at 11:29:02PM +0900, Sungwoo Park wrote: > Hi all, > > We would like to announce the release of Hive 4.0.0 on MR3. It is based on > Hive 4.0.0 (together with 17 commits from

Re: Hive ldap auth failing from db client tool using jdbc driver

Error message, as per log form hive server docker container I am not sure how this is showing, “hive.jdbc.client.authentication”, I do not have this in hive-site.xml. 2024-07-15T23:14:49,022 ERROR [HiveServer2-Handler-Pool: Thread-78] thrift.ThriftCLIService: Login attempt failed for user : <

Re: Iceberg HadoopCatalog and location_based_table

see `HadoopInputFile` as an example

Re: Iceberg HadoopCatalog and location_based_table

Hi lisoda, We keep hive-iceberg catalog and handler modules inside of Hive itself + provide a possibility to patch iceberg native core and api modules. If you'd like to continue using iceberg-hadoop-catalog, you could try to move relevant classes into iceberg/patched-iceberg-core. Regards, Den

Re:Re: Re: Support java/11/17/21

Thank you for your reply.Thank you for letting me know about this. At 2024-07-10 16:16:16, "Ayush Saxena" wrote: >I know about that, I only flagged that [1] :-) > >That got sorted by upgrading the protobuf to 3.23, [2], & merging [3] > >Hadoop can't even compile JDK-8+, there are bunch

Re: Re: Support java/11/17/21

I know about that, I only flagged that [1] :-) That got sorted by upgrading the protobuf to 3.23, [2], & merging [3] Hadoop can't even compile JDK-8+, there are bunch of issue, Jersey being the biggest one, All the JDK upgrade tickets are open https://issues.apache.org/jira/browse/HADOOP-16795 ht

Re: Support java/11/17/21

We are working towards supporting JDK-17, should take couple of months, we don’t have a planned deadline for that as of now Btw. hadoop didn’t drop support for JDK-8…. -Ayush > On 10 Jul 2024, at 12:52 PM, lisoda wrote: > >  > Hi. Currently, Iceberg/hadoop/spark and the rest of the third-par

Re: Re: Next Hive 4.0.1 minor release

Hi, I'm posting some more suggestions. # We may label HIVE-28352 as "hive-4.0.1-must" I found that users were unable to upgrade some system tables from Hive 3 through schematool. It would be better to fix. https://issues.apache.org/jira/browse/HIVE-28352 # We may drop HIVE-24167 I am the ticket

Re: Next Hive 4.0.1 minor release

Hi Zhihua, Thanks for driving the next release. We are actively testing 4.0.0 and would like to give some suggestions. # HIVE-27847: Prevent query Failures on Numeric <-> Timestamp We hit the issue when we ran Hive 4 with the option. I believe it is worth resolving for those who want to try Hive

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

ally appreciate it, it means a lot to me :) >>>>>>> The Apache Hive project and its community have truly inspired me . >>>>>>> I'm grateful for the chance to contribute to such a remarkable project. >>>>>>> >>>>&g

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

remarkable project. >>>>>> >>>>>> Thanks! >>>>>> Simhadri Govindappa >>>>>> >>>>>> On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan >>>>>> wrote: >>>>>> >>

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

27;m >>>>> grateful for the chance to contribute to such a remarkable project. >>>>> >>>>> Thanks! >>>>> Simhadri Govindappa >>>>> >>>>> On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan >>>

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

ppa >>>> >>>> On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan >>>> wrote: >>>> >>>>> Congrats Simhadri! >>>>> >>>>> >>>>> >>>>> -Sankar >>>>> >>>>>

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

oject. >>> >>> Thanks! >>> Simhadri Govindappa >>> >>> On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan >>> wrote: >>> >>>> Congrats Simhadri! >>>> >>>> >>>> >>>> -Sanka

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

t;> >>> Congrats Simhadri! >>> >>> >>> >>> -Sankar >>> >>> >>> >>> *From:* Butao Zhang >>> *Sent:* Thursday, April 18, 2024 5:39 PM >>> *To:* user@hive.apache.org; dev >>&

  1   2   3   4   5   6   7   8   9   10   >