Could you list your hive data dir, to see what files exist? like: _orc_acid_version
-- Best Regards Mingyu Chen Email: morning...@apache.org At 2023-10-05 22:31:58, "Mingyu Chen" <morning...@163.com> wrote: >Oh, I see, let me check it again. > > > > >-- > >Best Regards >Mingyu Chen > >Email: >morning...@apache.org > > > > > >At 2023-10-05 21:49:28, "Joice Jacob" <joicejacob1...@gmail.com> wrote: >>Hi, >>I wanted to share some important information regarding the CDP Hive version >>that I am currently using, which is Hive 3.1.3000.7.1.7.0-551. >> >>In CDP Hive version 3.x, a significant change has been introduced regarding >>managed tables. By default, managed tables in Hive 3.x are considered >>transactional. This means that if you create a managed table explicitly >>specifying it as transactional with a value of "false," it will be treated >>as an external table instead. >>Thank you for your attention to this matter, and I appreciate your >>continued support. >>https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/ >>Best regards, >>Joice >> >>On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen <morning...@163.com> wrote: >> >>> I saw that your hive table is with property "transactional" = "true", >>> And Doris only support ACID table with Hive 3, not support with Hive 2.x. >>> So you may need to create non-transactional hive table for Doris to visit. >>> >>> >>> -- >>> Best Regards >>> Mingyu Chen >>> >>> Email: >>> morning...@apache.org >>> >>> >>> 在 2023-10-05 13:30:25，"Joice Jacob" <joicejacob1...@gmail.com> 写道： >>> >>> Dear Doris Community & Jiafeng.Zhang >>> , >>> Thank you for your prompt response and your willingness to assist with the >>> issue I've been encountering with the integration of Doris and Cloudera's >>> Hive. >>> >>> To provide you with the necessary information, here are the details you >>> requested: >>> >>> *1. Hive Catalog Creation Statement:* >>> >>> My Hive catalog creation statement is as follows: >>> >>> CREATE CATALOG hive PROPERTIES ( >>> 'type'='hms', >>> 'hive.metastore.uris' = 'thrift://10.128.0.4:9083', >>> 'hive.version' = '3.1.3', >>> 'hive.metastore.sasl.enabled' = 'true', >>> 'hive.metastore.kerberos.principal' = 'hive/instanc...@hadoop.com', >>> 'hadoop.security.authentication' = 'kerberos', >>> 'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab', >>> 'hadoop.kerberos.principal' = 'hive/instanc...@hadoop.com', >>> 'yarn.resourcemanager.principal' = 'yarn/instanc...@hadoop.com' >>> ); >>> >>> *2. CDP Hive Version:* >>> >>> The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551 >>> >>> *3. Configuration Attempt:* >>> >>> I have tried specifying the Hive version in my catalog configuration with >>> the following statement: >>> "hive.version=2.1.0" >>> Despite attempting to set the Hive version to 2.1.0, I continue to >>> experience the same issue, which is detailed in my previous emails. >>> >>> *4. Logs: * >>> I will attach both the fe.log and fe.warn logs to this email for your >>> reference. These logs should provide additional context regarding the issue >>> I'm facing. >>> >>> Once again, I want to express my gratitude for your assistance and support >>> in resolving this matter. I look forward to your insights and >>> recommendations based on the provided logs and catalog configuration. >>> >>> Please feel free to let me know if you require any further information or >>> if there are additional steps I should take to assist in diagnosing and >>> resolving the issue. >>> >>> [image: doris_error.JPG] >>> fe.log log at hive catlog creation time >>> [image: fe_log_at_hivecatalog.JPG] >>> Hive managed table script >>> [image: image.png] >>> >>> >>> Best regards, >>> Joice >>> >>> On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <zhang...@gmail.com> wrote: >>> >>>> Can you provide your hive catalog creation statement, your fe.log log at >>>> that time, and your cdp hive version? This will help us locate the >>>> problem, >>>> thank you. >>>> You can also try specifying your hive version in your catalog statement: >>>> "hive.version=2.1.0" >>>> >>>> Joice Jacob <joicejacob1...@gmail.com> 于2023年10月5日周四 00:45写道： >>>> >>>> > I am reaching out once again to seek assistance and share a specific >>>> issue >>>> > I've encountered while integrating Doris with Cloudera-flavored Hive. >>>> The >>>> > error message I'm facing is as follows: >>>> > detailMessage = get file split failed for table: baby1, err: >>>> > java.lang.Exception: Hive 2.x versioned full-acid tables need to run >>>> major >>>> > compaction. >>>> > >>>> > This error message appears when I attempt to query a Hive table from >>>> > Doris, and it seems to be related to Hive 2.x versioned full-ACID tables >>>> > requiring a major compaction. >>>> > I would like to ask the Doris community for guidance on how to handle >>>> this >>>> > issue effectively. Specifically, I am interested in understanding the >>>> best >>>> > practices and steps to follow when dealing with Cloudera-flavored Hive >>>> > tables that require major compaction for Doris integration. >>>> > >>>> > If anyone in the community has successfully addressed this issue or can >>>> > provide insights into how to configure and manage Cloudera-flavored Hive >>>> > tables for integration with Doris, your expertise would be highly >>>> > appreciated. >>>> > >>>> > Thank you for your time and support, and I look forward to receiving >>>> your >>>> > valuable input. >>>> > Joice >>>> > >>>> > On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <joicejacob1...@gmail.com> >>>> > wrote: >>>> > >>>> >> Dear Doris Community, >>>> >> I have an update on the issue I previously mentioned regarding the >>>> >> integration of Hive on a CDP distribution with Doris 2.0.1.1. >>>> >> >>>> >> After further investigation, I have identified that the issue is >>>> related >>>> >> to Hive managed tables being transactional by default. This appears to >>>> be >>>> >> causing the error I encountered earlier. >>>> >> >>>> >> To address this issue, I am seeking guidance from the community on any >>>> >> specific configurations or settings that need to be adjusted for Hive >>>> >> transactional tables when using Hive as the catalog in Doris. Are >>>> there any >>>> >> recommended configurations or best practices that I should follow to >>>> ensure >>>> >> smooth integration and query execution? >>>> >> >>>> >> Any insights or recommendations from the Doris community would be >>>> greatly >>>> >> appreciated. Your expertise and guidance will be instrumental in >>>> helping me >>>> >> resolve this challenge. >>>> >> >>>> >> Thank you for your continued support, and I look forward to your >>>> valuable >>>> >> input. >>>> >> >>>> >> Best regards, >>>> >> Joice >>>> >> >>>> >> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <joicejacob1...@gmail.com> >>>> >> wrote: >>>> >> >>>> >>> Dear Doris Community, >>>> >>> >>>> >>> I am reaching out to the community to seek assistance with an >>>> >>> integration issue I've encountered while trying to use Hive on a CDP >>>> >>> distribution with Doris 2.0.1.1. >>>> >>> >>>> >>> Here are the details of my setup: >>>> >>> >>>> >>> Doris Version: 2.0.1.1 >>>> >>> Hive Version: 3.1.3 >>>> >>> Cluster Security: Kerberized >>>> >>> >>>> >>> I have successfully created a Hive catalog in Doris and have been able >>>> >>> to set up the integration between Hive and Doris. However, when I >>>> attempt >>>> >>> to query a Hive table using Doris, I encounter the following error: >>>> >>> >>>> >>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split failed >>>> >>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned >>>> full-acid >>>> >>> tables need to run major compaction. >>>> >>> >>>> >>> I have already performed a major compaction as recommended, but I am >>>> >>> still encountering the same error. >>>> >>> >>>> >>> I would greatly appreciate any insights, guidance, or solutions that >>>> the >>>> >>> Doris community can offer to help me resolve this issue. If anyone has >>>> >>> encountered a similar problem or has expertise in integrating Hive >>>> with >>>> >>> Doris, your assistance would be invaluable. >>>> >>> >>>> >>> Thank you in advance for your time and support. I look forward to >>>> >>> hearing from the community and working together to find a solution to >>>> this >>>> >>> challenge. >>>> >>> >>>> >>> [image: baby_table.JPG] >>>> >>> [image: doris_hive_catlog_result.JPG] >>>> >>> >>>> >>> [image: doris_fe_log.JPG] >>>> >>> >>>> >>> [image: hive_metastore_error.JPG] >>>> >>> >>>> >>> Thanks >>>> >>> Joice >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>>> -- >>>> 张家峰 >>>> >>>
