Oh, I see, let me check it again.
-- Best Regards Mingyu Chen Email: morning...@apache.org At 2023-10-05 21:49:28, "Joice Jacob" <joicejacob1...@gmail.com> wrote: >Hi, >I wanted to share some important information regarding the CDP Hive version >that I am currently using, which is Hive 3.1.3000.7.1.7.0-551. > >In CDP Hive version 3.x, a significant change has been introduced regarding >managed tables. By default, managed tables in Hive 3.x are considered >transactional. This means that if you create a managed table explicitly >specifying it as transactional with a value of "false," it will be treated >as an external table instead. >Thank you for your attention to this matter, and I appreciate your >continued support. >https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/ >Best regards, >Joice > >On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen <morning...@163.com> wrote: > >> I saw that your hive table is with property "transactional" = "true", >> And Doris only support ACID table with Hive 3, not support with Hive 2.x. >> So you may need to create non-transactional hive table for Doris to visit. >> >> >> -- >> Best Regards >> Mingyu Chen >> >> Email: >> morning...@apache.org >> >> >> 在 2023-10-05 13:30:25，"Joice Jacob" <joicejacob1...@gmail.com> 写道： >> >> Dear Doris Community & Jiafeng.Zhang >> , >> Thank you for your prompt response and your willingness to assist with the >> issue I've been encountering with the integration of Doris and Cloudera's >> Hive. >> >> To provide you with the necessary information, here are the details you >> requested: >> >> *1. Hive Catalog Creation Statement:* >> >> My Hive catalog creation statement is as follows: >> >> CREATE CATALOG hive PROPERTIES ( >> 'type'='hms', >> 'hive.metastore.uris' = 'thrift://10.128.0.4:9083', >> 'hive.version' = '3.1.3', >> 'hive.metastore.sasl.enabled' = 'true', >> 'hive.metastore.kerberos.principal' = 'hive/instanc...@hadoop.com', >> 'hadoop.security.authentication' = 'kerberos', >> 'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab', >> 'hadoop.kerberos.principal' = 'hive/instanc...@hadoop.com', >> 'yarn.resourcemanager.principal' = 'yarn/instanc...@hadoop.com' >> ); >> >> *2. CDP Hive Version:* >> >> The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551 >> >> *3. Configuration Attempt:* >> >> I have tried specifying the Hive version in my catalog configuration with >> the following statement: >> "hive.version=2.1.0" >> Despite attempting to set the Hive version to 2.1.0, I continue to >> experience the same issue, which is detailed in my previous emails. >> >> *4. Logs: * >> I will attach both the fe.log and fe.warn logs to this email for your >> reference. These logs should provide additional context regarding the issue >> I'm facing. >> >> Once again, I want to express my gratitude for your assistance and support >> in resolving this matter. I look forward to your insights and >> recommendations based on the provided logs and catalog configuration. >> >> Please feel free to let me know if you require any further information or >> if there are additional steps I should take to assist in diagnosing and >> resolving the issue. >> >> [image: doris_error.JPG] >> fe.log log at hive catlog creation time >> [image: fe_log_at_hivecatalog.JPG] >> Hive managed table script >> [image: image.png] >> >> >> Best regards, >> Joice >> >> On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <zhang...@gmail.com> wrote: >> >>> Can you provide your hive catalog creation statement, your fe.log log at >>> that time, and your cdp hive version? This will help us locate the >>> problem, >>> thank you. >>> You can also try specifying your hive version in your catalog statement: >>> "hive.version=2.1.0" >>> >>> Joice Jacob <joicejacob1...@gmail.com> 于2023年10月5日周四 00:45写道： >>> >>> > I am reaching out once again to seek assistance and share a specific >>> issue >>> > I've encountered while integrating Doris with Cloudera-flavored Hive. >>> The >>> > error message I'm facing is as follows: >>> > detailMessage = get file split failed for table: baby1, err: >>> > java.lang.Exception: Hive 2.x versioned full-acid tables need to run >>> major >>> > compaction. >>> > >>> > This error message appears when I attempt to query a Hive table from >>> > Doris, and it seems to be related to Hive 2.x versioned full-ACID tables >>> > requiring a major compaction. >>> > I would like to ask the Doris community for guidance on how to handle >>> this >>> > issue effectively. Specifically, I am interested in understanding the >>> best >>> > practices and steps to follow when dealing with Cloudera-flavored Hive >>> > tables that require major compaction for Doris integration. >>> > >>> > If anyone in the community has successfully addressed this issue or can >>> > provide insights into how to configure and manage Cloudera-flavored Hive >>> > tables for integration with Doris, your expertise would be highly >>> > appreciated. >>> > >>> > Thank you for your time and support, and I look forward to receiving >>> your >>> > valuable input. >>> > Joice >>> > >>> > On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <joicejacob1...@gmail.com> >>> > wrote: >>> > >>> >> Dear Doris Community, >>> >> I have an update on the issue I previously mentioned regarding the >>> >> integration of Hive on a CDP distribution with Doris 2.0.1.1. >>> >> >>> >> After further investigation, I have identified that the issue is >>> related >>> >> to Hive managed tables being transactional by default. This appears to >>> be >>> >> causing the error I encountered earlier. >>> >> >>> >> To address this issue, I am seeking guidance from the community on any >>> >> specific configurations or settings that need to be adjusted for Hive >>> >> transactional tables when using Hive as the catalog in Doris. Are >>> there any >>> >> recommended configurations or best practices that I should follow to >>> ensure >>> >> smooth integration and query execution? >>> >> >>> >> Any insights or recommendations from the Doris community would be >>> greatly >>> >> appreciated. Your expertise and guidance will be instrumental in >>> helping me >>> >> resolve this challenge. >>> >> >>> >> Thank you for your continued support, and I look forward to your >>> valuable >>> >> input. >>> >> >>> >> Best regards, >>> >> Joice >>> >> >>> >> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <joicejacob1...@gmail.com> >>> >> wrote: >>> >> >>> >>> Dear Doris Community, >>> >>> >>> >>> I am reaching out to the community to seek assistance with an >>> >>> integration issue I've encountered while trying to use Hive on a CDP >>> >>> distribution with Doris 2.0.1.1. >>> >>> >>> >>> Here are the details of my setup: >>> >>> >>> >>> Doris Version: 2.0.1.1 >>> >>> Hive Version: 3.1.3 >>> >>> Cluster Security: Kerberized >>> >>> >>> >>> I have successfully created a Hive catalog in Doris and have been able >>> >>> to set up the integration between Hive and Doris. However, when I >>> attempt >>> >>> to query a Hive table using Doris, I encounter the following error: >>> >>> >>> >>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split failed >>> >>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned >>> full-acid >>> >>> tables need to run major compaction. >>> >>> >>> >>> I have already performed a major compaction as recommended, but I am >>> >>> still encountering the same error. >>> >>> >>> >>> I would greatly appreciate any insights, guidance, or solutions that >>> the >>> >>> Doris community can offer to help me resolve this issue. If anyone has >>> >>> encountered a similar problem or has expertise in integrating Hive >>> with >>> >>> Doris, your assistance would be invaluable. >>> >>> >>> >>> Thank you in advance for your time and support. I look forward to >>> >>> hearing from the community and working together to find a solution to >>> this >>> >>> challenge. >>> >>> >>> >>> [image: baby_table.JPG] >>> >>> [image: doris_hive_catlog_result.JPG] >>> >>> >>> >>> [image: doris_fe_log.JPG] >>> >>> >>> >>> [image: hive_metastore_error.JPG] >>> >>> >>> >>> Thanks >>> >>> Joice >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> 张家峰 >>> >>
