Hi, I wanted to share some important information regarding the CDP Hive version that I am currently using, which is Hive 3.1.3000.7.1.7.0-551.
In CDP Hive version 3.x, a significant change has been introduced regarding managed tables. By default, managed tables in Hive 3.x are considered transactional. This means that if you create a managed table explicitly specifying it as transactional with a value of "false," it will be treated as an external table instead. Thank you for your attention to this matter, and I appreciate your continued support. https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/ Best regards, Joice On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen <morning...@163.com> wrote: > I saw that your hive table is with property "transactional" = "true", > And Doris only support ACID table with Hive 3, not support with Hive 2.x. > So you may need to create non-transactional hive table for Doris to visit. > > > -- > Best Regards > Mingyu Chen > > Email: > morning...@apache.org > > > 在 2023-10-05 13:30:25，"Joice Jacob" <joicejacob1...@gmail.com> 写道： > > Dear Doris Community & Jiafeng.Zhang > , > Thank you for your prompt response and your willingness to assist with the > issue I've been encountering with the integration of Doris and Cloudera's > Hive. > > To provide you with the necessary information, here are the details you > requested: > > *1. Hive Catalog Creation Statement:* > > My Hive catalog creation statement is as follows: > > CREATE CATALOG hive PROPERTIES ( > 'type'='hms', > 'hive.metastore.uris' = 'thrift://10.128.0.4:9083', > 'hive.version' = '3.1.3', > 'hive.metastore.sasl.enabled' = 'true', > 'hive.metastore.kerberos.principal' = 'hive/instanc...@hadoop.com', > 'hadoop.security.authentication' = 'kerberos', > 'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab', > 'hadoop.kerberos.principal' = 'hive/instanc...@hadoop.com', > 'yarn.resourcemanager.principal' = 'yarn/instanc...@hadoop.com' > ); > > *2. CDP Hive Version:* > > The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551 > > *3. Configuration Attempt:* > > I have tried specifying the Hive version in my catalog configuration with > the following statement: > "hive.version=2.1.0" > Despite attempting to set the Hive version to 2.1.0, I continue to > experience the same issue, which is detailed in my previous emails. > > *4. Logs: * > I will attach both the fe.log and fe.warn logs to this email for your > reference. These logs should provide additional context regarding the issue > I'm facing. > > Once again, I want to express my gratitude for your assistance and support > in resolving this matter. I look forward to your insights and > recommendations based on the provided logs and catalog configuration. > > Please feel free to let me know if you require any further information or > if there are additional steps I should take to assist in diagnosing and > resolving the issue. > > [image: doris_error.JPG] > fe.log log at hive catlog creation time > [image: fe_log_at_hivecatalog.JPG] > Hive managed table script > [image: image.png] > > > Best regards, > Joice > > On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <zhang...@gmail.com> wrote: > >> Can you provide your hive catalog creation statement, your fe.log log at >> that time, and your cdp hive version? This will help us locate the >> problem, >> thank you. >> You can also try specifying your hive version in your catalog statement: >> "hive.version=2.1.0" >> >> Joice Jacob <joicejacob1...@gmail.com> 于2023年10月5日周四 00:45写道： >> >> > I am reaching out once again to seek assistance and share a specific >> issue >> > I've encountered while integrating Doris with Cloudera-flavored Hive. >> The >> > error message I'm facing is as follows: >> > detailMessage = get file split failed for table: baby1, err: >> > java.lang.Exception: Hive 2.x versioned full-acid tables need to run >> major >> > compaction. >> > >> > This error message appears when I attempt to query a Hive table from >> > Doris, and it seems to be related to Hive 2.x versioned full-ACID tables >> > requiring a major compaction. >> > I would like to ask the Doris community for guidance on how to handle >> this >> > issue effectively. Specifically, I am interested in understanding the >> best >> > practices and steps to follow when dealing with Cloudera-flavored Hive >> > tables that require major compaction for Doris integration. >> > >> > If anyone in the community has successfully addressed this issue or can >> > provide insights into how to configure and manage Cloudera-flavored Hive >> > tables for integration with Doris, your expertise would be highly >> > appreciated. >> > >> > Thank you for your time and support, and I look forward to receiving >> your >> > valuable input. >> > Joice >> > >> > On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <joicejacob1...@gmail.com> >> > wrote: >> > >> >> Dear Doris Community, >> >> I have an update on the issue I previously mentioned regarding the >> >> integration of Hive on a CDP distribution with Doris 2.0.1.1. >> >> >> >> After further investigation, I have identified that the issue is >> related >> >> to Hive managed tables being transactional by default. This appears to >> be >> >> causing the error I encountered earlier. >> >> >> >> To address this issue, I am seeking guidance from the community on any >> >> specific configurations or settings that need to be adjusted for Hive >> >> transactional tables when using Hive as the catalog in Doris. Are >> there any >> >> recommended configurations or best practices that I should follow to >> ensure >> >> smooth integration and query execution? >> >> >> >> Any insights or recommendations from the Doris community would be >> greatly >> >> appreciated. Your expertise and guidance will be instrumental in >> helping me >> >> resolve this challenge. >> >> >> >> Thank you for your continued support, and I look forward to your >> valuable >> >> input. >> >> >> >> Best regards, >> >> Joice >> >> >> >> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <joicejacob1...@gmail.com> >> >> wrote: >> >> >> >>> Dear Doris Community, >> >>> >> >>> I am reaching out to the community to seek assistance with an >> >>> integration issue I've encountered while trying to use Hive on a CDP >> >>> distribution with Doris 2.0.1.1. >> >>> >> >>> Here are the details of my setup: >> >>> >> >>> Doris Version: 2.0.1.1 >> >>> Hive Version: 3.1.3 >> >>> Cluster Security: Kerberized >> >>> >> >>> I have successfully created a Hive catalog in Doris and have been able >> >>> to set up the integration between Hive and Doris. However, when I >> attempt >> >>> to query a Hive table using Doris, I encounter the following error: >> >>> >> >>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split failed >> >>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned >> full-acid >> >>> tables need to run major compaction. >> >>> >> >>> I have already performed a major compaction as recommended, but I am >> >>> still encountering the same error. >> >>> >> >>> I would greatly appreciate any insights, guidance, or solutions that >> the >> >>> Doris community can offer to help me resolve this issue. If anyone has >> >>> encountered a similar problem or has expertise in integrating Hive >> with >> >>> Doris, your assistance would be invaluable. >> >>> >> >>> Thank you in advance for your time and support. I look forward to >> >>> hearing from the community and working together to find a solution to >> this >> >>> challenge. >> >>> >> >>> [image: baby_table.JPG] >> >>> [image: doris_hive_catlog_result.JPG] >> >>> >> >>> [image: doris_fe_log.JPG] >> >>> >> >>> [image: hive_metastore_error.JPG] >> >>> >> >>> Thanks >> >>> Joice >> >>> >> >>> >> >>> >> >>> >> >>> >> >> -- >> 张家峰 >> >
