Hi Guangdong Liu,
please find the attachment for DESCRIBE FORMATTED baby output
On Thu, Oct 5, 2023 at 6:27 PM Guangdong Liu <[email protected]> wrote:
> Can you see the detailed information of the table baby?
>
> `DESCRIBE FORMATTED baby;`
>
> --
>
> Best Regards
>
> ------------
>
> Liugddx
> [email protected]
>
>
> Joice Jacob <[email protected]> 于2023年10月5日周四 13:30写道：
>
>> Dear Doris Community & Jiafeng.Zhang
>> ,
>> Thank you for your prompt response and your willingness to assist with
>> the issue I've been encountering with the integration of Doris and
>> Cloudera's Hive.
>>
>> To provide you with the necessary information, here are the details you
>> requested:
>>
>> *1. Hive Catalog Creation Statement:*
>>
>> My Hive catalog creation statement is as follows:
>>
>> CREATE CATALOG hive PROPERTIES (
>> 'type'='hms',
>> 'hive.metastore.uris' = 'thrift://10.128.0.4:9083',
>> 'hive.version' = '3.1.3',
>> 'hive.metastore.sasl.enabled' = 'true',
>> 'hive.metastore.kerberos.principal' = 'hive/[email protected]',
>> 'hadoop.security.authentication' = 'kerberos',
>> 'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab',
>> 'hadoop.kerberos.principal' = 'hive/[email protected]',
>> 'yarn.resourcemanager.principal' = 'yarn/[email protected]'
>> );
>>
>> *2. CDP Hive Version:*
>>
>> The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551
>>
>> *3. Configuration Attempt:*
>>
>> I have tried specifying the Hive version in my catalog configuration with
>> the following statement:
>> "hive.version=2.1.0"
>> Despite attempting to set the Hive version to 2.1.0, I continue to
>> experience the same issue, which is detailed in my previous emails.
>>
>> *4. Logs: *
>> I will attach both the fe.log and fe.warn logs to this email for your
>> reference. These logs should provide additional context regarding the issue
>> I'm facing.
>>
>> Once again, I want to express my gratitude for your assistance and
>> support in resolving this matter. I look forward to your insights and
>> recommendations based on the provided logs and catalog configuration.
>>
>> Please feel free to let me know if you require any further information or
>> if there are additional steps I should take to assist in diagnosing and
>> resolving the issue.
>>
>> [image: doris_error.JPG]
>> fe.log log at hive catlog creation time
>> [image: fe_log_at_hivecatalog.JPG]
>> Hive managed table script
>> [image: image.png]
>>
>>
>> Best regards,
>> Joice
>>
>> On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <[email protected]> wrote:
>>
>>> Can you provide your hive catalog creation statement, your fe.log log at
>>> that time, and your cdp hive version? This will help us locate the
>>> problem,
>>> thank you.
>>> You can also try specifying your hive version in your catalog statement:
>>> "hive.version=2.1.0"
>>>
>>> Joice Jacob <[email protected]> 于2023年10月5日周四 00:45写道：
>>>
>>> > I am reaching out once again to seek assistance and share a specific
>>> issue
>>> > I've encountered while integrating Doris with Cloudera-flavored Hive.
>>> The
>>> > error message I'm facing is as follows:
>>> > detailMessage = get file split failed for table: baby1, err:
>>> > java.lang.Exception: Hive 2.x versioned full-acid tables need to run
>>> major
>>> > compaction.
>>> >
>>> > This error message appears when I attempt to query a Hive table from
>>> > Doris, and it seems to be related to Hive 2.x versioned full-ACID
>>> tables
>>> > requiring a major compaction.
>>> > I would like to ask the Doris community for guidance on how to handle
>>> this
>>> > issue effectively. Specifically, I am interested in understanding the
>>> best
>>> > practices and steps to follow when dealing with Cloudera-flavored Hive
>>> > tables that require major compaction for Doris integration.
>>> >
>>> > If anyone in the community has successfully addressed this issue or can
>>> > provide insights into how to configure and manage Cloudera-flavored
>>> Hive
>>> > tables for integration with Doris, your expertise would be highly
>>> > appreciated.
>>> >
>>> > Thank you for your time and support, and I look forward to receiving
>>> your
>>> > valuable input.
>>> > Joice
>>> >
>>> > On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <[email protected]>
>>> > wrote:
>>> >
>>> >> Dear Doris Community,
>>> >> I have an update on the issue I previously mentioned regarding the
>>> >> integration of Hive on a CDP distribution with Doris 2.0.1.1.
>>> >>
>>> >> After further investigation, I have identified that the issue is
>>> related
>>> >> to Hive managed tables being transactional by default. This appears
>>> to be
>>> >> causing the error I encountered earlier.
>>> >>
>>> >> To address this issue, I am seeking guidance from the community on any
>>> >> specific configurations or settings that need to be adjusted for Hive
>>> >> transactional tables when using Hive as the catalog in Doris. Are
>>> there any
>>> >> recommended configurations or best practices that I should follow to
>>> ensure
>>> >> smooth integration and query execution?
>>> >>
>>> >> Any insights or recommendations from the Doris community would be
>>> greatly
>>> >> appreciated. Your expertise and guidance will be instrumental in
>>> helping me
>>> >> resolve this challenge.
>>> >>
>>> >> Thank you for your continued support, and I look forward to your
>>> valuable
>>> >> input.
>>> >>
>>> >> Best regards,
>>> >> Joice
>>> >>
>>> >> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <[email protected]>
>>> >> wrote:
>>> >>
>>> >>> Dear Doris Community,
>>> >>>
>>> >>> I am reaching out to the community to seek assistance with an
>>> >>> integration issue I've encountered while trying to use Hive on a CDP
>>> >>> distribution with Doris 2.0.1.1.
>>> >>>
>>> >>> Here are the details of my setup:
>>> >>>
>>> >>> Doris Version: 2.0.1.1
>>> >>> Hive Version: 3.1.3
>>> >>> Cluster Security: Kerberized
>>> >>>
>>> >>> I have successfully created a Hive catalog in Doris and have been
>>> able
>>> >>> to set up the integration between Hive and Doris. However, when I
>>> attempt
>>> >>> to query a Hive table using Doris, I encounter the following error:
>>> >>>
>>> >>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split
>>> failed
>>> >>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned
>>> full-acid
>>> >>> tables need to run major compaction.
>>> >>>
>>> >>> I have already performed a major compaction as recommended, but I am
>>> >>> still encountering the same error.
>>> >>>
>>> >>> I would greatly appreciate any insights, guidance, or solutions that
>>> the
>>> >>> Doris community can offer to help me resolve this issue. If anyone
>>> has
>>> >>> encountered a similar problem or has expertise in integrating Hive
>>> with
>>> >>> Doris, your assistance would be invaluable.
>>> >>>
>>> >>> Thank you in advance for your time and support. I look forward to
>>> >>> hearing from the community and working together to find a solution
>>> to this
>>> >>> challenge.
>>> >>>
>>> >>> [image: baby_table.JPG]
>>> >>> [image: doris_hive_catlog_result.JPG]
>>> >>>
>>> >>> [image: doris_fe_log.JPG]
>>> >>>
>>> >>> [image: hive_metastore_error.JPG]
>>> >>>
>>> >>> Thanks
>>> >>> Joice
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>>
>>> --
>>> 张家峰
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>
>
DESCRIBE FORMATTED baby;
+-------------------------------+----------------------------------------------------+-----------------------------+
| col_name | data_type
| comment |
+-------------------------------+----------------------------------------------------+-----------------------------+
| id | int
| |
| fname | varchar(50)
| |
| gender | varchar(2)
| |
| totcount | int
| |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name | data_type
| comment |
| data_id | int
| |
| | NULL
| NULL |
| # Detailed Table Information | NULL
| NULL |
| Database: | default
| NULL |
| OwnerType: | USER
| NULL |
| Owner: | hive
| NULL |
| CreateTime: | Wed Oct 04 08:38:14 UTC 2023
| NULL |
| LastAccessTime: | UNKNOWN
| NULL |
| Retention: | 0
| NULL |
| Location: |
hdfs://instance-3:8020/warehouse/tablespace/managed/hive/baby | NULL
|
| Table Type: | MANAGED_TABLE
| NULL |
| Table Parameters: | NULL
| NULL |
| | COLUMN_STATS_ACCURATE
| {\"BASIC_STATS\":\"true\"} |
| | bucketing_version
| 2 |
| | numFiles
| 10 |
| | numPartitions
| 10 |
| | numRows
| 10 |
| | orc.compress
| snappy |
| | rawDataSize
| 0 |
| | totalSize
| 10482 |
| | transactional
| true |
| | transactional_properties
| default |
| | transient_lastDdlTime
| 1696408694 |
| | NULL
| NULL |
| # Storage Information | NULL
| NULL |
| SerDe Library: | org.apache.hadoop.hive.ql.io.orc.OrcSerde
| NULL |
| InputFormat: |
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL
|
| OutputFormat: |
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL
|
| Compressed: | No
| NULL |
| Num Buckets: | -1
| NULL |
| Bucket Columns: | []
| NULL |
| Sort Columns: | []
| NULL |
+-------------------------------+----------------------------------------------------+-----------------------------+
39 rows selected (0.499 seconds)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
