[
https://issues.apache.org/jira/browse/HIVE-25871?focusedWorklogId=712874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-712874
]
ASF GitHub Bot logged work on HIVE-25871:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Jan/22 13:45
Start Date: 21/Jan/22 13:45
Worklog Time Spent: 10m
Work Description: boroknagyz commented on a change in pull request #2948:
URL: https://github.com/apache/hive/pull/2948#discussion_r789669255
##########
File path:
iceberg/iceberg-handler/src/test/queries/positive/describe_iceberg_table.q
##########
@@ -8,7 +8,7 @@ DROP TABLE IF EXISTS ice_t_transform;
CREATE EXTERNAL TABLE ice_t_transform (year_field date, month_field date,
day_field date, hour_field timestamp, truncate_field string, bucket_field int,
identity_field int) PARTITIONED BY SPEC (year(year_field), month(month_field),
day(day_field), hour(hour_field), truncate(2, truncate_field), bucket(2,
bucket_field), identity_field) STORED BY ICEBERG;
DROP TABLE IF EXISTS ice_t_transform_prop;
-CREATE EXTERNAL TABLE ice_t_transform_prop (id int, year_field date,
month_field date, day_field date, hour_field timestamp, truncate_field string,
bucket_field int, identity_field int) STORED BY ICEBERG TBLPROPERTIES
('iceberg.mr.table.partition.spec'='{"spec-id":0,"fields":[{"name":"year_field_year","transform":"year","source-id":1,"field-id":1000},{"name":"month_field_month","transform":"month","source-id":2,"field-id":1001},{"name":"day_field_day","transform":"day","source-id":3,"field-id":1002},{"name":"hour_field_hour","transform":"hour","source-id":4,"field-id":1003},{"name":"truncate_field_trunc","transform":"truncate[2]","source-id":5,"field-id":1004},{"name":"bucket_field_bucket","transform":"bucket[2]","source-id":6,"field-id":1005},{"name":"identity_field","transform":"identity","source-id":7,"field-id":1006}]}');
+CREATE EXTERNAL TABLE ice_t_transform_prop (id int, year_field date,
month_field date, day_field date, hour_field timestamp, truncate_field string,
bucket_field int, identity_field int) STORED BY ICEBERG TBLPROPERTIES
('iceberg.mr.table.partition.spec'='{"spec-id":0,"fields":[{"name":"year_field_year","transform":"year","source-id":2,"field-id":1000},{"name":"month_field_month","transform":"month","source-id":3,"field-id":1001},{"name":"day_field_day","transform":"day","source-id":4,"field-id":1002},{"name":"hour_field_hour","transform":"hour","source-id":5,"field-id":1003},{"name":"truncate_field_trunc","transform":"truncate[2]","source-id":6,"field-id":1004},{"name":"bucket_field_bucket","transform":"bucket[2]","source-id":7,"field-id":1005},{"name":"identity_field","transform":"identity","source-id":8,"field-id":1006}]}');
Review comment:
Prior to this patch `HiveSchemaConverter` used 0-based indexing when it
assigned the field ids. E.g. in the above statement it would assign field id 0
to `id`, field id 1 to `year_field`, and so on. Hence in
'iceberg.mr.table.partition.spec' the source-id 1 referred to the `year_field`.
Everything was fine, but when Iceberg creates a table it reassigns the field
ids using 1-based indexing (field id 1 is `id`, field id 2 is `year_field`).
And Iceberg is smart enough to use the correct ids in the partition spec, i.e.
it replaces source id 1 to source id 2 and so on.
So everything worked OK, but you had to specify different field/source ids
in Hive than the actual field/source ids assigned by Iceberg.
With this change, you need to use the same 1-based indexing in the partition
spec that Iceberg will use later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 712874)
Time Spent: 1h 20m (was: 1h 10m)
> Hive should set name mapping table property for migrated Iceberg tables
> -----------------------------------------------------------------------
>
> Key: HIVE-25871
> URL: https://issues.apache.org/jira/browse/HIVE-25871
> Project: Hive
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Hive should set the name-mapping table property during table migration.
> It would be useful for [column
> projection|https://iceberg.apache.org/#spec/#column-projection] for files
> without field ids.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
