[
https://issues.apache.org/jira/browse/HIVE-23253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-23253:
---------------------------------
Fix Version/s: (was: 3.0.0)
> Synchronization between external SerDe schemas and Metastore
> ------------------------------------------------------------
>
> Key: HIVE-23253
> URL: https://issues.apache.org/jira/browse/HIVE-23253
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
> Affects Versions: 3.1.2
> Reporter: Attila Magyar
> Priority: Major
>
> In HIVE-15995 an ALTER <table> UPDATE COLUMNS statement was introduce to sync
> external SerDe schema changes with the metastore. This command can only be
> manually invoked.
> See it in the documentation.
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionUpdatecolumns
>
> Maybe it would make sense to run an update columns automatically in certain
> cases to prevent problems coming from cases where the user forgets running
> the update columns manually.
>
> One way to reproduce the issue is to change the schema url via an alter table
> statement.
> {code:java}
> [root@c7401 vagrant]# cat test_schema1.avsc
> {
> "type":"record",
> "name":"test_schema",
> "namespace":"gdc_datascience_qa",
> "fields":[
> {
> "name":"name",
> "type":[
> "null",
> "string"
> ],
> "default":null
> }
> ]
> }[root@c7401 vagrant]# cat test_schema2.avsc
> {
> "type":"record",
> "name":"test_schema",
> "namespace":"gdc_datascience_qa",
> "fields":[
> {
> "name":"name",
> "type":[
> "null",
> "string"
> ],
> "default":null
> },
> {
> "name":"last_name",
> "type":[
> "null",
> "string"
> ],
> "default":null
> }
> ]
> }
> {code}
> {code:java}
> $ hadoop fs -copyFromLocal *.avsc /tmp/
> [beeline] create external table t1 stored as avro tblproperties
> ('avro.schema.url'='/tmp/test_schema1.avsc');
> [beeline] alter table t1 set
> tblproperties('avro.schema.url'='/tmp/test_schema2.avsc');
> [beeline] insert into t1 values ('n1', 'l1');
> [beeline] create external table t2 stored as avro tblproperties
> ('avro.schema.url'='/tmp/test_schema2.avsc');
> [beeline] insert into t2 values ('n2', 'l2');
> [beeline] insert overwrite table t1 select * from t2; {code}
> Error:
> {code:java}
> MetaException(message:Column last_name doesn't exist in table t1 in database
> default)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:8652)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:8602)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionColStats(ObjectStore.java:8416)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updateTableColumnStatistics(ObjectStore.java:8446
> {code}
> Running an ALTER UPDATE COLUMNS fixes the problem.
>
> cc: [~szita]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
