parisni opened a new issue #4723: URL: https://github.com/apache/hudi/issues/4723
hudi 0.10.1 / aws glue metastore so far, glue does not suport the syntax used by hudi to make schema evolution `alter table replace colums ... cascade` As a result, when a hudi table schema evolve, the glue metastore fail to be updated at this step. The issue has been marked as resolved in the bellow jira, which, from my experience is not the case. The [HiveSyncTool](https://github.com/apache/hudi/blob/4f6cdd73a352bee132213438e6d11a8aef56de8b/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java#L184-L197) code start by changing the schema and then add the partition. Workaround I can think of : - when a schema evolution exist, drop/recreate the schema from scratch in glue (last resort) - start by adding the partitions and then change the schema so that glue data get added, without schema evolution (so that user can get the data with the old schema) - implement the schema DDL evolution differently (there might be a way without cascade ?) - Provide a GlueSyncTool to overwrite the default way "hoodie.meta.sync.classes": "org.apache.hudi.hive.HiveSyncTool" related: - https://issues.apache.org/jira/browse/HUDI-874 - https://github.com/apache/hudi/issues/1856 - https://github.com/apache/hudi/issues/1581 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
