yikf opened a new pull request, #7622: URL: https://github.com/apache/kyuubi/pull/7622
### Why are the changes needed? Fixes [#6403](https://github.com/apache/kyuubi/issues/6403). Writing a partitioned Hive table via the Kyuubi Spark Hive connector fails when the partition column is **not the last column** of the schema, e.g. `df.write.partitionBy("mid_col").saveAsTable(...)`. There are two root causes: 1. `HiveWrite` split data/partition columns **positionally** with `take`/`takeRight`, assuming partition columns are always trailing. When a partition column is in the middle, columns are mis-classified and the metastore rejects the written partition spec: ``` Table$ValidationFailureSemanticException: Partition spec {favorite_color=, favorite_numbers=2} contains non-partition columns ``` 2. The Hive metastore reorders partition columns to the end of the schema, but the CTAS query output keeps the original column order. Spark's `ResolveOutputRelation` resolves output columns **by position**, so a non-last partition column is mis-cast onto a trailing data column (`INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST`). This fixes both: - `HiveWrite` now splits data/partition columns **by name** against `table.partitionColumnNames` (also for the write `dataSchema` derivation). - `HiveTableCatalog.createTable` preserves the requested schema order on the returned table, so the subsequent CTAS write resolves output columns against the original order. Affects 1.8.1 and current `master`. ### How was this patch tested? Added a regression test in `HiveQuerySuite` that writes a table whose partition column is in the middle of the schema, using **mixed column types** (`String`/`String`/`Int`) so the schema-order bug is genuinely exercised (an all-`String` case would not surface the cast failure). Verified locally with `-Pspark-3.5`: - `HiveQuerySuite`: 21 tests passed (incl. the new case) - `HiveCatalogSuite`: 24 tests passed - `dev/reformat` (Spotless + Scalastyle) passed ### Was this patch authored or co-authored using generative AI tooling? Assisted-by: Claude:claude-opus-4-8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
