yikf commented on code in PR #7622:
URL: https://github.com/apache/kyuubi/pull/7622#discussion_r3757278802
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala:
##########
@@ -66,8 +67,14 @@ case class HiveWrite(
private val tableLocation = hiveTable.getDataLocation
private val allColumns = info.schema().toAttributes
- private val dataColumns = allColumns.take(allColumns.length -
hiveTable.getPartCols.size())
- private val partColumns = allColumns.takeRight(hiveTable.getPartCols.size())
+ // Split data and partition columns by name rather than by position. Users
can create a
+ // partitioned table whose partition columns are not the trailing columns of
the schema
+ // (e.g. df.write.partitionBy("mid_col")), so a positional take/takeRight
would mis-classify
+ // columns and lead the metastore to reject the written partition spec. See
KYUUBI #6403.
+ private val dataColumns: Seq[AttributeReference] =
+ allColumns.filterNot(col => table.partitionColumnNames.contains(col.name))
+ private val partColumns: Seq[AttributeReference] =
+ allColumns.filter(col => table.partitionColumnNames.contains(col.name))
Review Comment:
Good catch - it didn't.
- Fixed by matching partition columns with the session resolver and
lower-casing the written partition dir names, same as Spark's
`V1WritesHiveUtils` (SPARK-28054).
- Added a `partitionBy("FavoriteColor")` regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
