maomaodev commented on code in PR #7622:
URL: https://github.com/apache/kyuubi/pull/7622#discussion_r3757515668
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala:
##########
@@ -66,8 +67,28 @@ case class HiveWrite(
private val tableLocation = hiveTable.getDataLocation
private val allColumns = info.schema().toAttributes
- private val dataColumns = allColumns.take(allColumns.length -
hiveTable.getPartCols.size())
- private val partColumns = allColumns.takeRight(hiveTable.getPartCols.size())
+ // Split data and partition columns by name rather than by position. Users
can create a
+ // partitioned table whose partition columns are not the trailing columns of
the schema
+ // (e.g. df.write.partitionBy("mid_col")), so a positional take/takeRight
would mis-classify
+ // columns and lead the metastore to reject the written partition spec.
Match names with the
+ // session resolver because the Hive metastore lower-cases partition column
names while the
+ // write schema preserves the user's original case. See KYUUBI #6403.
+ private val resolver = sparkSession.sessionState.conf.resolver
+ private def isPartitionColumn(name: String): Boolean =
+ table.partitionColumnNames.exists(resolver(_, name))
+ private val dataColumns: Seq[AttributeReference] =
+ allColumns.filterNot(col => isPartitionColumn(col.name))
+ // Keep the query's original column case here so requiredOrdering() resolves
against the query.
+ private val partColumns: Seq[AttributeReference] =
+ allColumns.filter(col => isPartitionColumn(col.name))
Review Comment:
Overall lgtm, nit: could we collapse the two passes into a single
`allColumns.partition(col => isPartitionColumn(col.name))`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
