aokolnychyi commented on code in PR #50246:
URL: https://github.com/apache/spark/pull/50246#discussion_r2023825620
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteRowLevelCommand.scala:
##########
@@ -273,9 +273,8 @@ trait RewriteRowLevelCommand extends Rule[LogicalPlan] {
outputs: Seq[Seq[Expression]],
colOrdinals: Seq[Int],
attrs: Seq[Attribute]): ProjectingInternalRow = {
- val schema = StructType(attrs.zipWithIndex.map { case (attr, index) =>
- val nullable = outputs.exists(output =>
output(colOrdinals(index)).nullable)
- StructField(attr.name, attr.dataType, nullable, attr.metadata)
+ val schema = StructType(attrs.zipWithIndex.map { case (attr, _) =>
+ StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
Review Comment:
@huaxingao, I think the Spark behavior here is correct but Iceberg would
need to relax its check.
PR #49493 added a notion of reinsert to `DeltaWriter` to support row
lineage. Iceberg leverages reinserts. Previously, Spark never passed `metadata`
with reinsert and the metadata attributes preserved its nullability. This is no
longer the case in 4.0. Spark now passes `metadata` with reinsert and the
metadata attributes are actually nullified. Therefore, Spark seems to pass a
correct schema info as `metadata` attributes are now nullable.
```
/**
* Reinserts a row with metadata.
* <p>
* This method handles the insert portion of updated rows split into
deletes and inserts.
*
* @param metadata values for metadata columns
* @param row a row to reinsert
* @throws IOException if failure happens during disk/network IO like
writing files
*
* @since 4.0.0
*/
default void reinsert(T metadata, T row) throws IOException {
insert(row);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
