wombatu-kun commented on code in PR #20041:
URL: https://github.com/apache/hudi/pull/20041#discussion_r4089365879
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRecordContext.java:
##########
@@ -214,10 +215,30 @@ public InternalRow toBinaryRow(HoodieSchema schema,
InternalRow internalRow) {
return unsafeProjection.apply(internalRow);
}
+ /**
+ * Installs the Spark type the rows of an engine schema actually carry in
this read; null, the default, means the
+ * plain conversion. {@code SparkFileFormatInternalRowReaderContext}
installs the PushVariantIntoScan overlay here
+ * (see its setSchemaHandler), because every row writer this context builds
from an engine schema has to be typed
+ * over that shape: a VariantType-typed writer re-encodes a projection
struct through UnsafeRow.getVariant instead
+ * of copying it across.
+ */
+ public void setRowShape(UnaryOperator<StructType> rowShape) {
+ this.rowShape = rowShape;
+ }
+
+ /**
+ * The Spark type the rows of {@code schema} carry in this read: the plain
conversion, or the row shape installed
+ * by {@link #setRowShape} when the reader hands its rows over in a
rewritten shape.
+ */
+ public StructType getRowStructType(HoodieSchema schema) {
+ StructType structType = getCachedSchema(schema);
+ return rowShape == null ? structType : rowShape.apply(structType);
+ }
+
@Override
public UnaryOperator<InternalRow> projectRecord(HoodieSchema from,
HoodieSchema to, Map<String, String> renamedColumns) {
Function1<InternalRow, UnsafeRow> unsafeRowWriter =
- HoodieInternalRowUtils.getCachedUnsafeRowWriter(getCachedSchema(from),
getCachedSchema(to), renamedColumns, Collections.emptyMap());
+
HoodieInternalRowUtils.getCachedUnsafeRowWriter(getRowStructType(from),
getRowStructType(to), renamedColumns, Collections.emptyMap());
Review Comment:
The shape now types `projectRecord`, but not the other engine-schema
accessors in this class (`getValue`, `toBinaryRow`) or
`SparkRecordMergingUtils.mergePartialRecords`, so a merge that builds a fresh
row puts a `VariantVal` where this writer now expects a struct. Does a one-path
`variant_get` on a MOR table now throw ClassCastException after a MERGE INTO
that updates only some columns of a base-file record, where it used to
round-trip?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
