[GitHub] [hudi] nsivabalan commented on a diff in pull request #5294: [HUDI-3838] Implemented drop partition column feature for delta streamer code path

[GitBox]

nsivabalan commented on code in PR #5294:
URL: https://github.com/apache/hudi/pull/5294#discussion_r848568345


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -478,13 +481,14 @@ private Pair<SchemaProvider, Pair<String, 
JavaRDD<HoodieRecord>>> fetchFromSourc
 
     boolean shouldCombine = cfg.filterDupes || 
cfg.operation.equals(WriteOperationType.UPSERT);
     JavaRDD<GenericRecord> avroRDD = avroRDDOptional.get();
-    JavaRDD<HoodieRecord> records = avroRDD.map(gr -> {
+    JavaRDD<HoodieRecord> records = avroRDD.map(record -> {
+      GenericRecord gr = isDropPartitionColumns() ? 
HoodieAvroUtils.removeFields(record, getPartitionColumns(keyGenerator, props)) 
: record;

Review Comment:
   getPartitionColumns(keyGenerator, props) could have been done once in the 
driver. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org