nsivabalan commented on code in PR #8053:
URL: https://github.com/apache/hudi/pull/8053#discussion_r1119186983
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -75,26 +87,32 @@ public class BigQuerySyncConfig extends HoodieSyncConfig
implements Serializable
public static final ConfigProperty<String> BIGQUERY_SYNC_SYNC_BASE_PATH =
ConfigProperty
.key("hoodie.gcp.bigquery.sync.base_path")
.noDefaultValue()
+ .withInferFunction(cfg ->
Option.ofNullable(cfg.getString(META_SYNC_BASE_PATH)))
.withDocumentation("Base path of the hoodie table to sync");
public static final ConfigProperty<String> BIGQUERY_SYNC_PARTITION_FIELDS =
ConfigProperty
.key("hoodie.gcp.bigquery.sync.partition_fields")
.noDefaultValue()
+ .withInferFunction(cfg ->
Option.ofNullable(cfg.getString(HoodieTableConfig.PARTITION_FIELDS))
+ .or(() ->
Option.ofNullable(cfg.getString(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME))))
.withDocumentation("Comma-delimited partition fields. Default to
non-partitioned.");
public static final ConfigProperty<Boolean>
BIGQUERY_SYNC_USE_FILE_LISTING_FROM_METADATA = ConfigProperty
.key("hoodie.gcp.bigquery.sync.use_file_listing_from_metadata")
- .defaultValue(false)
+ .defaultValue(DEFAULT_METADATA_ENABLE_FOR_READERS)
+ .withInferFunction(cfg ->
Option.of(cfg.getBooleanOrDefault(HoodieMetadataConfig.ENABLE,
DEFAULT_METADATA_ENABLE_FOR_READERS)))
Review Comment:
not sure if this is right. we are fetching from write config to deduce the
read value for enabling metadata.
##########
hudi-gcp/src/main/java/org/apache/hudi/gcp/bigquery/BigQuerySyncConfig.java:
##########
@@ -75,26 +87,32 @@ public class BigQuerySyncConfig extends HoodieSyncConfig
implements Serializable
public static final ConfigProperty<String> BIGQUERY_SYNC_SYNC_BASE_PATH =
ConfigProperty
.key("hoodie.gcp.bigquery.sync.base_path")
.noDefaultValue()
+ .withInferFunction(cfg ->
Option.ofNullable(cfg.getString(META_SYNC_BASE_PATH)))
.withDocumentation("Base path of the hoodie table to sync");
public static final ConfigProperty<String> BIGQUERY_SYNC_PARTITION_FIELDS =
ConfigProperty
.key("hoodie.gcp.bigquery.sync.partition_fields")
.noDefaultValue()
+ .withInferFunction(cfg ->
Option.ofNullable(cfg.getString(HoodieTableConfig.PARTITION_FIELDS))
Review Comment:
Incase of custom key generator, I guess format of
tableConfig.PARTITION_FIELDS and KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME
might differ.
for eg
simple:col1
vs col1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
