chrischnweiss opened a new issue #4585: URL: https://github.com/apache/hudi/issues/4585
Hi guys, we ran into a problem setting the target schema of our Hudi table using the MultiTableDeltaStreamer. Using a normal DeltaStreamer, we are able to set our source and target schemas using the properties: - hoodie.deltastreamer.schemaprovider.registry.url - hoodie.deltastreamer.schemaprovider.registry.targetUrl We found that we are not able to set these properties on a table basis using the MultiTableDeltaStreamer, since the MTDS builds SchemaRegistry URLs for target and source schema using the properties: - hoodie.deltastreamer.schemaprovider.registry.baseUrl - hoodie.deltastreamer.schemaprovider.registry.sourceUrlSuffix - hoodie.deltastreamer.schemaprovider.registry.targetUrlSuffix Later the MultiTableDeltaStreamer uses the source Kafka Topic name also for setting the name of the target schema: https://github.com/apache/hudi/blob/9fe28e56b49c7bf68ae2d83bfe89755314aa793b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java#L167 We think, that schema names should be more configurable, like the origin DeltaStreamer would handle it. Actually the names of the schemas you want to use for reading or writing the data are very tight coupled to the name of the Kafka topic the data is loaded from. What did you think? Cheers, Christian -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
