JB-data edited a comment on issue #3905: URL: https://github.com/apache/hudi/issues/3905#issuecomment-971693314
Indeed, original issue clear. Thanks for feedback. Thanks for links: related to the SRC table-my confusion was: where does this SRC come from, what table is it? But after looking at the java code you suggested (https://github.com/a0x8o/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/SqlQueryBasedTransformer.java), this is I believe what happens: 1)WITH SQL TRANSFORMATION When you are using a transformation he will register a temporary table in hive in order to perform the SQL transformation. So the data from the kafka topic is pulled into hive for a bit (in the default db), the SQL transformation is performed, and then the data is written according to the target schema. This means we need to make sure that the user that is running the job has all permissions to access the hive metastore's default db as this is where he will temporarily register this <SRC> table, which in reality wil be called "HOODIE_SRC_TMP_TABLE_some_random_number". 2)WITHOUT SQL TRANSFORMATION While if you dont do any sql transformation, the data is directly landed in the target table and target db specified in the props. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
