leilinen opened a new issue, #13690: URL: https://github.com/apache/hudi/issues/13690
**Describe the problem you faced** We created a MOR table in Spark and writing records to it using Flink (YARN Application Mode). While Hudi is configured to sync metadata to the Hive Metastore after each checkpoint, this operation fails with the exception shown below. Although we've placed (using -Dyarn.ship-files) the hive-site.xml file in the YARN container's root path (and Flink logs confirm reading Thrift configurations), the sync still fails. We also tested the same process and it successful when using Flink YARN Session Mode with the --ship parameter to deploy hive-site.xml. <img width="1532" height="788" alt="Image" src="https://github.com/user-attachments/assets/8bc13ad4-6502-4cb2-ab1c-3337db4d6d31"; /> The table DDL in spark below > CREATE TABLE hudi_test.hudi_oc_contract_goods_all_0725( CONTRACT_GOODS_ID BIGINT, CONTRACT_GOODS_CODE STRING, CONTRACT_BILLCODE STRING, SKU_CODE STRING, SKU_NAME STRING, SPU_CODE STRING, updated_at STRING, created_at STRING, GOODS_CAMOUNT DECIMAL(16, 6), CONTRACT_GOODS_MONEY DECIMAL(25, 8), CONTRACT_GOODS_INMONEY DECIMAL(25, 8), GOODS_NO STRING, PRICESET_NPRICE DECIMAL(25,8), TRAFFIC_CHANNEL STRING, MSCHANNEL_CODE STRING, DATA_STATE BIGINT, GOODS_CODE STRING, TENANT_CODE STRING, ts TIMESTAMP ) USING HUDI TBLPROPERTIES( 'hoodie.cleaner.policy' = 'KEEP_LATEST_BY_HOURS', 'hoodie.cleaner.hours.retained' = '72', 'hoodie.datasource.write.precombine.field' = 'updated_at', 'hoodie.datasource.write.recordkey.field' = 'CONTRACT_GOODS_ID', 'hoodie.datasource.write.table.type' = 'MERGE_ON_READ', 'hoodie.datasource.write.operation' = 'upsert', 'hoodie.datasource.meta.sync.enable' = 'true', 'hoodie.datasource.hive_sync.mode' = 'hms', 'hoodie.datasource.hive_sync.conf.dir'='/data1/hadoop/tools/spark/conf/hadoop-27', 'hoodie.datasource.hive_sync.database' = 'hudi_test_0622', 'hoodie.datasource.hive_sync.table' = 'hudi_oc_contract_goods_all_0725', 'changelog.enabled'='true' ); **Expected behavior** sync metadata to hive metastore in flink application mode successful **Environment Description** * Hudi version : 0.15.0 * Spark version : 3.3.1 * Hive version : 1.18.1 * Hadoop version : 3.1.4 * Storage (HDFS/S3/GCS..) : HDFS * Running on Docker? (yes/no) : no **Additional context** Add any other context about the problem here. **Stacktrace** ``` 2025-08-07 19:29:13,549 INFO hive.metastore [] - Trying to connect to metastore with URI thrift://hive36-132-192.bd:9083 2025-08-07 19:29:13,552 INFO hive.metastore [] - Opened a connection to metastore, current connections: 2 2025-08-07 19:29:13,612 WARN hive.metastore [] - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it. org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4247) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4233) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:338) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:163) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.getDatabase(Unknown Source) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.getDatabase(Unknown Source) ~[?:?] at org.apache.hudi.hive.HoodieHiveSyncClient.databaseExists(HoodieHiveSyncClient.java:277) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:242) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:206) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:180) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:343) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281] 2025-08-07 19:29:13,616 INFO hive.metastore [] - Connected to metastore. 2025-08-07 19:29:13,618 WARN org.apache.hudi.hive.HiveSyncTool [] - Unable to create database org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if database exists hudi_test at org.apache.hudi.hive.HoodieHiveSyncClient.databaseExists(HoodieHiveSyncClient.java:283) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:242) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:206) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:180) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:343) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281] Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:782) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:769) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1288) ~[hive-exec-2.3.9.jar:2.3.9] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.getDatabase(Unknown Source) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.getDatabase(Unknown Source) ~[?:?] at org.apache.hudi.hive.HoodieHiveSyncClient.databaseExists(HoodieHiveSyncClient.java:277) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] ... 8 more 2025-08-07 19:29:13,621 DEBUG hive.metastore [] - No current SessionState, skipping temp tables 2025-08-07 19:29:13,622 WARN org.apache.hadoop.hive.metastore.RetryingMetaStoreClient [] - MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. tableExists org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table(ThriftHiveMetastore.java:1508) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1499) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1455) ~[hive-exec-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266) ~[hive-exec-2.3.9.jar:2.3.9] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.tableExists(Unknown Source) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_281] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_281] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_281] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_281] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327) ~[hive-exec-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy67.tableExists(Unknown Source) ~[?:?] at org.apache.hudi.hive.HoodieHiveSyncClient.tableExists(HoodieHiveSyncClient.java:268) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:256) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:206) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:180) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:343) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.18-bundle-0.15.0.jar:0.15.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_281] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_281] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281] Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_281] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_281] at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_281] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_281] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_281] at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) ~[hive-exec-2.3.9.jar:2.3.9] ... 27 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
