omlomloml opened a new issue, #5698: URL: https://github.com/apache/hudi/issues/5698
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** A clear and concise description of the problem. **To Reproduce** Steps to reproduce the behavior: 1. Build hudi 0.11 with mvn clean package -DskipTests -Dspark3.2 2. go to hudi/hudi-sync/hudi-hive-sync run hudi sync with hms, something like /run_sync_tool.sh --database xx --table yyy --metastore-uris thrift://hive-metastore:9083 --base-path s3a://test-hudi/yyy/ --sync-mode hms 3.no matter I use hive-sync bundle or utility bundle, it failed to get table schema 4. **Expected behavior** expect meta data sync through hms to hive metadata store **Environment Description** * Hudi version : 0.11.0 * Spark version : 3.2.1 * Hive version : 3.1 * Hadoop version : 3.2 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : yes in EKS **Additional context** Add any other context about the problem here. root@hudi-cli-59c5dd55f-4shnr:/opt/hudi/hudi-sync/hudi-hive-sync# ./run_sync_tool.sh --database gpr --table runs --metastore-uris thrift://hive-metastore:9083 --base-path s3a://wavesense-test-hudi/runs/ --sync-mode hms setting hadoop conf dir Running Command : java -cp /opt/hive/lib/hive-metastore-3.1.3.jar::/opt/hive/lib/hive-service-3.1.3.jar::/opt/hive/lib/hive-exec-3.1.3.jar::/opt/hive/lib/hive-jdbc-3.1.3.jar:/opt/hive/lib/hive-jdbc-handler-3.1.3.jar::/opt/hive/lib/jackson-annotations-2.12.0.jar:/opt/hive/lib/jackson-core-2.12.0.jar:/opt/hive/lib/jackson-core-asl-1.9.13.jar:/opt/hive/lib/jackson-databind-2.12.0.jar:/opt/hive/lib/jackson-dataformat-smile-2.12.0.jar:/opt/hive/lib/jackson-mapper-asl-1.9.13.jar:/opt/hive/lib/jackson-module-scala_2.11-2.12.0.jar::/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/etc/hadoop:/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.11.0.jar org.apache.hudi.hive.HiveSyncTool --database gpr --table runs --metastore-uris thrift://hive-metastore:9083 --base-path s3a://wavesense-test-hudi/r uns/ --sync-mode hms 1 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 198 [main] INFO org.apache.hadoop.metrics2.impl.MetricsConfig - Loaded properties from hadoop-metrics2.properties 253 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s). 253 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics system started 1372 [main] INFO org.apache.hadoop.hive.conf.HiveConf - Found configuration file null 1552 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading HoodieTableMetaClient from s3a://wavesense-test-hudi/runs/ 1700 [main] INFO org.apache.hudi.common.table.HoodieTableConfig - Loading table properties from s3a://wavesense-test-hudi/runs/.hoodie/hoodie.properties 1749 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://wavesense-test-hudi/runs/ 1749 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient - Loading Active commit timeline for s3a://wavesense-test-hudi/runs/ 1878 [main] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20220524201523059__clean__COMPLETED]} 2291 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Trying to connect to metastore with URI thrift://hive-metastore.default.svc.cluster.local:9083 2308 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Opened a connection to metastore, current connections: 1 2327 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Connected to metastore. 2327 [main] INFO org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0 2441 [main] INFO org.apache.hudi.hive.HiveSyncTool - Syncing target hoodie table with hive table(runs). Hive metastore URL :jdbc:hive2://localhost:10000, basePath :s3a://wavesense-test-hudi/runs/ 2441 [main] INFO org.apache.hudi.hive.HiveSyncTool - Trying to sync hoodie table runs with base path s3a://wavesense-test-hudi/runs of type COPY_ON_WRITE 2737 [main] INFO org.apache.hudi.common.table.TableSchemaResolver - Reading schema from s3a://wavesense-test-hudi/runs/2022-02/61d91e74-8331-492c-89ce-4ae7792cd8cc-0_0-13934-418302_20220524201445667.parquet 2832 [main] INFO org.apache.hadoop.fs.s3a.S3AInputStream - Switching to Random IO seek policy 3097 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Closed a connection to metastore, current connections: 0 Exception in thread "main" java.lang.NoSuchMethodError: org.apache.parquet.schema.Type.getLogicalTypeAnnotation()Lorg/apache/parquet/schema/LogicalTypeAnnotation; at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:296) at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:275) at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:264) at org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:341) at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:148) at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:565) at org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:82) at org.apache.hudi.sync.common.AbstractSyncHoodieClient.getDataSchema(AbstractSyncHoodieClient.java:164) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:204) at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:150) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:138) at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:433) 3101 [shutdown-hook-0] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Stopping s3a-file-system metrics system... 3101 [shutdown-hook-0] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics system stopped. 3102 [shutdown-hook-0] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - s3a-file-system metrics system shutdown complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
