vinothchandar commented on issue #954: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: <hivedb.tableName> table not found URL: https://github.com/apache/incubator-hudi/issues/954#issuecomment-542298488 What I see is that it created the table, ``` 19/10/15 03:05:38 INFO HiveSyncTool: Table hudi_gwpl_pc_policy is not found. Creating it 19/10/15 03:05:38 INFO HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE IF NOT EXISTS uat_hoodie_staging.hudi_gwpl_pc_policy( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `deleteTime` bigint, `NewProducerCode_Ext` bigint, `DoNotPurge` boolean, `PublicID` string, `PriorPremiums` string, `IssueDate` bigint, `PriorPremiums_cur` int, `MovedPolicySourceAccountID` bigint, `AccountID` bigint, `CreateTime` bigint, `LossHistoryType` int, `ExcludedFromArchive` boolean, `ArchiveState` int, `ArchiveSchemaInfo` bigint, `ArchiveFailureDetailsID` bigint, `PackageRisk` int, `NumPriorLosses` int, `UpdateTime` bigint, `PrimaryLanguage` int, `DoNotArchive` boolean, `ID` bigint, `PrimaryLocale` int, `ProductCode` string, `ExcludeReason` string, `CreateUserID` bigint, `ArchiveFailureID` bigint, `OriginalEffectiveDate` bigint, `BeanVersion` int, `ArchivePartition` bigint, `Retired` bigint, `LossHistoryType_Ext` int, `UpdateUserID` bigint, `PriorTotalIncurred` string, `ArchiveDate` bigint, `PriorTotalIncurred_cur` int, `ProducerCodeOfServiceID` bigint, `UL_BOPEligibility_Ext` boolean, `isDmvReported` boolean, `ClueStatusExt` boolean, `LossHistoryTypeComm_Ext` int, `ClueStatusDetail` bigint, `uniqueId` string, `pctl_archivestate_typecode` string, `pctl_archivestate_name` string, `pctl_archivestate_description` string, `pctl_losshistorytype_typecode2` string, `pctl_losshistorytype_name2` string, `pctl_losshistorytype_description2` string, `pctl_losshistorytype_typecode1` string, `pctl_losshistorytype_name1` string, `pctl_losshistorytype_description1` string, `pctl_losshistorytype_ext_typecode` string, `pctl_losshistorytype_ext_name` string, `pctl_losshistorytype_ext_description` string, `pctl_packagerisk_typecode` string, `pctl_packagerisk_name` string, `pctl_packagerisk_description` string, `pctl_languagetype_typecode` string, `pctl_languagetype_name` string, `pctl_languagetype_description` string, `pctl_localetype_typecode` string, `pctl_localetype_name` string, `pctl_localetype_description` string, `pctl_currency_typecode1` string, `pctl_currency_name1` string, `pctl_currency_description1` string, `pctl_currency_typecode2` string, `pctl_currency_name2` string, `pctl_currency_description2` string, `ingestiondt` string) PARTITIONED BY (`batch` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 's3://sa-l3-uat-emr-edl-processed/staging/hoodie/pc_policy' ``` Found the 1 storage partition (i.e from your data on s3) to sync ``` 19/10/15 03:05:41 INFO HiveSyncTool: Schema sync complete. Syncing partitions for hudi_gwpl_pc_policy 19/10/15 03:05:41 INFO HiveSyncTool: Last commit time synced was found to be null 19/10/15 03:05:41 INFO HoodieHiveClient: Last commit time synced is not known, listing all partitions in s3://sa-l3-uat-emr-edl-processed/staging/hoodie/pc_policy,FS :com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem@36c6ad3c 19/10/15 03:05:41 INFO HiveSyncTool: Storage partitions scan complete. Found 1 ``` But cannot find the table when trying to sync them ``` org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_gwpl_pc_policy at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:172) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:107) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:67) ... Caused by: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: uat_hoodie_staging.hudi_gwpl_pc_policy table not found ... at org.apache.hudi.org.apache.hadoop_hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1162) at org.apache.hudi.hive.HoodieHiveClient.scanTablePartitions(HoodieHiveClient.java:240) at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:162) ``` It almost seems like your metastore is not providing read-after-write consistency? what is the Hive metastore backed by, s3? I am guessing glue catalog is different from Hive metastore? Could you give it a shot on EMR with Hive metastore?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
