sushant-searce commented on issue #12189: URL: https://github.com/apache/hudi/issues/12189#issuecomment-2451355732
As you can see the test cases I have shared above Pipeline was working yesterday after disabling and then enabling the hive_sync. Yesterday we disabled hive_sync in the pipeline and it ran successfully and enabled it again in next and that run successfully as well **but IN TODAYs run it FAILED with same error** It is very concerning is there anything we are missing here Sharinf Hoodie Options as well hudiOptions: `hoodie.cleaner.commits.retained: 10 hoodie.metadata.keep.max.commits: 30 hoodie.metadata.clean.async: false hoodie.keep.max.commits: 30 hoodie.metadata.keep.min.commits: 20 hoodie.archive.async: false hoodie.clean.automatic: true hoodie.finalize.write.parallelism: 200 hoodie.fail.on.timeline.archiving: false hoodie.clean.async: false hoodie.parquet.max.file.size: 128000000 hoodie.datasource.hive_sync.support_timestamp : true #DISABLING METADATA TO REDUCE FREQUENT CALLS TO GCS hoodie.metadata.enable: false hoodie.datasource.write.hive_style_partitioning : true hoodie.parquet.small.file.limit: 100000000 hoodie.datasource.hive_sync.enable: true hoodie.bulkinsert.shuffle.parallelism: 200 hoodie.keep.min.commits: 11 hoodie.datasource.meta.sync.enable: true hoodie.metadata.cleaner.commits.retained: 3 hoodie.cleaner.incremental.mode: true hoodie.commits.archival.batch: 12 hoodie.upsert.shuffle.parallelism: 200 hive_sync.support_timestamp: true hoodie.insert.shuffle.parallelism: 200 hoodie.metadata.compact.max.delta.commits: 10 compaction.delta_commits: 5 metadata.compaction.delta_commits: 10 hoodie.compact.inline.max.delta.commits: 5 hoodie.archive.automatic: true hoodie.cleaner.parallelism: 200` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
