HeadJk opened a new issue, #13880:
URL: https://github.com/apache/hudi/issues/13880
**Describe the problem you faced**
We are experiencing excessive S3 API calls and poor performance in our
Hudi-based datalake pipeline cdc application. Each spark executor in the
application is opening the 130 rollback files in S3 repeatedly, approximately
1000 times per rollback file per executor. This is causing S3 503 throttling
errors and significant performance degradation during our CDC data processing
pipeline.
**To Reproduce**
Steps to reproduce the behavior:
1. Set up a hudi table in s3 with approximately 130 rollback files
1. Run an EMR Serverlesss Spark application that upserts data into this
table using the configurations provided below
5. Observe S3 access patterns throgh Spark executor stderr.gz logs in Spark
History Server
**Expected behavior**
Rollback files data should be read efficiently with minimal redundant S3 API
calls. Each executor should not need to repeatedly list the same rollback files
1000+ times, and the application should not encounter S3 throttling errors due
to excessive s3 api operations. Rollback file data should likely be cached
after being read for the first time.
**Environment Description**
* Hudi version : 0.13.1-amzn-0
* Spark version : 3.4.0-amzn-0
* Hive version : 3.1.3-amzn-5
* Hadoop version : 3.3.3-amzn-4
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No (AWS EMR Serverless)
Hoodie configuration:
```
"hoodie.datasource.write.recordkey.field": "uuid",
"hoodie.datasource.write.keygenerator.class":
"org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.payload.class":
"org.apache.hudi.common.modelOverwriteWithLatestAvroPayload",
"hoodie.table.name": "*********",
"hoodie.datasource.write.table.type": "merge_on_read",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.schema.on.read.enable": "false",
"hoodie.datasource.write.reconcile.schema": "true",
"hoodie.datasource.write.precombine.field": "CaptureDate",
"hoodie.compact.inline": "true",
"hoodie.simple.index.parallelism": "1920",
"hoodie.global.simple.index.parallelism": "1920",
"hoodie.index.type": "global_bloom",
"hoodie.index.class":
"org.apache.hudi.index.bloom.HoodieSemiGlobalWarehouseBloomIndex",
"hoodie.datasource.write.partitionpath.field": "****,****,****",
"hoodie.datasource.hive_sync.partition_fields": "****,****,****",
"hoodie.upsert.shuffle.parallelism": "1920",
"hoodie.datasource.meta_sync.condition.sync": "true",
"hoodie.datasource.meta.sync.enable": "true",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.skip_ro_suffix": "true",
"hoodie.datasource.hive_sync.database": "p*******4",
"hoodie.datasource.hive_sync.table": "*******",
"hoodie.datasource.hive_sync.support_timestamp": "true",
"hoodie.meta.sync.client.tool.class":
"org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool",
"hoodie.metadata.enable": "true",
"hoodie.meta.sync.metadata_file_listing": "true",
"hoodie.archive.automatic": "true",
"hoodie.keep.min.commits": "200",
"hoodie.keep.max.commits": "210",
"hoodie.clean.automatic": "true",
"hoodie.cleaner.parallelism": "1920",
"hoodie.cleaner.policy": "KEEP_LATEST_COMMITS",
"hoodie.cleaner.commits.retained": "90",
"hoodie.parquet.max.file.size": "1073741824",
"hoodie.parquet.small.file.limit": "838860800",
"hoodie.write.concurrency.mode": "OPTIMISTIC_CONCURRENCY_CONTROL",
"hoodie.cleaner.policy.failed.writes": "LAZY",
"hoodie.write.lock.provider":
"org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
"hoodie.write.lock.dynamodb.table": "p*******r",
"hoodie.write.lock.dynamodb.partition_key": "*******",
"hoodie.write.lock.dynamodb.region": "us-east-1",
"hoodie.write.lock.dynamodb.endpoint_url":
"dynamodb.us-east-1.amazonaws.com",
"hoodie.write.lock.dynamodb.billing_mode": "PAY_PER_REQUEST"
```
AWS EMR Serverless
- EMR Release Label: emr-6.12.0
- Application Limits
Maximum CPU (vCPU): 2250
Maximum memory (GB): 17550
Maximum disk (GB): 112500
- Job Configurations
```
fs.s3.aimd.maxAttempts: 1000
fs.s3.maxRetries: 50
fs.s3.aimd.enabled: true
hive.metastore.client.factory.class:
com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.executor.memory: 26G
spark.driver.memory: 26G
spark.emr-serverless.driver.disk: 200G
spark.driver.maxResultSize: 5G
spark.driver.cores: 4
spark.emr-serverless.memoryOverheadFactor: 0.1
spark.hadoop.fs.s3.serverSideEncryption.kms.keyId: alias/********
spark.hadoop.fs.s3.enableServerSideEncryption: true
spark.hadoop.fs.s3.maxConnections: 9999
spark.emr-serverless.executor.disk: 200G
spark.hadoop.hive.metastore.client.factory.class:
com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
spark.serializer: org.apache.spark.serializer.KryoSerializer
spark.jars:
/usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/hudi/hudi-aws-bundle.jar
```
- Java Runtime: Java 17
**Additional context**
Hudi Table Size:
- The hudi table which we are upserting into has an estimated size of 10
million files and 20 TB's of data
Why Hudi Verion 0.13.1 is Required:
- The company I work for is strictly unable to change hudi versions for
various reasons. Updating to a later version of Hudi would update our table
version and break the reader applications that consume this data. We don't
control these reader applications and cannot risk breaking any of them by
requiring updates they may not implement. Changing our version of Hudi is
therefore not a possible solution for us.
Root Cause of Rollback Accumulation:
- The 130+ rollback files have accumulated due to occasional job failures
over the past year. Our archive retention settings (hoodie.keep.min.commits:
200, hoodie.keep.max.commits: 210) are intentionally configured to maintain
extensive commit history for downstream processing requirements. These settings
are necessary for our specific use case requirements where we need extensive
commit history for downstream processes. Additionally, we maintain this large
retention window to ensure the archiver never removes timeline entries before
the cleaner processes the corresponding commits, preventing potential data
consistency issues.
**Stacktrace**
Spark history server executor stderr.gz
```
25/09/09 15:03:35 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:35 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240807003441694.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240812203551496.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822014339510.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822015759845.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:36 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240822182728413.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240823030824469.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170803546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170807059.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170810335.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170813517.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170817281.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240827170820570.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240828135048938.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240830165641261.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150914282.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150925674.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150930546.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:37 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906150934788.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240906151203832.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909140753262.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240909184625794.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240910130154536.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240920171740272.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240921153540839.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240922182329367.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923121259866.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20240923173606554.rollback' for reading
25/09/09 15:03:38 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241005054221114.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241006133732033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241210190944069.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241213131144553.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164635772.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241217164639571.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180918757.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20241218180923701.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203510872.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203512529.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250220203514074.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250222083508448.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250306172823755.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250314180840033.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250507004949483.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250522083838588.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:39 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250528103012780.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214319692.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214342091.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214356578.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214423608.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214443938.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214501281.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214521140.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609234916106.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250611192638465.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214538229.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250611192815332.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250608214548343.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250611192821763.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609234916106.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123207319.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250611192827451.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250611192638465.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123339708.rollback' for reading
25/09/09 15:03:40 INFO S3NativeFileSystem: Opening
's3://********/******.all_hudi/.hoodie/20250609123302138.rollback' for reading
... continues this pattern consistently until 25/09/09 17:17:54, does so on
multiple spark executors
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
