[I] [Bug] [Connector v2] Sink to iceberg MinIO; but getting org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO [seatunnel]

[via GitHub]

larryloi opened a new issue, #8585:
URL: https://github.com/apache/seatunnel/issues/8585

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### What happened
   
   Iceberg sink Connector got error to spark-iceberg
   
    Cannot initialize FileIO, org.apache.iceberg.aws.s3.S3FileIO does not 
implement FileIO
   
   === With below spark iceberg environment
   
   spark iceberg environment from 
https://github.com/tabular-io/docker-spark-iceberg 
   
   ENV SPARK_VERSION=3.5.2
   ENV SPARK_MAJOR_VERSION=3.5
   ENV ICEBERG_VERSION=1.6.0
   
   seatunnel environment build with below docker compose
       image: apache/seatunnel:2.3.9
       container_name: seatunnel-master
       environment:
         - TZ=Asia/Macau
         - ST_DOCKER_MEMBER_LIST=172.19.0.11,172.19.0.12,172.19.0.13
       entrypoint: >
         /bin/sh -c "
         /opt/seatunnel/bin/seatunnel-cluster.sh -r master -DJvmOption=\"-Xms2G 
-Xmx2G\"
         "    
       ports:
         - "5701:5701"
       volumes:
         - ./config:/opt/seatunnel/config
       networks:
         integration:
           ipv4_address: 172.19.0.11
   
   (2 worker nodes are the same)
   
   
   
   ### SeaTunnel Version
   
   tried 2.3.8 and 2.3.9 docker images
   
   ### SeaTunnel Config
   
   ```conf
   env {
     job.name = "mssql_starrocks.inventory.INV.orders__ODS_startup_latest"
     parallelism = 1
     job.mode = "STREAMING"
     checkpoint.interval = 5000
   }
   
   source {
     SqlServer-CDC {
       base-url = "jdbc:sqlserver://db01:1433;databaseName=inventory"
       plugin_output = "orders_demo00"
       username = "seatunnel_src"
       password = ""
       startup.mode="latest"
       database-names = ["inventory"]
       table-names = ["inventory.INV.orders_demo00"]
       table-names-config = [
         {
           table = "inventory.INV.orders_demo00"
           primaryKeys = ["id"]
         }
       ]
     }
   }
   
   
   sink {
     Iceberg {
       catalog_name = "ods"
       iceberg.catalog.config = {
         "type" = "rest"
         "uri" = "http://iceberg-rest:8181";
         "warehouse" = "s3://warehouse/"
         "io-impl" = "org.apache.iceberg.aws.s3.S3FileIO"
         "s3.endpoint" = "http://minio:9000";
         "s3.access-key-id" = "admin"
         "s3.secret-access-key" = "password"
         "s3.path-style-access" = true
       }
       namespace = "ods_namespace"
       table = "iceberg_sink_orders_demo00"
       iceberg.table.write-props = {
         write.format.default = "parquet"
         write.target-file-size-bytes = 536870912
       }
       iceberg.table.primary-keys = ["id"]
       iceberg.table.partition-keys = ["f_datetime"]
       iceberg.table.upsert-mode-enabled = true
       iceberg.table.schema-evolution-enabled = true
       case_sensitive = true
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh --config 
./JOBS/mssql_iceberg/inventory.INV.orders__ODS_startup_latest.yaml
   ```
   
   ### Error Exception
   
   ```log
   === But got the below error, and already tried different verisons of 
iceberg-aws-bundle-<version>.jar and 
iceberg-spark-runtime-<SPARK_MAJOR_VERSION>_2.12-<ICEBERG_VERSION>.jar. but 
also get the same problem. is it FileIO did not implement in 
org.apache.iceberg.aws.s3.S3FileIO  or org.apache.iceberg.io.FileIO ????
   
   Caused by: java.lang.IllegalArgumentException: Cannot initialize FileIO, 
org.apache.iceberg.aws.s3.S3FileIO does not implement FileIO.
           at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:320)
           at 
org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:827)
           at 
org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:204)
           at 
org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:72)
           at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:239)
           at 
org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:284)
           at 
org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergCatalogLoader.loadCatalog(IcebergCatalogLoader.java:61)
           at 
org.apache.seatunnel.connectors.seatunnel.iceberg.catalog.IcebergCatalog.open(IcebergCatalog.java:91)
           at 
org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:78)
           at 
org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:523)
           ... 21 more
   Caused by: java.lang.ClassCastException: org.apache.iceberg.aws.s3.S3FileIO 
cannot be cast to org.apache.iceberg.io.FileIO
           at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:317)
           ... 30 more
   ```
   
   ### Zeta or Flink or Spark Version
   
   ENV SPARK_VERSION=3.5.2
   ENV SPARK_MAJOR_VERSION=3.5
   ENV ICEBERG_VERSION=1.6.0
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org