helong001 opened a new issue, #7395: URL: https://github.com/apache/gravitino/issues/7395
### Version main branch ### Describe what's wrong ## Problem Summary When deploying Gravitino on Kubernetes with the auxiliary Iceberg REST service enabled and using the dynamic-config-provider, everything starts up correctly. A custom catalog (e.g., testCatalog) is successfully added. However, when a client (such as Spark, Trino, or a REST client) attempts to list namespaces, the request unexpectedly uses default_catalog, which results in a failure. This occurs even if a valid catalog was already created and no such default_catalog was explicitly configured. ### Error message and/or stacktrace ## Error from web ui  ## Error log from gravitino pod with auxService iceberg-rest ```java 2025-06-15 08:08:30 INFO [Gravitino-webserver-36] CatalogOperations:227 - Catalog loaded: test.testCatalog 2025-06-15 08:08:35 INFO [iceberg-rest-25] IcebergNamespaceOperations:89 - List Iceberg namespaces, catalog: default_catalog, parentNamespace: 2025-06-15 08:08:35 INFO [iceberg-rest-25] IcebergExceptionMapper:80 - Iceberg REST server error maybe caused by user request, response http status: 400, exception: class java.lang.IllegalArgumentException, exception message: default_catalog is illegal in gravitino-based-provider 2025-06-15 08:08:35 WARN [Gravitino-webserver-42] ExceptionHandlers:267 - Failed to operate schema(s) operation [LIST] under catalog [testCatalog], reason [default_catalog is illegal in gravitino-based-provider] java.lang.IllegalArgumentException: default_catalog is illegal in gravitino-based-provider at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:206) ~[?:?] at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:180) ~[?:?] at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:166) ~[?:?] at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211) ~[?:?] at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323) ~[?:?] at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262) ~[?:?] at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358) ~[?:?] at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:96) ~[?:?] at org.apache.iceberg.rest.RESTSessionCatalog.listNamespaces(RESTSessionCatalog.java:562) ~[?:?] at org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.listNamespaces(BaseSessionCatalog.java:133) ~[?:?] at org.apache.iceberg.rest.RESTCatalog.listNamespaces(RESTCatalog.java:228) ~[?:?] at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74) ~[?:?] at org.apache.iceberg.rest.CatalogHandlers.listNamespaces(CatalogHandlers.java:115) ~[?:?] at org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.listNamespace(IcebergCatalogWrapper.java:137) ~[?:?] at org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.listSchemas(IcebergCatalogOperations.java:144) ~[?:?] at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$0(SchemaOperationDispatcher.java:82) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:150) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:145) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$1(SchemaOperationDispatcher.java:82) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:99) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$2(SchemaOperationDispatcher.java:80) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.SchemaOperationDispatcher.listSchemas(SchemaOperationDispatcher.java:76) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.hook.SchemaHookDispatcher.listSchemas(SchemaHookDispatcher.java:54) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.catalog.SchemaNormalizeDispatcher.listSchemas(SchemaNormalizeDispatcher.java:48) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.listener.SchemaEventDispatcher.listSchemas(SchemaEventDispatcher.java:77) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$0(SchemaOperations.java:85) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?] at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:39) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.server.web.Utils.doAs(Utils.java:198) ~[gravitino-server-common-1.0.0-SNAPSHOT.jar:?] at org.apache.gravitino.server.web.rest.SchemaOperations.listSchemas(SchemaOperations.java:81) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.41.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.41.jar:?] at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.41.jar:?] at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.41.jar:?] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.41.jar:?] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.41.jar:?] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.41.jar:?] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:86) ~[gravitino-server-common-1.0.0-SNAPSHOT.jar:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.apache.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:111) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217] at java.base/java.lang.Thread.run(Thread.java:833) [?:?] (base) ➜ dev-cluster-deploy ``` ```bash kubectl get svc -n lakehouse-dev ``` > NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gravitino ClusterIP 10.43.21.104 <none> 8090/TCP,9001/TCP 16m gravitino-mysql ClusterIP 10.43.179.149 <none> 3306/TCP 16m gravitino-mysql-headless ClusterIP None <none> 3306/TCP 16m ### How to reproduce ## Deployment Context ```yaml mysql: ## @param mysql.enabled Deploy MySQL container(s) ## enabled: true envFrom: [] log4j2Properties: rootLoggerLevel: INFO auxService: ## Auxiliary service names, separate by ',' ## names: iceberg-rest icebergRest: ## Iceberg REST service classpath ## classpath: "iceberg-rest-server/libs, iceberg-rest-server/conf" ## Iceberg REST service host ## host: 0.0.0.0 ## Iceberg REST service http port ## httpPort: 9001 # Rest backend configs. additionalConfigItems: ## THE CONFIGURATION EXAMPLE FOR JDBC CATALOG BACKEND WITH S3 SUPPORT gravitino.iceberg-rest.io-impl: org.apache.iceberg.aws.s3.S3FileIO gravitino.iceberg-rest.s3-endpoint: http://minio.lakehouse-dev.svc.cluster.local:9000 gravitino.iceberg-rest.s3-region: minio gravitino.iceberg-rest.s3-access-key-id: xxx gravitino.iceberg-rest.s3-secret-access-key: xxx gravitino.iceberg-rest.catalog-backend: jdbc gravitino.iceberg-rest.uri: jdbc:mysql://gravitino-mysql.lakehouse-dev.svc.cluster.local:3306/ gravitino.iceberg-rest.warehouse: s3://lakehouse-dev-iceberg gravitino.iceberg-rest.jdbc-user: gravitino gravitino.iceberg-rest.jdbc-password: gravitino gravitino.iceberg-rest.jdbc-initialize: "false" gravitino.iceberg-rest.jdbc-driver: com.mysql.cj.jdbc.Driver gravitino.iceberg-rest.catalog-cache-eviction-interval-ms: "100000" gravitino.iceberg-rest.catalog-config-provider: dynamic-config-provider gravitino.iceberg-rest.gravitino-uri: http://gravitino.lakehouse-dev.svc.cluster.local:8090 gravitino.iceberg-rest.gravitino-metalake: test ``` ## Catalog Configuration  ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
