[
https://issues.apache.org/jira/browse/IMPALA-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016242#comment-18016242
]
Quanlong Huang edited comment on IMPALA-14306 at 8/26/25 10:24 AM:
-------------------------------------------------------------------
After IMPALA-14283, coordinator will throw an
InconsistentMetadataFetchException when witness catalogd service ID changes.
The FE code should retry the request to use fresh metadata. However, this is
not done yet in
[Frontend.getDataSrcs()|https://github.com/apache/impala/blob/2fb56afb5e645a0c4e2b9bc3ab55e52cbee9fac0/fe/src/main/java/org/apache/impala/service/Frontend.java#L1623].
This leads to the query fail to list data sources:
{noformat}
I20250813 04:50:39.946695 367628 Frontend.java:2398]
fc42367bad15d507:67c38b5800000000] Analyzing query: SHOW DATA SOURCES LIKE
'test_failover_*' db: default
...
W20250813 04:50:39.972594 367628 CatalogdMetaProvider.java:1660]
fc42367bad15d507:67c38b5800000000] Detected catalog service restart: service ID
changed from 9d98517ba2d04d43:b49e6b33d7dce730 to
5925d93729534601:a99aa18792a5f34c. Invalidating all cached metadata on this
coordinator.
I20250813 04:50:39.975730 367628 LocalCatalog.java:249]
fc42367bad15d507:67c38b5800000000] Unable to load DataSource objects,
Java exception follows:
org.apache.impala.catalog.local.InconsistentMetadataFetchException: Catalog
service ID changed to 5925d93729534601:a99aa18792a5f34c
at
org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:496)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:209)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$11.call(CatalogdMetaProvider.java:1378)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$11.call(CatalogdMetaProvider.java:1374)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:612)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadDataSources(CatalogdMetaProvider.java:1372)
at
org.apache.impala.catalog.local.LocalCatalog.getDataSources(LocalCatalog.java:246)
at org.apache.impala.service.Frontend.getDataSrcs(Frontend.java:1624)
at
org.apache.impala.service.JniFrontend.getDataSrcMetadata(JniFrontend.java:434)
I20250813 04:50:39.976109 367628 impala-hs2-server.cc:651]
fc42367bad15d507:67c38b5800000000] ExecuteStatement():
return_val=TExecuteStatementResp {
01: status (struct) = TStatus {
01: statusCode (i32) = 0,
},
02: operationHandle (struct) = TOperationHandle {
01: operationId (struct) = THandleIdentifier {
01: guid (string) = "\a\xd5\x15\xad{6B\xfc\x00\x00\x00\x00X\x8b\xc3g",
02: secret (string) = "<HIDDEN>",
},
02: operationType (i32) = 0,
03: hasResultSet (bool) = true,
},
}{noformat}
-However, though not related to this bug, the test should fail earlier in
verifying the result status
[here|https://github.com/apache/impala/blob/2fb56afb5e645a0c4e2b9bc3ab55e52cbee9fac0/tests/custom_cluster/test_ext_data_sources.py#L169].
It's weird that the statement still returns as succeeded, i.e. statusCode = 0.
Could this be related to the recent protocol change to HS2 in tests?- [~rizaon]
was (Author: stiga-huang):
After IMPALA-14283, coordinator will throw an
InconsistentMetadataFetchException when witness catalogd service ID changes.
The FE code should retry the request to use fresh metadata. However, this is
not done yet in
[Frontend.getDataSrcs()|https://github.com/apache/impala/blob/2fb56afb5e645a0c4e2b9bc3ab55e52cbee9fac0/fe/src/main/java/org/apache/impala/service/Frontend.java#L1623].
This leads to the query fail to list data sources:
{noformat}
I20250813 04:50:39.946695 367628 Frontend.java:2398]
fc42367bad15d507:67c38b5800000000] Analyzing query: SHOW DATA SOURCES LIKE
'test_failover_*' db: default
...
W20250813 04:50:39.972594 367628 CatalogdMetaProvider.java:1660]
fc42367bad15d507:67c38b5800000000] Detected catalog service restart: service ID
changed from 9d98517ba2d04d43:b49e6b33d7dce730 to
5925d93729534601:a99aa18792a5f34c. Invalidating all cached metadata on this
coordinator.
I20250813 04:50:39.975730 367628 LocalCatalog.java:249]
fc42367bad15d507:67c38b5800000000] Unable to load DataSource objects,
Java exception follows:
org.apache.impala.catalog.local.InconsistentMetadataFetchException: Catalog
service ID changed to 5925d93729534601:a99aa18792a5f34c
at
org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:496)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:209)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$11.call(CatalogdMetaProvider.java:1378)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$11.call(CatalogdMetaProvider.java:1374)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:612)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadDataSources(CatalogdMetaProvider.java:1372)
at
org.apache.impala.catalog.local.LocalCatalog.getDataSources(LocalCatalog.java:246)
at org.apache.impala.service.Frontend.getDataSrcs(Frontend.java:1624)
at
org.apache.impala.service.JniFrontend.getDataSrcMetadata(JniFrontend.java:434)
I20250813 04:50:39.976109 367628 impala-hs2-server.cc:651]
fc42367bad15d507:67c38b5800000000] ExecuteStatement():
return_val=TExecuteStatementResp {
01: status (struct) = TStatus {
01: statusCode (i32) = 0,
},
02: operationHandle (struct) = TOperationHandle {
01: operationId (struct) = THandleIdentifier {
01: guid (string) = "\a\xd5\x15\xad{6B\xfc\x00\x00\x00\x00X\x8b\xc3g",
02: secret (string) = "<HIDDEN>",
},
02: operationType (i32) = 0,
03: hasResultSet (bool) = true,
},
}{noformat}
However, though not related to this bug, the test should fail earlier in
verifying the result status
[here|https://github.com/apache/impala/blob/2fb56afb5e645a0c4e2b9bc3ab55e52cbee9fac0/tests/custom_cluster/test_ext_data_sources.py#L169].
It's weird that the statement still returns as succeeded, i.e. statusCode = 0.
Could this be related to the recent protocol change to HS2 in tests? [~rizaon]
> TestExtDataSources.test_catalogd_ha_failover failed
> ---------------------------------------------------
>
> Key: IMPALA-14306
> URL: https://issues.apache.org/jira/browse/IMPALA-14306
> Project: IMPALA
> Issue Type: Bug
> Reporter: Xuebin Su
> Assignee: Quanlong Huang
> Priority: Critical
>
> h3. Stacktrace
> {code:java}
> custom_cluster/test_ext_data_sources.py:170: in test_catalogd_ha_failover
> assert "FailoverInSyncJdbcDataSource" in result.get_data()
> E assert 'FailoverInSyncJdbcDataSource' in ''
> E + where '' = <bound method ImpylaHS2ResultSet.get_data of
> <tests.common.impala_connection.ImpylaHS2ResultSet object at
> 0x7f4b7adfcad0>>()
> E + where <bound method ImpylaHS2ResultSet.get_data of
> <tests.common.impala_connection.ImpylaHS2ResultSet object at 0x7f4b7adfcad0>>
> = <tests.common.impala_connection.ImpylaHS2ResultSet object at
> 0x7f4b7adfcad0>.get_data{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
