[
https://issues.apache.org/jira/browse/IMPALA-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13867:
------------------------------------
Epic Link: IMPALA-13915
> Improves logging of GetTables on all dbs
> ----------------------------------------
>
> Key: IMPALA-13867
> URL: https://issues.apache.org/jira/browse/IMPALA-13867
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> When authorization is enabled, GetTables or SHOW TABLES requests will check
> whether a table is visible to the user before adding the table to the result
> list. This requires checking the ownership info. However, when the table is
> unloaded in catalog (i.e. only the table name is known), the ownership info
> is missing. Coordinator will log these:
> {noformat}
> Owner of table_name is unknown due to msTable is unloaded
> Table table_name not yet loaded, ignoring it in table listing.
> {noformat}
> Note that the last log is misleading. The table might still be returned if
> privilege checks allowed it, e.g. when user has ALL privilege on SERVER. We
> should only add these logs when the table is unloaded and actually ignored.
> On the other hand, when there are lots of concurrent global GetTables
> requests (i.e. on all dbs) and they have a huge result list (e.g. 1M tables),
> the request could be blocked in logging:
> {noformat}
> "Thread-6985" Id=31986 BLOCKED on org.apache.log4j.spi.RootLogger@6bd98257
> owned by "Thread-6968" Id=31969
> at org.apache.log4j.Category.callAppenders(Category.java:204)
> at org.apache.log4j.Category.forcedLog(Category.java:384)
> at org.apache.log4j.Category.log(Category.java:810)
> at
> org.slf4j.reload4j.Reload4jLoggerAdapter.handleNormalizedLoggingCall(Reload4jLoggerAdapter.java:137)
> at
> org.slf4j.helpers.AbstractLogger.handle_1ArgsCall(AbstractLogger.java:386)
> at org.slf4j.helpers.AbstractLogger.warn(AbstractLogger.java:269)
> at
> org.apache.impala.catalog.local.LocalTable.getOwnerUser(LocalTable.java:234)
> // LOG.warn("Owner of {} is unknown due to msTable is unloaded"
> at
> org.apache.impala.catalog.local.LocalIncompleteTable.getOwnerUser(LocalIncompleteTable.java:34)
> at
> org.apache.impala.service.Frontend.filterTablesIfAuthNeeded(Frontend.java:1251)
> at
> org.apache.impala.service.Frontend.doGetCatalogTableNames(Frontend.java:1269)
> at
> org.apache.impala.service.Frontend.getTableNames(Frontend.java:1181)
> at
> org.apache.impala.service.Frontend.getTableNames(Frontend.java:1168)
> at
> org.apache.impala.service.MetadataOp.doGetDbsMetadata(MetadataOp.java:367)
> at
> org.apache.impala.service.MetadataOp.getDbsMetadata(MetadataOp.java:332)
> at org.apache.impala.service.MetadataOp.getTables(MetadataOp.java:605)
> at
> org.apache.impala.catalog.Hive3MetastoreShimBase.execGetTables(Hive3MetastoreShimBase.java:243)
> at
> org.apache.impala.service.Frontend.execHiveServer2MetadataOp(Frontend.java:2947)
> at
> org.apache.impala.service.JniFrontend.execHiveServer2MetadataOp(JniFrontend.java:633){noformat}
> Logging too many warnings for each unloaded tables might not be useful. We
> should limit the logs, e.g. only for the first several tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
