heguanhui commented on issue #65479: URL: https://github.com/apache/doris/issues/65479#issuecomment-4945099718
@morningman Hi PMC, after carefully reading the design note and deeply analyzing the current codebase (specifically MetadataGenerator.java, TSchemaTableRequestParams.thrift, PhysicalSchemaScan, and the FE RPC entry fetchSchemaTableData), I completely agree with the identified performance risks and architectural technical debts. I would like to claim the following batch of tasks to establish the foundational architecture and migrate the core SHOW commands. I will start by fixing the existing high-risk tables to establish the pattern, and then apply this unified pattern to the low-risk SHOW commands: ### Phase 1: High-Risk POC (Establishing the framework) - information_schema.partitions - information_schema.backend_tablets (and related tablet metadata) ### Phase 2: Low-Risk SHOW Commands Migration (Eliminating logic redundancy) - SHOW FRONTENDS / frontends() → Create information_schema.frontends - SHOW BACKENDS / backends() → Create information_schema.backends - SHOW CLUSTERS / SHOW COMPUTE GROUPS → Create information_schema.clusters - SHOW TRASH → Create information_schema.trash - SHOW CATALOG RECYCLE BIN → Create information_schema.catalog_recycle_bin - SHOW STORAGE VAULT(S) → Create information_schema.storage_vaults - SHOW STORAGE POLICY → Create information_schema.storage_policies - SHOW RESOURCES → Create information_schema.resources - SHOW TRANSACTION → Create information_schema.transactions - SHOW ROLES → Create information_schema.roles - SHOW (ALL) GRANTS → Create information_schema.grants ### Implementation Plan based on your design: - **Thrift Extension:** Extend TSchemaTableRequestParams to carry structured constraints (e.g., table_name, schema_name) explicitly, instead of relying solely on serialized frontend_conjuncts for this critical path. - **Execution Path & Result Generation Unification:** Currently, the FE-side fetchSchemaTableData RPC router splits queries into getSchemaTableData (which does full scans) and getMetadataTable (which is precise). We plan to unify the execution logic by routing SHOW commands through PhysicalSchemaScan (replacing the split ShowExecutor). Concurrently, we will consolidate the split logic in MetadataGenerator to ensure all metadata queries share the same structured predicate extraction and precise execution path. - **Planner-side Guard:** Introduce a registry mechanism in SchemaTable declaring risk levels and RequiredAny predicates (e.g., TABLE_SCHEMA, TABLE_NAME). Reject unsafe full scans for high-risk tables, while allowing low-risk tables to be fully scanned. - **Predicate Extraction & Pushdown:** Ensure the planner extracts predicates and pushes down structured constraints so FE can fetch precisely (e.g., calling database.getTable(tableName)), avoiding full database scans. - **BE-side Filtering:** The original SQL predicates will remain in BE to ensure complete correctness. This will give us a testable, end-to-end POC for the high-risk path, followed by the migration of the SHOW commands using the established pattern. Please let me know if this scope and approach aligns with your expectations! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
