Lisa Cao created SPARK-60206:
--------------------------------
Summary: Metric views resolve unqualified names in the reader's
current namespace instead of the creation namespace
Key: SPARK-60206
URL: https://issues.apache.org/jira/browse/SPARK-60206
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Lisa Cao
A metric view resolves unqualified names against the current catalog and
namespace of whoever queries it, not the ones it was created in. Regular views
use the creation namespace.
{code:sql}
CREATE TABLE default.t USING parquet AS SELECT * FROM VALUES ('a', 1), ('b', 2)
AS v(k, x);
CREATE DATABASE other;
CREATE TABLE other.t USING parquet AS SELECT * FROM VALUES ('z', 1000) AS v(k,
x);
USE default;
CREATE VIEW default.mv WITH METRICS LANGUAGE YAML AS $$
version: "0.1"
source: t
dimensions:
- name: k
expr: k
measures:
- name: total
expr: sum(x)
$$;
CREATE VIEW default.v AS SELECT sum(x) AS total FROM t;
USE other;
SELECT measure(total) FROM default.mv; -- 1000, reads other.t
SELECT total FROM default.v; -- 3
{code}
The same happens for:
* SQL sources ({{source: SELECT * FROM t}})
* functions in dimension expressions ({{expr: f(k)}} calls {{other.f}})
* subqueries in {{filter}}
* temp views: with {{CREATE TEMPORARY VIEW t}} (or
{{df.createOrReplaceTempView("t")}}) in the reader's session, the metric view
reads the temp view, even from {{default}}
* v2 catalog metric views, including after {{USE}} of another catalog
>From a namespace without {{t}} it fails with {{TABLE_OR_VIEW_NOT_FOUND}}.
>Reproduced on 4.2.0 and 4.3.0-rc1, with in-memory and Hive catalogs, and over
>Spark Connect.
The creation namespace is stored ({{view.catalogAndNamespace.*}}), but
{{SessionCatalog.parseMetricViewDefinition}} only applies the view's SQL
configs. Regular views resolve under
{{AnalysisContext.withAnalysisContext(view.desc)}} ({{ViewResolution}}); metric
views skip it.
Workaround: fully qualify names in the definition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
