wayne-x-wang opened a new pull request, #7792:
URL: https://github.com/apache/kyuubi/pull/7792
### Why are the changes needed?
Tables used only by filter subqueries are missing from input lineage. For
example:
```sql
INSERT OVERWRITE v2_catalog.db.tb3
SELECT t1.* FROM v2_catalog.db.tb1 t1
WHERE EXISTS (
SELECT 1 FROM v2_catalog.db.tb2 t2 WHERE t2.col1 = t1.col1
);
```
Before this change, the analyzed plan contains `tb2` under
`Filter.condition`, but lineage lists only `tb1`. Ordinary `children` traversal
does not visit the subquery plan.
Visit `SubqueryExpression.plan` in filters and collect its input tables
without merging its column mappings. Output columns keep their existing
sources. The existing
`spark.kyuubi.plugin.lineage.legacy.collectInputTablesByColumn=true` mode
retains its previous input-table semantics. No new configuration or dependency
is introduced.
This follows the traversal approach discussed in #7207 while avoiding the
additional configuration questioned in review. Five existing input-table
expectations are corrected; their column mappings are unchanged.
Fixes #7206.
### How was this patch tested?
The correlated `EXISTS` regression failed in two independent JVM runs on the
unmodified master implementation (Spark 3.5.8):
```text
Set("v2_catalog.db.tb1") did not equal Set("v2_catalog.db.tb1",
"v2_catalog.db.tb2")
```
Its output-table and three column-source assertions passed before the
input-table assertion failed. With the regression added, the pre-fix lineage
suite had 75 passing tests and the same failure in both catalog suites.
Added 36 compatibility cases: nine scenarios × two legacy settings × two
catalog suites. These cover `EXISTS`, `NOT EXISTS`, `IN`, `NOT IN`, scalar
subqueries, `HAVING`, nested subqueries, composite conditions and a filter
without subqueries. They check input/output tables and column mappings, and
restore the previous configuration in `finally`. Tests use Spark's
parser/analyzer and the existing in-memory V2 catalogs, matching the listener's
analyzed-plan input.
After the fix, **all 113 lineage tests passed** on each matrix entry:
| Spark | Scala | Passed | Failed |
| --- | --- | ---: | ---: |
| 3.5.8 | 2.12.19 | 113 | 0 |
| 4.2.0 | 2.13.17 | 113 | 0 |
Both runs also passed Spotless and Scalastyle. Commands below omit only the
local Maven mirror settings path:
```bash
build/mvn -B -ntp test spotless:check scalastyle:check \
-pl extensions/spark/kyuubi-spark-lineage -am \
-Pspark-3.5,spark-provided,flink-provided,hive-provided \
-Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage
build/mvn -B -ntp test spotless:check scalastyle:check \
-pl extensions/spark/kyuubi-spark-lineage -am \
-Pspark-4.2,scala-2.13,spark-provided,flink-provided,hive-provided \
-Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage
```
`dev/reformat` and `git diff --check` passed. The final source hashes match
the files used in both successful test runs.
### Was this patch assisted by generative AI tooling?
Assisted-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
