[
https://issues.apache.org/jira/browse/SPARK-58089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58089:
-----------------------------------
Labels: pull-request-available (was: )
> Variant extraction pushdown does not work for Aggregate, Sort, and Join
> -----------------------------------------------------------------------
>
> Key: SPARK-58089
> URL: https://issues.apache.org/jira/browse/SPARK-58089
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.2
> Reporter: Qiegang Long
> Priority: Major
> Labels: pull-request-available
>
> While running performance test, noticed that variant extraction pushdown does
> does not extend to aggregate and join, losing performance benefits of
> shreddred columns.
> Extaction pushdown not working for these examples:
> Example 1:
> {code:sql}
> SELECT AVG(variant_get(data, '$.qty', 'double'))
> FROM store_sales
> GROUP BY variant_get(data, '$.id', 'string')
> {code}
> Example 2:
> {code:sql}
> SELECT AVG(variant_get(ss.data, '$.qty', 'double'))
> FROM store_sales ss
> JOIN date_dim d ON ss.date_sk = d.date_sk
> JOIN store st ON ss.store_sk = st.store_sk
> GROUP BY variant_get(ss.data, '$.id', 'string')
> {code}
> Example 3:
> {code:sql}
> SELECT ss.k
> FROM store_sales ss
> JOIN date_dim d ON ss.date_sk = variant_get(d.data, '$.sk', 'int')
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
