Bettle created FLINK-40658:
------------------------------
Summary: UNNEST silently drops null ROW elements from ARRAY
Key: FLINK-40658
URL: https://issues.apache.org/jira/browse/FLINK-40658
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 2.4.0
Reporter: Bettle
While implementing DataFrame.explode() for FLINK-40435, I noticed that UNNEST
silently drops null ROW elements from an ARRAY.
This reproduces through direct Table API/SQL calls on an unchanged baseline,
without importing or using the new DataFrame implementation.
h3. Reproduction
Run the following code in a configured PyFlink environment:
{code:python}
from pyflink.table import EnvironmentSettings, TableEnvironment
t_env = TableEnvironment.create(EnvironmentSettings.in_batch_mode())
source = t_env.sql_query(
"SELECT ARRAY[CAST(NULL AS ROW<n INT, s STRING>), "
"ROW(1, 'a')] AS items"
)
for join, condition in [
("CROSS JOIN", ""),
("LEFT JOIN", " ON TRUE"),
]:
query = (
f"SELECT expanded.* FROM `{str(source)}` AS src "
f"{join} UNNEST(src.items) AS expanded(n, s){condition}"
)
with t_env.sql_query(query).execute().collect() as rows:
actual = list(rows)
print(join, actual)
{code}
h3. Expected result
Both join variants should return two rows, regardless of order:
{code}
(NULL, NULL)
(1, 'a')
{code}
h3. Actual result
Both join variants return only:
{code}
(1, 'a')
{code}
Execution completes without an exception, but the null ROW element is missing.
The input is a nonempty array containing a null ROW element. This is different
from an empty array, a null array, or a non-null ROW whose fields are all null.
h3. Environment
* Flink 2.4-SNAPSHOT, baseline e01bbcacd96
* Batch mode
* Java 17.0.14
* Python 3.12.11
The equivalent direct Table API/SQL tests reproduced this behavior for both
CROSS JOIN and LEFT JOIN. Released versions and streaming mode have not been
verified for this issue.
h3. Related work
This was discovered during FLINK-40435, but reproduces without the new
DataFrame wrappers. I am reporting it separately to keep that work focused on
the DataFrame API. No underlying runtime fix is included in that work.
If this is confirmed to be a bug that needs fixing, I would be happy to
investigate further and try to contribute a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
