Hi,
I have tried using lateral join on two tables in flink sql similar to the
example

https://github.com/apache/flink/blob/c724168fad4215626b5596dd63cb66e477948aa0/flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/basics/UpdatingTopCityExample.java#L130

But I get the the following error
Caused by: org.apache.flink.table.api.TableException: unexpected correlate
variable $cor0 in the plan

And to confirm that this error is not specific to my use case, I have tried
the flink's UpdatingTopCityExample too with flink deployment on flink
kubernetes operator and I got the same error.

I have found that there is an unresolved bug that is causing this issue.
https://issues.apache.org/jira/browse/FLINK-31163

As a workaround I have tried using a left join, but unlike a lateral join
where I can specify the limit in the correlated subquery, the same cannot
be done in left join, instead I have to assign a rownumber and then filter
on rownumber to reduce the join output. And this is not the optimal way to
do incase of datasets with millions of records.

Is there an optimal way of doing this in huge datasets?
Or is there a workaround similar to lateral join where the join does not
need to calculate all the matches?

Any help would be much appreciated.

Thanks
Yashoda

Reply via email to