Re: Bugs with joins and SQL in Structured Streaming

Hello! Thank you for looking into these issues! I'm happy that you identified the root cause for OuterJoinTest and SqlSyntaxTest and working on the fix. Regarding IntervalJoinTest, I think I understand your point. Thank you for explaining that. However, this can be confusing to a user. Let's mayb

Re: Bugs with joins and SQL in Structured Streaming

I figured out the issue which breaks the second test in SqlSyntaxTest. This is also a correctness issue, unfortunately. Issue and the fix for OuterJoinTest: https://issues.apache.org/jira/browse/SPARK-49829 Issue and the fix for SqlSyntaxTest: https://issues.apache.org/jira/browse/SPARK-49836 Tha

Re: Bugs with joins and SQL in Structured Streaming

I just quickly looked into SqlSyntaxTest - the first broken test looks to be fixed via SPARK-46062 which was released in Spark 3.5.1. The second broken test is a valid issue and I'm yet to know why this is happening. I'll file a JIRA ticket and le

Re: Bugs with joins and SQL in Structured Streaming

Sorry I totally missed this email. This is forgotten for 6 months but I'm happy that we have smart users reporting such complex edge-case issues! I haven't had time to validate all of them but OuterJoinTest is a valid correctness issue indeed. Thanks for reporting to us! I figured out the root cau

Re: Bugs with joins and SQL in Structured Streaming

Hi, Do you think there is any chance for this issue to get resolved? Should I create another bug report? As mentioned in my message, there is one open already: https://issues.apache.org/jira/browse/SPARK-45637 but it covers only one of the problems. Andrzej wt., 27 lut 2024 o 09:58 Andrzej Zera

Re: Bugs with joins and SQL in Structured Streaming

Hi, Yes, I tested all of them on spark 3.5. Regards, Andrzej pon., 26 lut 2024 o 23:24 Mich Talebzadeh napisał(a): > Hi, > > These are all on spark 3.5, correct? > > Mich Talebzadeh, > Dad | Technologist | Solutions Architect | Engineer > London > United Kingdom > > >view my Linkedin prof

Re: Bugs with joins and SQL in Structured Streaming

Hi, These are all on spark 3.5, correct? Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The informa

Bugs with joins and SQL in Structured Streaming

Hey all, I've been using Structured Streaming in production for almost a year already and I want to share the bugs I found in this time. I created a test for each of the issues and put them all here: https://github.com/andrzejzera/spark-bugs/tree/main/spark-3.5/src/test/scala I split the issues i