Hi, I'm using Structured Streaming to consume data from Kafka and load it into HDFS.
I discovered that AQE (Adaptive Query Execution) is forcibly disabled when using Structured Streaming, as noted in this issue: https://issues.apache.org/jira/browse/SPARK-19873 In my case, the streaming query does not involve any state. I don’t use window functions or joins; I simply write the DataFrame to HDFS. Could AQE still cause any problems in this scenario? I’d like to understand the specific issues that AQE might cause. Best regards, Junyoung Cho --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
