Re: In built Optimizer on Spark

Hi Felix, As you may be aware Spark sql does have a catalyst optimiser. What is the Catalyst Optimizer? - Databricks You Mentioned Spark Structured Streaming. What specifics are you looking for? Have you considered the Spark GUI streaming tab

In built Optimizer on Spark

Hello, Is there any in-built optimizer in Spark as in Flink, to avoid manual configuration tuning to achieve better performance of your structured streaming pipeline? Or is there any work happening to achieve this? Regards, Felix K Jose