Re: Spark 3 is Slower than Spark 2 for TPCDS Q04 query.

2021-12-20 Thread Senthil Kumar
Also we checked that we have already backported https://issues.apache.org/jira/browse/SPARK-33557 jira. On Mon, Dec 20, 2021 at 11:08 AM Senthil Kumar wrote: > @abhishek. We use spark 3.1* > > On Mon, 20 Dec 2021, 09:50 Rao, Abhishek (Nokia - IN/Bangalore), < > abhishek@nokia.com> wrote: > >

RE: Spark 3 is Slower than Spark 2 for TPCDS Q04 query.

2021-12-20 Thread Luca Canali
Hi Senthil, I have just run a couple of quick tests for TPCDS Q4, using the TPCDS schema created at scale 1500 that I have on a Hadoop/YARN cluster, and was not able to reproduce the difference in execution time between Spark 2 and Spark 3 that you report in your mail. This is the Spark c

Re: Spark 3 is Slower than Spark 2 for TPCDS Q04 query.

2021-12-20 Thread Senthil Kumar
Hi Luca, I m collecting logical n physical plan. So that it will be helpful to find the root cause of this issue On Mon, 20 Dec 2021, 16:46 Luca Canali, wrote: > Hi Senthil, > > > > I have just run a couple of quick tests for TPCDS Q4, using the TPCDS > schema created at scale 1500 that I have