Re: Migration from Spark 2.4.0 to Spark 3.1.1 caused SortMergeJoin to change to BroadcastHashJoin

/docs/latest/sql-ref-syntax-qry-select-hints.html i hope this helps .Best Tufan  On Wed, 6 Jul 2022 at 17:11, igor cabral uchoa wrote: Hi all, I hope everyone is doing well.  I'm currently working on a Spark migration project that aims to migrate all Spark SQL pipelines for Spark 3.x version

Migration from Spark 2.4.0 to Spark 3.1.1 caused SortMergeJoin to change to BroadcastHashJoin

Hi all, I hope everyone is doing well.  I'm currently working on a Spark migration project that aims to migrate all Spark SQL pipelines for Spark 3.x version and take advantage of all performance improvements on it. My company is using Spark 2.4.0 but we are targeting to use officially the 3.1.1

Re: Spark job fails because of timeout to Driver

XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError=\'kill -9 %p\'' '--conf' 'spark.executor.userClassPathFirst=true' '--conf' 'spark.submit.deployMode=cluster' '--conf' 

Re: Spark job fails because of timeout to Driver

Hi Roland! What deploy mode are you using when you submit your applications? It is client or cluster mode? Regards, Sent from Yahoo Mail for iPhone On Friday, October 4, 2019, 12:37 PM, Roland Johann wrote: This are dynamic port ranges and dependa on configuration of your cluster. Per job