Re: Speeding up Catalyst engine

2017-07-24 Thread Liang-Chi Hsieh
Hi Maciej, For backportting https://issues.apache.org/jira/browse/SPARK-20392, you can see the suggestion from committers on the PR. I think we don't expect it will be merged into 2.2. Maciej Bryński wrote > Hi Everyone, > I'm trying to speed up my Spark streaming application and I have follow

Speeding up Catalyst engine

2017-07-24 Thread Maciej Bryński
Hi Everyone, I'm trying to speed up my Spark streaming application and I have following problem. I'm using a lot of joins in my app and full catalyst analysis is triggered during every join. I found 2 options to speed up. 1) spark.sql.selfJoinAutoResolveAmbiguity option But looking at code: http