Re: Another Interesting Question on SPARK SQL

2016-11-17 Thread Herman van Hövell tot Westerflier
The diagram you have included, is a depiction of the steps Catalyst (the spark optimizer) takes to create an executable plan. Tungsten mainly comes into play during code generation and the actual execution. A datasource is represented by a LogicalRelation during analysis & optimization. The spark

Another Interesting Question on SPARK SQL

2016-11-17 Thread kant kodali
​ Which parts in the diagram above are executed by DataSource connectors and which parts are executed by Tungsten? or to put it in another way which phase in the diagram above does Tungsten leverages the Datasource connectors (such as say cassandra connector ) ? My understanding so far is that con