Re: [Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread Marco Gaido
Hi, you can check your execution plan and you can find from there which *Exec classes are used. Please notice that in case of wholeStageCodegen, its children operators are executed inside the wholeStageCodegenExec. Bests, Marco Il giorno gio 28 mar 2019 alle ore 15:21 ehsan shams < ehsan.shams.r

Re: [Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread Sean Owen
I'd suggest loading the source in an IDE if you want to explore the code base. It will let you answer this in one click. Here it's Dataset, as a DataFrame is a Dataset[Row]. On Thu, Mar 28, 2019 at 9:21 AM ehsan shams wrote: > > Hi > > I would like to know where exactly(which class/function) spar

[Spark SQL]: looking for place operators apply on the dataset / dataframe

2019-03-28 Thread ehsan shams
Hi I would like to know where exactly(which class/function) spark sql will apply the operators on dataset / dataframe rows. For example by applying the following filter or groupby which class is responsible for? And will iterate over the rows to do its operation? Kind regards Ehsan Shams val df1