/SparkSession.html
With Regards,
Vibhatha Abeykoon, PhD
On Wed, Aug 2, 2023 at 4:22 PM Vibhatha Abeykoon wrote:
> Hello Winston,
>
> Thanks again for this response, I will check this one out.
>
> On Wed, Aug 2, 2023 at 3:50 PM Winston Lai wrote:
>
>>
>> Hi Vibhatha
column") > 10)
> > val resultDF = filteredDF.select("column1", "column2")
> >
> > // Trigger the execution of the DF to invoke the listener
> > resultDF.show()
>
> Thank You & Best Regards
> Winston Lai
> --
&
but it is not a LogicalPlan
> object.
>
> On Wed, Aug 2, 2023 at 4:36 PM Vibhatha Abeykoon
> wrote:
>
>> Hello Ruifeng,
>>
>> Thank you for these pointers. Would it be different if I use the Spark
>> connect? I am not using the regular SparkSession. I a
== Physical Plan ==
> *(1) Range (0, 10, step=1, splits=12)
>
> scala> df.queryExecution.optimizedPlan
> res1: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
> Range (0, 10, step=1, splits=Some(12))
> ```
>
>
>
> On Wed, Aug 2, 2023 at 3:58 PM Vibhatha Abeykoon
> wrote:
>
>> Hi W
dependent from the language (e.g.,
> scala/Python/R) that you use to run Spark.
>
>
> On Wednesday, August 2, 2023, Vibhatha Abeykoon
> wrote:
>
>> Hi Winston,
>>
>> I am looking for a way to access the LogicalPlan object in Scala. Not
>> sure if explain fu
ch as what platform you are running
> your Spark jobs on, what cloud servies you are using ...
>
> On Wednesday, August 2, 2023, Vibhatha Abeykoon
> wrote:
>
>> Hello,
>>
>> I recently upgraded the Spark version to 3.4.1 and I have encountered a
>> few is
Hello,
I recently upgraded the Spark version to 3.4.1 and I have encountered a few
issues. In my previous code, I was able to extract the logical plan using
`df.queryExecution` (df: DataFrame and in Scala), but it seems like in the
latest API it is not supported. Is there a way to extract the logi
