Hi Vibhatha,
I think those APIs are still avaiable?
```
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.1
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 11.0.19)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.range(0, 10)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> df.queryExecution
res0: org.apache.spark.sql.execution.QueryExecution =
== Parsed Logical Plan ==
Range (0, 10, step=1, splits=Some(12))
== Analyzed Logical Plan ==
id: bigint
Range (0, 10, step=1, splits=Some(12))
== Optimized Logical Plan ==
Range (0, 10, step=1, splits=Some(12))
== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
scala> df.queryExecution.optimizedPlan
res1: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
Range (0, 10, step=1, splits=Some(12))
```
On Wed, Aug 2, 2023 at 3:58 PM Vibhatha Abeykoon <vibha...@gmail.com> wrote:
> Hi Winston,
>
> I need to use the LogicalPlan object and process it with another function
> I have written. In earlier Spark versions we can access that via the
> dataframe object. So if it can be accessed via the UI, is there an API to
> access the object?
>
> On Wed, Aug 2, 2023 at 1:24 PM Winston Lai <weiruanl...@gmail.com> wrote:
>
>> Hi Vibhatha,
>>
>> How about reading the logical plan from Spark UI, do you have access to
>> the Spark UI? I am not sure what infra you run your Spark jobs on. Usually
>> you should be able to view the logical and physical plan under Spark UI in
>> text version at least. It is independent from the language (e.g.,
>> scala/Python/R) that you use to run Spark.
>>
>>
>> On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com>
>> wrote:
>>
>>> Hi Winston,
>>>
>>> I am looking for a way to access the LogicalPlan object in Scala. Not
>>> sure if explain function would serve the purpose.
>>>
>>> On Wed, Aug 2, 2023 at 9:14 AM Winston Lai <weiruanl...@gmail.com>
>>> wrote:
>>>
>>>> Hi Vibhatha,
>>>>
>>>> Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1
>>>> documentation (apache.org)
>>>> <https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html#pyspark.sql.DataFrame.explain>
>>>> before?
>>>> I am not sure what infra that you have, you can try this first. If it
>>>> doesn't work, you may share more info such as what platform you are running
>>>> your Spark jobs on, what cloud servies you are using ...
>>>>
>>>> On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I recently upgraded the Spark version to 3.4.1 and I have encountered
>>>>> a few issues. In my previous code, I was able to extract the logical plan
>>>>> using `df.queryExecution` (df: DataFrame and in Scala), but it seems like
>>>>> in the latest API it is not supported. Is there a way to extract the
>>>>> logical plan or optimized plan from a dataframe or dataset in Spark 3.4.1?
>>>>>
>>>>> Best,
>>>>> Vibhatha
>>>>>
>>>> --
>>> Vibhatha Abeykoon
>>>
>> --
> Vibhatha Abeykoon
>
--
Ruifeng Zheng
E-mail: zrfli...@gmail.com
