Hello Winston,
Thanks again for this response, I will check this one out.
On Wed, Aug 2, 2023 at 3:50 PM Winston Lai wrote:
>
> Hi Vibhatha,
>
> I helped you post this question to another community. There is one answer
> by someone else for your reference.
>
> To access the logical plan or opti
Hi Vibhatha,
I helped you post this question to another community. There is one answer by
someone else for your reference.
To access the logical plan or optimized plan, you can register a custom
QueryExecutionListener and retrieve the plans during the query execution
process. Here's an exampl
I understand. I sort of drew the same conclusion. But I wasn’t sure. Thanks
everyone for taking time on this.
On Wed, Aug 2, 2023 at 2:29 PM Ruifeng Zheng wrote:
> In Spark Connect, I think the only API to show optimized plan is
> `df.explain("extended")` as Winston mentioned, but it is not a Lo
In Spark Connect, I think the only API to show optimized plan is
`df.explain("extended")` as Winston mentioned, but it is not a LogicalPlan
object.
On Wed, Aug 2, 2023 at 4:36 PM Vibhatha Abeykoon wrote:
> Hello Ruifeng,
>
> Thank you for these pointers. Would it be different if I use the Spark
Hello Ruifeng,
Thank you for these pointers. Would it be different if I use the Spark
connect? I am not using the regular SparkSession. I am pretty new to these
APIs. Appreciate your thoughts.
On Wed, Aug 2, 2023 at 2:00 PM Ruifeng Zheng wrote:
> Hi Vibhatha,
>I think those APIs are still a
Hi Vibhatha,
I think those APIs are still avaiable?
```
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.1
/_/
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 11.0.19)
Type in expressio
Hi Winston,
I need to use the LogicalPlan object and process it with another function I
have written. In earlier Spark versions we can access that via the
dataframe object. So if it can be accessed via the UI, is there an API to
access the object?
On Wed, Aug 2, 2023 at 1:24 PM Winston Lai wrote
Hi Vibhatha,
How about reading the logical plan from Spark UI, do you have access to the
Spark UI? I am not sure what infra you run your Spark jobs on. Usually you
should be able to view the logical and physical plan under Spark UI in text
version at least. It is independent from the language (e.g
