/SparkSession.html
With Regards,
Vibhatha Abeykoon, PhD
On Wed, Aug 2, 2023 at 4:22 PM Vibhatha Abeykoon wrote:
> Hello Winston,
>
> Thanks again for this response, I will check this one out.
>
> On Wed, Aug 2, 2023 at 3:50 PM Winston Lai wrote:
>
>>
>> Hi Vibhatha
Hello Winston,
Thanks again for this response, I will check this one out.
On Wed, Aug 2, 2023 at 3:50 PM Winston Lai wrote:
>
> Hi Vibhatha,
>
> I helped you post this question to another community. There is one answer
> by someone else for your reference.
>
> To acces
but it is not a LogicalPlan
> object.
>
> On Wed, Aug 2, 2023 at 4:36 PM Vibhatha Abeykoon
> wrote:
>
>> Hello Ruifeng,
>>
>> Thank you for these pointers. Would it be different if I use the Spark
>> connect? I am not using the regular SparkSession. I a
Hello Ruifeng,
Thank you for these pointers. Would it be different if I use the Spark
connect? I am not using the regular SparkSession. I am pretty new to these
APIs. Appreciate your thoughts.
On Wed, Aug 2, 2023 at 2:00 PM Ruifeng Zheng wrote:
> Hi Vibhatha,
>I think those APIs are
wrote:
> Hi Vibhatha,
>
> How about reading the logical plan from Spark UI, do you have access to
> the Spark UI? I am not sure what infra you run your Spark jobs on. Usually
> you should be able to view the logical and physical plan under Spark UI in
> text version at least. It is in
Hi Winston,
I am looking for a way to access the LogicalPlan object in Scala. Not sure
if explain function would serve the purpose.
On Wed, Aug 2, 2023 at 9:14 AM Winston Lai wrote:
> Hi Vibhatha,
>
> Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1
> documentation
logical plan
or optimized plan from a dataframe or dataset in Spark 3.4.1?
Best,
Vibhatha
Hi,
I am doing a benchmark in Flink, Storm, Spark for an iterative streaming
application.
The goal is to make a window for a stream and do an iterative computation
per window.
Both Flink and Storm provides a window function with a list or iterator. But
in Spark,
I am not quite sure how to do
Hi,
I have an issue is reduce in streaming. I don't get a reduced stream when I
use a custom object.
Here is the code snippet that I used to test this.
Issue is, the reduction clearly works for a simple sequence, but what I
really want is to do is
send an array by adding such an array at a t
