Thanks Mich,
I have tried this solution, but i want all the columns from the dataframe
df_1, if i explode the df_1 i am getting only data column. But the
resultant should get the all the column from the df_1 with distinct result
like below.
Results in
*df:*
+---+
|column1|
+---+
| a
Hi,
I understand that Zstd compression can optionally be provided a dictionary
object to improve performance. See “training mode” here
https://facebook.github.io/zstd/
Does Spark surface a way to provide this dictionary object when writing/reading
data? What about for intermediate shuffle resu
Hi Spark Community,
I have a question regarding the support for User-Defined Functions (UDFs)
in Spark Connect, specifically when using Kubernetes as the Cluster Manager.
According to the Spark documentation, UDFs are supported by default for the
shell and in standalone applications with addition
