That's a parquet library error. It might be this:
https://issues.apache.org/jira/browse/PARQUET-1633 That's fixed in recent
versions of Parquet. You didn't say what versions of libraries you are
using, but try the latest Spark.
On Mon, May 9, 2022 at 8:49 AM wrote:
> # python:
>
> import pandas
# python:
import pandas as pd
a = pd.DataFrame([[1, [2.3, 1.2]]], columns=['a', 'b'])
a.to_parquet('a.parquet')
# pyspark:
d2 = spark.read.parquet('a.parquet')
will return error:
An error was encountered: An error occurred while calling o277.showString. :
org.apache.spark.SparkException: Job
