Thanks all for the answer. Unfortunately while I wasn’t able to use the
extra parameters to get the needed information, I did solve my issue. It
was an issue of using pureconfig to read a certain config from hadoop
before the spark session initialized, therefore pureconfig would error out
in deseri
Hi Ruijing,
Spark uses SerializationDebugger (
https://spark.apache.org/docs/latest/api/java/org/apache/spark/serializer/SerializationDebugger.html)
as default debugger to detect the serialization issues. You can take more
detailed serialization exception information by setting the following while
