[bug] Scala reflection "assertion failed: class Byte" in Dataset.toJSON

[Brandon Vincent]

Hi all,

I have a job that executes a query and collects the results as JSON using
Dataset.toJSON. For the most part it is stable, but sometimes it fails
randomly with a scala assertion error. Here is the stack trace:

                                      org.apache.spark.sql.Dataset.toJSON
         Dataset.scala: 3222
                                    org.apache.spark.sql.Encoders$.STRING
        Encoders.scala:   96
          org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply
 ExpressionEncoder.scala:   72
           org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor
 ScalaReflection.scala:  161
           org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor
 ScalaReflection.scala:  173
  org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects
 ScalaReflection.scala:   49
  org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$
 ScalaReflection.scala:  925
   org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects
 ScalaReflection.scala:  926
                  scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo
 TypeConstraints.scala:   68
org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$deserializerFor$1
 ScalaReflection.scala:  260
               org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf
 ScalaReflection.scala:   49
               org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf$
 ScalaReflection.scala:  939
                org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf
 ScalaReflection.scala:  941
                           scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe
        TypeTags.scala:  237
                scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute
        TypeTags.scala:  237
      org.apache.spark.sql.catalyst.ScalaReflection$$typecreator7$1.apply
 ScalaReflection.scala:  260
              scala.reflect.internal.Symbols$TypeSymbol.toTypeConstructor
         Symbols.scala: 3081
    scala.reflect.internal.Symbols$SymbolContextApiImpl.toTypeConstructor
         Symbols.scala:  194
                scala.reflect.internal.Symbols$TypeSymbol.typeConstructor
         Symbols.scala: 3154
                  scala.reflect.internal.Symbols$TypeSymbol.setTyconCache
         Symbols.scala: 3163
                   scala.reflect.internal.SymbolTable.throwAssertionError
     SymbolTable.scala:  183
java.lang.AssertionError: assertion failed: class Byte

It can also come up with "class Boolean" with the same stack trace.
Any clues on this? I wasn't able to find information about this specific
assertion error.

The spark version is 2.4.4 compiled with scala 2.12.

Thank you.