[
https://issues.apache.org/jira/browse/SPARK-20314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975744#comment-15975744
]
Eric Wasserman commented on SPARK-20314:
----------------------------------------
h2. Cause
The cause of the error appears to be a misuse of the Jackson JSON parser.
In {{org.apache.spark.sql.catalyst.expressions}} both {{JsonTuple}} and
{{GetJsonObject}} are following a pattern:
{code:scala}
val jsonStr = json.eval(input).asInstanceOf[UTF8String]
...
try {
Utils.tryWithResource(jsonFactory.createParser(jsonStr.getBytes)) {
parser =>
...
parser.nextToken
}
} catch {
case _: JsonProcessingException => null
}
{code}
Both are utilizing the Jackson {{com.fasterxml.jackson.core.JsonFactory}}
created in the {{org.apache.spark.sql.catalyst.expressions.jsonExpressions}}
package
{code:scala}
val jsonFactory = new JsonFactory()
{code}
By using the
{{public JsonParser createParser(byte[] data)}}
method of the {{JsonFactory}} rather than the
{{public JsonParser createParser(String content)}}
the Spark SQL JSON functions are causing Jackson to "sniff" the first few bytes
to determine the encoding of the string in the byte array. However, we *know*
the byte array is UTF-8 encoded (since it came from a
{{org.apache.spark.unsafe.types.UTF8String}}).
In the example above (with a mal-formed JSON document), Jackson correctly
follows [RFC-4627|https://www.ietf.org/rfc/rfc4627.txt] and tries to infer the
character encoding of the byte array. In doing so it *assumes* well-formed JSON
and is fooled into thinking the bytes are UTF-32 encoded. This
misinterpretation and the exception ensues.
h2. Cure
Simply changing the lines
{code:java}
jsonFactory.createParser(jsonStr.getBytes)
{code}
to
{code:java}
jsonFactory.createParser(jsonStr.toString)
{code}
in {{JsonTuple}} and {{GetJsonObject}} should fix the problem demonstrated in
this example.
> Inconsistent error handling in JSON parsing SQL functions
> ---------------------------------------------------------
>
> Key: SPARK-20314
> URL: https://issues.apache.org/jira/browse/SPARK-20314
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Eric Wasserman
>
> Most parse errors in the JSON parsing SQL functions (e.g. json_tuple,
> get_json_object) will return a null(s) if the JSON is badly formed. However,
> if Jackson determines that the string includes invalid characters it will
> throw an exception (java.io.CharConversionException: Invalid UTF-32
> character) that Spark does not catch. This creates a robustness problem in
> that these functions cannot be used at all when there may be dirty data as
> these exceptions will kill the jobs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
