Moritz Manner created FLINK-41062:
-------------------------------------
Summary: PARSE_XML and TRY_PARSE_XML can run out of memory for
large documents
Key: FLINK-41062
URL: https://issues.apache.org/jira/browse/FLINK-41062
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Reporter: Moritz Manner
{{PARSE_XML}} reads the whole document into an intermediate tree before it
writes the {{{}VARIANT{}}}. The tree is a lot larger than the input: a 40 MB
document with 10 million empty elements needs between 2 and 3 GB of heap.
The parser limits the nesting depth, entity expansion, name length and number
of attributes, but not the size of the input or the number of elements. The 16
MB limit of a {{VARIANT}} is only checked when the variant is built, after the
tree is complete.
An {{OutOfMemoryError}} isn’t caught, so {{TRY_PARSE_XML}} doesn’t return
{{NULL}} for such a document, and the job fails. With enough heap, the same
document returns {{{}NULL{}}}, because the variant would be larger than 16 MB.
The generated code also keeps the last read document until the next record.
Query to reproduce:
{noformat}
SELECT TRY_PARSE_XML(CONCAT('<a>', REPEAT('<b/>', 10000000), '</a>'));{noformat}
With 2 GB of heap, this fails with {{{}java.lang.OutOfMemoryError: Java heap
space{}}}. With 3 GB, it returns {{{}NULL{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
