twalthr commented on code in PR #29341:
URL: https://github.com/apache/flink/pull/29341#discussion_r4258444997
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
Review Comment:
```suggestion
The `PARSE_XML` function maps root element of an XML document to a VARIANT
object. For example:
```
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
+`PARSE_XML(xml, TRUE)`, every child element, text, and position in a `#` field
is stored as an array,
+even if it occurs only once. The root element is never wrapped, since a
document has exactly one.
+
+```sql
+-- {"book":{"title":["Dune"]}}
+PARSE_XML('<book><title>Dune</title></book>', TRUE)
+```
+
+An `xsi:type` attribute stores the text of an element as the matching kind,
which replaces the text,
Review Comment:
```suggestion
An `xsi:type` attribute parses the text of an element as the matching kind,
which replaces the text,
```
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
Review Comment:
```suggestion
so two documents that follow the same schema can map to different shapes.
The default behavior ensures a space-efficient representation. With
```
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
+`PARSE_XML(xml, TRUE)`, every child element, text, and position in a `#` field
is stored as an array,
+even if it occurs only once. The root element is never wrapped, since a
document has exactly one.
+
+```sql
+-- {"book":{"title":["Dune"]}}
+PARSE_XML('<book><title>Dune</title></book>', TRUE)
+```
+
+An `xsi:type` attribute stores the text of an element as the matching kind,
which replaces the text,
+or the `$` field if the element has other attributes. The prefix of the type
is ignored, so `xs:int`,
+`xsd:int`, and `int` are the same type. If the element has child elements or
no text, the type isn't
+listed below, or the text isn't a valid value of the type, the text stays a
`STRING` and `xsi:type`
+is kept as an attribute. For example, `<a xsi:type="int">5</a>` maps to
`{"a":5}`, and
+`<a xsi:type="int">five</a>` maps to `{"a":{"$":"five","@xsi:type":"int"}}`.
The text has to be in
+the lexical form that XML Schema defines for the type, so `1e5` isn't a
`decimal`, `1f` isn't a
+`float`, and `12:30` isn't a `time`. Years after 9999 and the time `24:00:00`
stay a `STRING` as well.
+
+| `xsi:type` | Stored `VARIANT` kind |
+|------------------------------|----------------------------------------------|
+| `string` | `STRING` |
+| `boolean` | `BOOLEAN`, from `true`, `false`, `1`, or `0` |
+| `byte` | `TINYINT` |
+| `short` | `SMALLINT` |
+| `int` | `INT` |
+| `long` | `BIGINT` |
+| `integer`, `decimal` | `DECIMAL`, with at most 38 digits |
+| `float` | `FLOAT`, except for `NaN` and infinity |
+| `double` | `DOUBLE`, except for `NaN` and infinity |
+| `date` without an offset | `DATE` |
+| `time` without an offset | `TIME` |
+| `dateTime` with an offset | `TIMESTAMP_LTZ` |
+| `dateTime` without an offset | `TIMESTAMP` |
+
+Fractional seconds beyond the precision of the kind are truncated, like in a
cast, instead of
+keeping the text as a `STRING`.
Review Comment:
this only applies to TIME type, right?
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/GeneratedCodeTestUtils.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.codegen;
+
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.runtime.operators.CodeGenOperatorFactory;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CollectionUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Pattern;
+
+/** Utilities for tests that check the code generated for a query. */
+final class GeneratedCodeTestUtils {
Review Comment:
Can we use BuiltInFunctionTestBase instead?
##########
docs/data/sql_functions.yml:
##########
@@ -1395,6 +1395,45 @@ variant:
- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
+ - sql: PARSE_XML(xml_string[, force_array])
+ table: STRING.parseXml([forceArray])
+ description: |
+ Parses an XML string into a Variant. If the XML string is invalid, an
error is thrown. To
+ return NULL instead of an error, use the `TRY_PARSE_XML` function.
+
+ The result is an object with a single field for the root element.
Attributes are stored as
+ `@name` fields, text as a `$` field, and child elements as fields named
after their tag. An
+ element without attributes and child elements is stored as its text
directly. A tag that
+ occurs more than once in an element is stored as an array. The `#` field
records the order
+ of the children and text of an element.
+
+ If `force_array` is true, every child element, text, and position in a
`#` field is stored
+ as an array, even if it occurs only once. This keeps the structure the
same across documents
+ that repeat an element a different number of times. The default value of
`force_array` is
+ false. A NULL `force_array` is treated as false.
+
+ External entities and external DTDs are not supported, and a document
that references one
+ fails to parse.
+
+ ```sql
+ --
{"book":{"#":{"price":1,"title":0},"@pages":"320","price":"12.50","title":"Dune"}}
+ PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price></book>')
+
+ -- 'Dune'
+ CAST(PARSE_XML('<book><title>Dune</title></book>')['book']['title'] AS
STRING)
+
+ -- {"book":{"title":["Dune"]}}
+ PARSE_XML('<book><title>Dune</title></book>', TRUE)
+ ```
+
+ See XML to VARIANT below the function table for how XML is mapped to a
Variant.
Review Comment:
```suggestion
See "XML to VARIANT" section below the function table for how XML is
mapped to a Variant.
```
##########
flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/functions/XmlToVariantParserTest.java:
##########
@@ -0,0 +1,586 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.variant.Variant;
+
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.xml.sax.SAXException;
+
+import javax.xml.transform.stream.StreamSource;
+import javax.xml.validation.SchemaFactory;
+
+import java.io.StringReader;
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.stream.Stream;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatNoException;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.params.provider.Arguments.arguments;
+
+/** Tests for {@link XmlToVariantParser}. */
+class XmlToVariantParserTest {
+
+ private final XmlToVariantParser parser = new XmlToVariantParser();
+
+ /** How elements, attributes, and text map to a variant. */
+ @Nested
+ @TestInstance(TestInstance.Lifecycle.PER_CLASS)
+ class Mapping {
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("mappings")
+ void testMapping(String xml, String expectedJson) {
+ assertThat(parser.parse(xml,
false).toJson()).isEqualTo(expectedJson);
+ }
+
+ Stream<Arguments> mappings() {
+ return Stream.of(
+ arguments("<title>Dune</title>", "{\"title\":\"Dune\"}"),
+ arguments("<title></title>", "{\"title\":\"\"}"),
+ arguments("<title/>", "{\"title\":\"\"}"),
+ arguments(
+ "<book pages=\"320\">\n Dune\n</book>",
+ "{\"book\":{\"$\":\"Dune\",\"@pages\":\"320\"}}"),
+ arguments("<book pages=\"320\"/>",
"{\"book\":{\"@pages\":\"320\"}}"),
+ arguments(
+ "<book>\n"
+ + " <author>Terry Pratchett</author>\n"
+ + " <author>Neil Gaiman</author>\n"
+ + "</book>",
+ "{\"book\":{\"author\":[\"Terry Pratchett\",\"Neil
Gaiman\"]}}"),
+ arguments(
+ "<book>\n <title>Dune</title>\n
<price>12.5</price>\n</book>",
+
"{\"book\":{\"#\":{\"price\":1,\"title\":0},\"price\":\"12.5\",\"title\":\"Dune\"}}"),
+ arguments(
+ "<book>\n"
+ + " This is a description\n"
+ + " <title>Dune</title>\n"
+ + " and this is another text\n"
+ + "</book>",
+ "{\"book\":{\"#\":{\"$\":[0,2],\"title\":1},"
+ + "\"$\":[\"This is a description\",\"and
this is another text\"],"
+ + "\"title\":\"Dune\"}}"),
+ arguments(
+ "<book pages=\"320\">\n"
+ + " <title>Dune</title>\n"
+ + " This is a description\n"
+ + " <price>12.50</price>\n"
+ + " Text in between\n"
+ + " <price
currency=\"EUR\">13.50</price>\n"
+ + "</book>",
+
"{\"book\":{\"#\":{\"$\":[1,3],\"price\":[2,4],\"title\":0},"
+ + "\"$\":[\"This is a description\",\"Text
in between\"],"
+ + "\"@pages\":\"320\","
+ +
"\"price\":[\"12.50\",{\"$\":\"13.50\",\"@currency\":\"EUR\"}],"
+ + "\"title\":\"Dune\"}}"),
+ arguments("<a><b><c>deep</c></b></a>",
"{\"a\":{\"b\":{\"c\":\"deep\"}}}"),
+ // Namespace prefixes and declarations are kept as written.
+ arguments(
+ "<ns:a xmlns:ns=\"urn:ns\" xmlns=\"urn:default\"
ns:id=\"1\">"
+ + "<ns:b>x</ns:b></ns:a>",
+
"{\"ns:a\":{\"@ns:id\":\"1\",\"@xmlns\":\"urn:default\","
+ +
"\"@xmlns:ns\":\"urn:ns\",\"ns:b\":\"x\"}}"),
+ // Attribute values are not trimmed.
+ arguments("<a b=\" x \"/>", "{\"a\":{\"@b\":\" x \"}}"),
+ // CDATA is text, and it merges with the text around it.
+ arguments("<a><![CDATA[<b>&</b>]]></a>",
"{\"a\":\"<b>&</b>\"}"),
+ arguments("<a>x<![CDATA[y]]>z</a>", "{\"a\":\"xyz\"}"),
+ // Comments and processing instructions are dropped, the
text around them
+ // merges.
+ arguments(
+ "<?xml version=\"1.0\"?><!-- c --><a>foo<!-- c
-->bar<?pi x?></a><!-- c -->",
+ "{\"a\":\"foobar\"}"),
+ // The input is a string, so an encoding declared in the
document is ignored.
+ arguments(
+ "<?xml version=\"1.0\"
encoding=\"ISO-8859-1\"?><a>\u00e4</a>",
+ "{\"a\":\"\u00e4\"}"),
+ // Whitespace-only text is dropped, other text is trimmed
of XML whitespace
+ // only.
+ arguments("<a>\n <b>x</b>\n</a>",
"{\"a\":{\"b\":\"x\"}}"),
+ arguments("<a> \t\r\n </a>", "{\"a\":\"\"}"),
+ arguments("<a>\u00a0x\u00a0</a>",
"{\"a\":\"\u00a0x\u00a0\"}"),
+ // Predefined entities and character references are
expanded, in text and in
+ // attribute values.
+ arguments("<a><>&'</a>",
"{\"a\":\"<>&'\"}"),
+ arguments("<a>AB</a>", "{\"a\":\"AB\"}"),
+ arguments("<a b=\"<A\"/>",
"{\"a\":{\"@b\":\"<A\"}}"),
+ // Entities declared in the DTD are expanded.
+ arguments(
+ "<!DOCTYPE a [<!ENTITY title
\"Dune\">]><a>&title;</a>",
+ "{\"a\":\"Dune\"}"));
+ }
+ }
+
+ /** How {@code xsi:nil} and {@code xsi:type} change the mapping. */
+ @Nested
+ @TestInstance(TestInstance.Lifecycle.PER_CLASS)
+ class XsiAttributes {
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("xsiMappings")
+ void testXsiMapping(String xml, String expectedJson) {
+ assertThat(parser.parse(xml,
false).toJson()).isEqualTo(expectedJson);
+ }
+
+ Stream<Arguments> xsiMappings() {
+ return Stream.of(
+ // xsi:nil makes an element without other attributes null,
and drops the
+ // content.
+ arguments("<a xsi:nil=\"true\"/>", "{\"a\":null}"),
+ arguments("<a xsi:nil=\"1\"></a>", "{\"a\":null}"),
+ arguments("<a xsi:nil=\"true\">x</a>", "{\"a\":null}"),
+ arguments("<a xsi:nil=\"true\"><b>x</b></a>",
"{\"a\":null}"),
+ arguments("<a xsi:nil=\" true \" xsi:type=\"int\"/>",
"{\"a\":null}"),
+ arguments("<a xsi:nil=\"true\" xsi:type=\"Dog\"/>",
"{\"a\":null}"),
+ // The declaration of the xsi prefix is dropped.
+ arguments(
+ "<root
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\";>"
+ + "<a xsi:nil=\"true\"/></root>",
+ "{\"root\":{\"a\":null}}"),
+ // xsi:nil that doesn't make the element null is kept,
unless it is false.
+ arguments(
+ "<a xsi:nil=\"true\" id=\"1\"/>",
+ "{\"a\":{\"@id\":\"1\",\"@xsi:nil\":\"true\"}}"),
+ arguments(
+ "<a xsi:nil=\"true\" id=\"1\">x</a>",
+ "{\"a\":{\"@id\":\"1\",\"@xsi:nil\":\"true\"}}"),
+ arguments(
+ "<a xsi:nil=\" 1 \" id=\"1\"/>",
+ "{\"a\":{\"@id\":\"1\",\"@xsi:nil\":\"true\"}}"),
+ arguments(
+ "<a xsi:nil=\"true\" xsi:type=\"string\"
id=\"1\"/>",
+
"{\"a\":{\"@id\":\"1\",\"@xsi:nil\":\"true\",\"@xsi:type\":\"string\"}}"),
+ arguments("<a xsi:nil=\"yes\"/>",
"{\"a\":{\"@xsi:nil\":\"yes\"}}"),
+ arguments("<a xsi:nil=\"TRUE\"/>",
"{\"a\":{\"@xsi:nil\":\"TRUE\"}}"),
+ arguments("<a xsi:nil=\"false\">x</a>", "{\"a\":\"x\"}"),
+ arguments("<a xsi:nil=\"0\"/>", "{\"a\":\"\"}"),
+ // xsi:type types the text of an element with text but
without child elements.
+ arguments("<a xsi:type=\"int\">5</a>", "{\"a\":5}"),
+ arguments("<a xsi:type=\"boolean\">true</a>",
"{\"a\":true}"),
+ arguments("<a xsi:type=\"decimal\">-1.23</a>",
"{\"a\":-1.23}"),
+ arguments(
+ "<price xsi:type=\"decimal\"
currency=\"EUR\">13.50</price>",
+ "{\"price\":{\"$\":13.5,\"@currency\":\"EUR\"}}"),
+ // xsi:type that doesn't type the text is kept.
+ arguments("<a xsi:type=\"int\"></a>",
"{\"a\":{\"@xsi:type\":\"int\"}}"),
+ arguments("<a xsi:type=\"string\"/>",
"{\"a\":{\"@xsi:type\":\"string\"}}"),
+ arguments(
+ "<a id=\"1\" xsi:type=\"string\"/>",
+
"{\"a\":{\"@id\":\"1\",\"@xsi:type\":\"string\"}}"),
+ arguments(
+ "<a xsi:type=\"int\">five</a>",
+ "{\"a\":{\"$\":\"five\",\"@xsi:type\":\"int\"}}"),
+ arguments(
+ "<a xsi:type=\"xs:token\">x</a>",
+
"{\"a\":{\"$\":\"x\",\"@xsi:type\":\"xs:token\"}}"),
+ arguments(
+ "<animal xsi:type=\"Dog\" name=\"Rex\"/>",
+
"{\"animal\":{\"@name\":\"Rex\",\"@xsi:type\":\"Dog\"}}"),
+ arguments(
+ "<shape
xsi:type=\"Circle\"><radius>1</radius></shape>",
+
"{\"shape\":{\"@xsi:type\":\"Circle\",\"radius\":\"1\"}}"),
+ // Other xsi: attributes are regular attributes.
+ arguments(
+ "<a
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"; "
+ + "xsi:schemaLocation=\"urn:a
a.xsd\">x</a>",
+
"{\"a\":{\"$\":\"x\",\"@xsi:schemaLocation\":\"urn:a a.xsd\"}}"));
+ }
+ }
+
+ /** Which texts {@code xsi:type} stores with their type. */
+ @Nested
+ @TestInstance(TestInstance.Lifecycle.PER_CLASS)
+ class XsiTypes {
+
+ @ParameterizedTest(name = "{0}: {1}")
+ @MethodSource("typedValues")
+ void testXsiType(
+ String xsiType, String text, Variant.Type expectedType, Object
expectedValue) {
+ final Variant value = parseTyped(xsiType, text);
+ assertThat(value.getType()).isEqualTo(expectedType);
+ if (expectedValue instanceof BigDecimal) {
+
assertThat(value.getDecimal()).isEqualByComparingTo((BigDecimal) expectedValue);
+ } else {
+ assertThat(value.get()).isEqualTo(expectedValue);
+ }
+ }
+
+ Stream<Arguments> typedValues() {
+ return Stream.of(
+ arguments("string", "5", Variant.Type.STRING, "5"),
+ arguments("boolean", "true", Variant.Type.BOOLEAN, true),
+ arguments("boolean", "false", Variant.Type.BOOLEAN, false),
+ arguments("boolean", "1", Variant.Type.BOOLEAN, true),
+ arguments("boolean", "0", Variant.Type.BOOLEAN, false),
+ arguments("byte", "-128", Variant.Type.TINYINT, (byte)
-128),
+ arguments("short", "32767", Variant.Type.SMALLINT, (short)
32767),
+ arguments("int", "+5", Variant.Type.INT, 5),
+ arguments("int", " 5 ", Variant.Type.INT, 5),
+ arguments("long", "9223372036854775807",
Variant.Type.BIGINT, Long.MAX_VALUE),
+ arguments(
+ "integer",
+ "123456789012345678901234567890",
+ Variant.Type.DECIMAL,
+ new BigDecimal("123456789012345678901234567890")),
+ arguments("decimal", "12.50", Variant.Type.DECIMAL, new
BigDecimal("12.50")),
+ arguments("decimal", ".5", Variant.Type.DECIMAL, new
BigDecimal("0.5")),
+ // A sign is allowed, also for zero.
+ arguments("decimal", "-0.0", Variant.Type.DECIMAL, new
BigDecimal("0.0")),
+ arguments("float", "+0.0", Variant.Type.FLOAT, 0.0f),
+ arguments("double", "-0.0", Variant.Type.DOUBLE, -0.0d),
+ arguments("float", "1.5", Variant.Type.FLOAT, 1.5f),
+ arguments("double", "1.5E3", Variant.Type.DOUBLE, 1500.0),
+ arguments("double", "-.5e+2", Variant.Type.DOUBLE, -50.0),
+ arguments("date", "2026-09-23", Variant.Type.DATE,
LocalDate.of(2026, 9, 23)),
+ arguments(
+ "time",
+ "12:30:45.123456",
+ Variant.Type.TIME,
+ LocalTime.of(12, 30, 45, 123_456_000)),
+ // Precision that doesn't fit is dropped, like in a cast.
+ arguments(
+ "time",
+ "12:30:45.123456789",
+ Variant.Type.TIME,
+ LocalTime.of(12, 30, 45, 123_456_000)),
+ arguments(
+ "time",
+ "12:30:45.1234567890123",
+ Variant.Type.TIME,
+ LocalTime.of(12, 30, 45, 123_456_000)),
+ arguments(
+ "dateTime",
+ "2026-09-23T12:30:00.1234567890123456789",
+ Variant.Type.TIMESTAMP_NS,
+ LocalDateTime.of(2026, 9, 23, 12, 30, 0,
123_456_789)),
+ arguments(
Review Comment:
add the maximum that variant supports as a test for all time types
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
+`PARSE_XML(xml, TRUE)`, every child element, text, and position in a `#` field
is stored as an array,
+even if it occurs only once. The root element is never wrapped, since a
document has exactly one.
+
+```sql
+-- {"book":{"title":["Dune"]}}
+PARSE_XML('<book><title>Dune</title></book>', TRUE)
+```
+
+An `xsi:type` attribute stores the text of an element as the matching kind,
which replaces the text,
+or the `$` field if the element has other attributes. The prefix of the type
is ignored, so `xs:int`,
+`xsd:int`, and `int` are the same type. If the element has child elements or
no text, the type isn't
+listed below, or the text isn't a valid value of the type, the text stays a
`STRING` and `xsi:type`
+is kept as an attribute. For example, `<a xsi:type="int">5</a>` maps to
`{"a":5}`, and
+`<a xsi:type="int">five</a>` maps to `{"a":{"$":"five","@xsi:type":"int"}}`.
The text has to be in
+the lexical form that XML Schema defines for the type, so `1e5` isn't a
`decimal`, `1f` isn't a
+`float`, and `12:30` isn't a `time`. Years after 9999 and the time `24:00:00`
stay a `STRING` as well.
+
+| `xsi:type` | Stored `VARIANT` kind |
+|------------------------------|----------------------------------------------|
+| `string` | `STRING` |
+| `boolean` | `BOOLEAN`, from `true`, `false`, `1`, or `0` |
+| `byte` | `TINYINT` |
+| `short` | `SMALLINT` |
+| `int` | `INT` |
+| `long` | `BIGINT` |
+| `integer`, `decimal` | `DECIMAL`, with at most 38 digits |
+| `float` | `FLOAT`, except for `NaN` and infinity |
+| `double` | `DOUBLE`, except for `NaN` and infinity |
+| `date` without an offset | `DATE` |
+| `time` without an offset | `TIME` |
+| `dateTime` with an offset | `TIMESTAMP_LTZ` |
+| `dateTime` without an offset | `TIMESTAMP` |
+
+Fractional seconds beyond the precision of the kind are truncated, like in a
cast, instead of
+keeping the text as a `STRING`.
Review Comment:
Or maybe I don't understand this part.
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
+`PARSE_XML(xml, TRUE)`, every child element, text, and position in a `#` field
is stored as an array,
+even if it occurs only once. The root element is never wrapped, since a
document has exactly one.
+
+```sql
+-- {"book":{"title":["Dune"]}}
+PARSE_XML('<book><title>Dune</title></book>', TRUE)
+```
+
+An `xsi:type` attribute stores the text of an element as the matching kind,
which replaces the text,
+or the `$` field if the element has other attributes. The prefix of the type
is ignored, so `xs:int`,
+`xsd:int`, and `int` are the same type. If the element has child elements or
no text, the type isn't
+listed below, or the text isn't a valid value of the type, the text stays a
`STRING` and `xsi:type`
+is kept as an attribute. For example, `<a xsi:type="int">5</a>` maps to
`{"a":5}`, and
+`<a xsi:type="int">five</a>` maps to `{"a":{"$":"five","@xsi:type":"int"}}`.
The text has to be in
+the lexical form that XML Schema defines for the type, so `1e5` isn't a
`decimal`, `1f` isn't a
+`float`, and `12:30` isn't a `time`. Years after 9999 and the time `24:00:00`
stay a `STRING` as well.
+
+| `xsi:type` | Stored `VARIANT` kind |
+|------------------------------|----------------------------------------------|
+| `string` | `STRING` |
+| `boolean` | `BOOLEAN`, from `true`, `false`, `1`, or `0` |
+| `byte` | `TINYINT` |
+| `short` | `SMALLINT` |
+| `int` | `INT` |
+| `long` | `BIGINT` |
+| `integer`, `decimal` | `DECIMAL`, with at most 38 digits |
+| `float` | `FLOAT`, except for `NaN` and infinity |
+| `double` | `DOUBLE`, except for `NaN` and infinity |
+| `date` without an offset | `DATE` |
+| `time` without an offset | `TIME` |
+| `dateTime` with an offset | `TIMESTAMP_LTZ` |
Review Comment:
Can you give an example here. What kind of offsets do we support?
##########
docs/content/docs/sql/functions/built-in-functions.md:
##########
@@ -94,6 +94,139 @@ Known Limitations:
{{< sql_functions "variant" >}}
+#### XML to VARIANT
+
+The `PARSE_XML` function maps an XML document to an object with a single field
named after the root
+element. For example:
+
+```sql
+--
{"book":{"#":{"price":[1,2],"title":0},"@pages":"320","price":["12.50","13.50"],"title":"Dune"}}
+PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price><price>13.50</price></book>')
+```
+
+Attributes become fields with an `@` prefix, and a child element that occurs
more than once becomes
+an array. The `#` field records the order of the children, as described below.
It can be ignored
+when reading fields.
+
+Values are strings, unless the document declares a type with `xsi:type`, see
below. Since a cast
+from a `VARIANT` never parses a string, cast a value to `STRING` first and
convert it with a regular
+cast:
+
+```sql
+CAST(v['book']['title'] AS STRING) -- 'Dune'
+CAST(v['book']['price'][2] AS STRING) -- '13.50'
+CAST(CAST(v['book']['@pages'] AS STRING) AS INT) -- 320
+```
+
+Each element is mapped by the following rules:
+
+| XML input | Stored
`VARIANT` value |
+|---------------------------------------------------------------------|----------------------------------------------|
+| Element without attributes and child elements | its
text as `STRING`, or `""` if it is empty |
+| Any other element | object
|
+| Attribute `name` | field
`@name`, a `STRING` |
+| Child element `name` | field
`name` |
+| Text of an element with attributes or child elements | field
`$` |
+| Child element or text that occurs more than once | array,
in document order |
+| Element with differently named children, or with children and text | field
`#` with the order of its content |
+
+A `VARIANT` object doesn't keep the order of its fields, so the `#` field
records it. It maps each
+key to the position of its item, or to an array of positions if the key occurs
more than once.
+Positions count the child elements and texts of the element, starting at 0.
For example:
+
+```sql
+-- {"book":{"#":{"$":[1,3],"price":[2,4],"title":0},
+-- "$":["This is a description","Text in between"],
+-- "@pages":"320",
+-- "price":["12.50",{"$":"13.50","@currency":"EUR"}],
+-- "title":"Dune"}}
+PARSE_XML('
+ <book pages="320">
+ <title>Dune</title>
+ This is a description
+ <price>12.50</price>
+ Text in between
+ <price currency="EUR">13.50</price>
+ </book>')
+```
+
+Here, `title` is the first item, the two texts are the second and the fourth,
and the two `price`
+elements are the third and the fifth. The second `price` has an attribute, so
it is stored as an
+object, while the first one is a string. An element whose content has only one
key has no `#` field,
+since the order of an array is the document order.
+
+Whether a child element is stored as a single value or as an array depends on
how often it occurs,
+so two documents that follow the same schema can map to different shapes. With
+`PARSE_XML(xml, TRUE)`, every child element, text, and position in a `#` field
is stored as an array,
+even if it occurs only once. The root element is never wrapped, since a
document has exactly one.
+
+```sql
+-- {"book":{"title":["Dune"]}}
+PARSE_XML('<book><title>Dune</title></book>', TRUE)
+```
+
+An `xsi:type` attribute stores the text of an element as the matching kind,
which replaces the text,
+or the `$` field if the element has other attributes. The prefix of the type
is ignored, so `xs:int`,
+`xsd:int`, and `int` are the same type. If the element has child elements or
no text, the type isn't
+listed below, or the text isn't a valid value of the type, the text stays a
`STRING` and `xsi:type`
+is kept as an attribute. For example, `<a xsi:type="int">5</a>` maps to
`{"a":5}`, and
+`<a xsi:type="int">five</a>` maps to `{"a":{"$":"five","@xsi:type":"int"}}`.
The text has to be in
+the lexical form that XML Schema defines for the type, so `1e5` isn't a
`decimal`, `1f` isn't a
+`float`, and `12:30` isn't a `time`. Years after 9999 and the time `24:00:00`
stay a `STRING` as well.
+
+| `xsi:type` | Stored `VARIANT` kind |
+|------------------------------|----------------------------------------------|
+| `string` | `STRING` |
+| `boolean` | `BOOLEAN`, from `true`, `false`, `1`, or `0` |
+| `byte` | `TINYINT` |
+| `short` | `SMALLINT` |
+| `int` | `INT` |
+| `long` | `BIGINT` |
+| `integer`, `decimal` | `DECIMAL`, with at most 38 digits |
+| `float` | `FLOAT`, except for `NaN` and infinity |
+| `double` | `DOUBLE`, except for `NaN` and infinity |
+| `date` without an offset | `DATE` |
+| `time` without an offset | `TIME` |
+| `dateTime` with an offset | `TIMESTAMP_LTZ` |
+| `dateTime` without an offset | `TIMESTAMP` |
+
+Fractional seconds beyond the precision of the kind are truncated, like in a
cast, instead of
+keeping the text as a `STRING`.
+
+A typed value casts directly to its type, without the cast to `STRING`:
+
+```sql
+CAST(PARSE_XML('<a xsi:type="int">5</a>')['a'] AS INT)
-- 5
+CAST(PARSE_XML('<a xsi:type="dateTime">2026-09-30T12:30:00</a>')['a'] AS
TIMESTAMP(3)) -- 2026-09-30 12:30:00.000
+```
+
+An `xsi:nil="true"` attribute takes precedence over the content of an element,
so
+`<price xsi:nil="true">12.50</price>` maps to `{"price":null}`. If the element
has other attributes,
+only its content is dropped, and it keeps `@xsi:nil` and `@xsi:type` next to
the other attributes.
Review Comment:
why do we keep the attribute if it has been applied to content?
##########
docs/data/sql_functions.yml:
##########
@@ -1395,6 +1395,45 @@ variant:
- sql: JSON_STRING(variant)
description: |
Generate a json string from a Variant object.
+ - sql: PARSE_XML(xml_string[, force_array])
+ table: STRING.parseXml([forceArray])
+ description: |
+ Parses an XML string into a Variant. If the XML string is invalid, an
error is thrown. To
+ return NULL instead of an error, use the `TRY_PARSE_XML` function.
+
+ The result is an object with a single field for the root element.
Attributes are stored as
+ `@name` fields, text as a `$` field, and child elements as fields named
after their tag. An
+ element without attributes and child elements is stored as its text
directly. A tag that
+ occurs more than once in an element is stored as an array. The `#` field
records the order
+ of the children and text of an element.
+
+ If `force_array` is true, every child element, text, and position in a
`#` field is stored
+ as an array, even if it occurs only once. This keeps the structure the
same across documents
+ that repeat an element a different number of times. The default value of
`force_array` is
+ false. A NULL `force_array` is treated as false.
+
+ External entities and external DTDs are not supported, and a document
that references one
+ fails to parse.
+
+ ```sql
+ --
{"book":{"#":{"price":1,"title":0},"@pages":"320","price":"12.50","title":"Dune"}}
+ PARSE_XML('<book
pages="320"><title>Dune</title><price>12.50</price></book>')
+
+ -- 'Dune'
+ CAST(PARSE_XML('<book><title>Dune</title></book>')['book']['title'] AS
STRING)
+
+ -- {"book":{"title":["Dune"]}}
+ PARSE_XML('<book><title>Dune</title></book>', TRUE)
+ ```
+
+ See XML to VARIANT below the function table for how XML is mapped to a
Variant.
+ - sql: TRY_PARSE_XML(xml_string[, force_array])
+ table: STRING.tryParseXml([forceArray])
+ description: |
+ Parses an XML string into a Variant. If the XML string is invalid,
returns NULL. To throw an
+ error instead of returning NULL, use the `PARSE_XML` function.
+
+ See `PARSE_XML` for the meaning of `force_array`.
Review Comment:
```suggestion
See `PARSE_XML` for the full documentation.
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/XmlReadReuseTest.java:
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.codegen;
+
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.api.config.OptimizerConfigOptions;
+import org.apache.flink.table.api.config.TableConfigOptions;
+import org.apache.flink.table.codesplit.JavaCodeSplitter;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.runtime.functions.SqlXmlUtils;
+import org.apache.flink.types.Row;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests that the {@code PARSE_XML} and {@code TRY_PARSE_XML} calls on the
same input share it. */
+class XmlReadReuseTest {
+
+ private static final Pattern READ_PATTERN =
+ Pattern.compile(Pattern.quote(SqlXmlUtils.class.getCanonicalName()
+ ".read("));
+ private static final Pattern PARSE_XML_PATTERN =
+ Pattern.compile(Pattern.quote(SqlXmlUtils.class.getCanonicalName()
+ ".parseXml("));
+
+ private StreamTableEnvironment tEnv;
+
+ @AfterEach
+ void clearData() {
+ TestValuesTableFactory.clearAllData();
+ }
+
+ @BeforeEach
+ void setUp() {
+ tEnv =
+ StreamTableEnvironment.create(
+ StreamExecutionEnvironment.getExecutionEnvironment(),
+ EnvironmentSettings.inStreamingMode());
+ tEnv.createTemporaryView(
+ "xml_src",
+ tEnv.fromValues(
+ Row.of(
+ "<book><title>Dune</title></book>",
+ "<a>1</a>",
+ "{\"a\":1}",
+ "<book>"),
+ Row.of(
+ "<book><title>Emma</title></book>",
+ "<a>2</a>",
+ "<a/>",
+ "<a>"))
+ .as("xml", "other_xml", "json", "invalid_xml"));
+ }
+
+ @Test
+ void testCallsOnTheSameInputReadItOnce() {
+ final String sql =
+ "SELECT CAST(PARSE_XML(xml)['book']['title'] AS STRING), "
+ + "JSON_STRING(PARSE_XML(xml, TRUE)), "
+ + "JSON_STRING(TRY_PARSE_XML(xml, TRUE)) FROM xml_src";
+ assertThat(collect(sql))
+ .containsExactlyInAnyOrder(
+ Row.of(
+ "Dune",
+ "{\"book\":{\"title\":[\"Dune\"]}}",
+ "{\"book\":{\"title\":[\"Dune\"]}}"),
+ Row.of(
+ "Emma",
+ "{\"book\":{\"title\":[\"Emma\"]}}",
+ "{\"book\":{\"title\":[\"Emma\"]}}"));
+ assertThat(countReads(sql)).isOne();
+ }
+
+ @Test
+ void testFilterAndProjectionReadOnce() {
+ final String sql =
+ "SELECT CAST(PARSE_XML(xml)['book']['title'] AS STRING) FROM
xml_src "
+ + "WHERE CAST(PARSE_XML(xml, TRUE)['book']['title'][1]
AS STRING) <> 'Emma'";
+ assertThat(collect(sql)).containsExactly(Row.of("Dune"));
+ assertThat(countReads(sql)).isOne();
+ }
+
+ @Test
+ void testCallsOnDifferentInputsReadEachOnce() {
+ final String sql =
+ "SELECT JSON_STRING(PARSE_XML(xml)),
JSON_STRING(TRY_PARSE_XML(xml, TRUE)), "
+ + "JSON_STRING(PARSE_XML(other_xml)), "
+ + "JSON_STRING(TRY_PARSE_XML(other_xml, TRUE)) FROM
xml_src";
+ assertThat(collect(sql))
+ .containsExactlyInAnyOrder(
+ Row.of(
+ "{\"book\":{\"title\":\"Dune\"}}",
+ "{\"book\":{\"title\":[\"Dune\"]}}",
+ "{\"a\":\"1\"}",
+ "{\"a\":\"1\"}"),
+ Row.of(
+ "{\"book\":{\"title\":\"Emma\"}}",
+ "{\"book\":{\"title\":[\"Emma\"]}}",
+ "{\"a\":\"2\"}",
+ "{\"a\":\"2\"}"));
+ assertThat(countReads(sql)).isEqualTo(2);
+ }
+
+ @Test
+ void testFirstCallInBranchThatIsNotTaken() {
+ // The first call is never taken, so the second one has to read the
document.
+ final String sql =
+ "SELECT CASE WHEN CHARACTER_LENGTH(other_xml) > 100 "
+ + "THEN CAST(PARSE_XML(xml)['book']['title'] AS
STRING) ELSE 'none' END, "
+ + "CAST(PARSE_XML(xml, TRUE)['book']['title'][1] AS
STRING) FROM xml_src";
+ assertThat(collect(sql))
+ .containsExactlyInAnyOrder(Row.of("none", "Dune"),
Row.of("none", "Emma"));
+ assertThat(countCalls(sql, PARSE_XML_PATTERN))
+ .as("the branch with the first call must survive the
optimizer")
+ .isEqualTo(2);
+ assertThat(countReads(sql)).isOne();
+ }
+
+ @Test
+ void testInvalidDocumentIsReadOnce() {
+ final String sql =
+ "SELECT JSON_STRING(TRY_PARSE_XML(invalid_xml)), "
+ + "JSON_STRING(TRY_PARSE_XML(invalid_xml, TRUE)) FROM
xml_src";
+ assertThat(collect(sql)).containsExactlyInAnyOrder(Row.of(null, null),
Row.of(null, null));
+ assertThat(countReads(sql)).isOne();
+ }
+
+ @Test
+ void testJsonFunctionOnTheSameInput() {
+ // The JSON functions share their parsed input as well, which must not
be mixed up.
+ final String sql =
+ "SELECT JSON_VALUE(json, '$.a'),
JSON_STRING(TRY_PARSE_XML(json)) FROM xml_src";
+ assertThat(collect(sql))
+ .containsExactlyInAnyOrder(Row.of("1", null), Row.of(null,
"{\"a\":\"\"}"));
+ assertThat(countReads(sql)).isOne();
+ }
+
+ @Test
+ void testReuseSurvivesCodeSplitting() {
+ tEnv.getConfig()
+ .set(TableConfigOptions.MAX_LENGTH_GENERATED_CODE, 1)
+ .set(TableConfigOptions.MAX_MEMBERS_GENERATED_CODE, 1);
+ final String sql =
+ "SELECT CAST(PARSE_XML(xml)['book']['title'] AS STRING), "
+ + "JSON_STRING(PARSE_XML(xml, TRUE)), "
+ + "JSON_STRING(TRY_PARSE_XML(xml, TRUE)) FROM xml_src";
+ // Only the split code is compiled, so correct results show that the
reuse survives it.
+ assertThat(collect(sql))
+ .containsExactlyInAnyOrder(
+ Row.of(
+ "Dune",
+ "{\"book\":{\"title\":[\"Dune\"]}}",
+ "{\"book\":{\"title\":[\"Dune\"]}}"),
+ Row.of(
+ "Emma",
+ "{\"book\":{\"title\":[\"Emma\"]}}",
+ "{\"book\":{\"title\":[\"Emma\"]}}"));
+ final List<String> splitCodes =
+ GeneratedCodeTestUtils.generatedClassCodes(tEnv, sql).stream()
+ .map(code -> JavaCodeSplitter.split(code, 1, 1))
+ .collect(Collectors.toList());
+ assertThat(GeneratedCodeTestUtils.generatedClassCodes(tEnv, sql))
+ .as("the limits must actually split a generated class")
+ .anySatisfy(
+ code -> assertThat(JavaCodeSplitter.split(code, 1,
1)).isNotEqualTo(code));
+ assertThat(
+ splitCodes.stream()
+ .mapToInt(
+ code ->
+
GeneratedCodeTestUtils.countMatches(
+ READ_PATTERN, code))
+ .sum())
+ .isOne();
+ }
+
+ @Test
+ void testDocumentIsReadForEachRowInMatchRecognize() {
Review Comment:
This is a lot of tests. Can we reduce the number? I feel this tests test the
general functionality of code generation reuse, not specifically parse_xml.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/codegen/XmlCodeGenUtils.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.codegen;
+
+import org.apache.flink.table.planner.utils.JavaScalaConversionUtil;
+import org.apache.flink.table.runtime.functions.SqlXmlUtils;
+import org.apache.flink.table.runtime.functions.XmlToVariantParser;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.util.Collections;
+
+import scala.Option;
+import scala.Tuple2;
+import scala.collection.Seq;
+
+/**
+ * Code generation of {@code PARSE_XML} and {@code TRY_PARSE_XML}.
Review Comment:
Do we really need code generation. Or would a regular Java function work as
well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
