Hi Fabian, Good point, and I agree this is a real problem. A consistent VARIANT structure across documents with the same schema is important, especially for streaming SQL where you don't want CASE WHEN everywhere, or a job failing because it tries to access an array that doesn't exist. Initially, the structure optimized for queryability, i.e. a single element stays a single node instead of an array. But I think consistency should probably win here.
The idea would be to make ARRAY the default for all elements, so the structure stays the same no matter how many elements are actually there. The downside is that for simple XMLs, where most elements only ever appear once, you'd need to add [1] everywhere just to access them. One option would be to start with ARRAY by default and see how it goes based on feedback. If it turns out to be too cumbersome for simple XMLs, we could add an opt-out parameter later, that would be a backward-compatible addition so it doesn't need to be solved now. Does this make sense, or is there a better way to approach this? Best, Moritz On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <[email protected]> wrote: > Hi everyone and thanks for the proposal Moritz, > > The FLIP reads very well and I agree (mostly) with the design. > > I have only one concern, which is about the representation of repeated > elements as arrays. > I think it is very important that a batch of XML documents with the same > structure converts into an identical VARIANT structure. > If I understand the current proposal correctly, this would not be the case > for two documents: one with a single element and one with a repeated > element. > The first document would have a single node, while the second document > would have an array. > If you want to access the nodes, you would need a cumbersome CASE WHEN > construct to handle both cases. > It would be better to convert both documents into a VARIANT containing an > ARRAY (one only holding a single element). > > Since deriving the right conversion strategy from a single XML document > isn't possible, could we add an optional argument to pass conversion hints? > There might be other solutions to this problem. > > IMO, it would also be fine to continue with this FLIP and address this > issue later, but having a proposal for it would be good. > Do you have any thoughts on this, Moritz? > > Best, Fabian > > > On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <[email protected]> wrote: > > > Hi Moritz, > > > > this is an excellent design document with the right level of detail. I > > really like the design, it is a nice mixture of concerns. The result of > > PARSE_XML looks and feels like a VARIANT while the XML content is > > preserved in a lossless fashion and all attributes and text is > > accessible. It is a nice combination of what other vendors offer (modulo > > the historical legacy they have to deal with). > > > > Looking at various UDF implementations, XML handling is by far the most > > important reason for the need of a custom function. So +1 for this FLIP. > > > > Thanks, > > Timo > > > > On 11.09.26 14:54, Moritz Manner wrote: > > > Hey everyone, > > > > > > I'd like to start a discussion on FLIP-612: Native XML Functions for > > Flink > > > SQL [1]. > > > > > > The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT > > > instead of requiring custom UDFs. > > > > > > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an > > XML > > > string into a VARIANT, and XML_STRING turns a VARIANT back into an XML > > > string. > > > This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON, > and > > > JSON_STRING. > > > > > > Once XML is a VARIANT, you query it the same way as the output of > > PARSE_JSON: > > > with the variant accessors (variant.key, variant['key'], > variant[index]) > > > and a CAST to the type you need. > > > > > > The main part is the XML→VARIANT mapping: attributes and text go into > @/$ > > > fields, and since Variant objects store fields sorted by key rather > than > > in > > > document order, a # field is used to recover the original order between > > > differently-named siblings. Details and examples are in the FLIP. > > > > > > It's a small, additive API, no new SQL grammar or Table API methods > > needed. > > > > > > Looking forward to your thoughts! > > > > > > Best, > > > Moritz > > > > > > [1] > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$ > > > > >
