Hi Moritz, hi all, > The idea would be to make ARRAY the default for all elements, so the > structure stays the same no matter how many elements are actually there. > The downside is that for simple XMLs, where most elements only ever appear > once, you'd need to add [1] everywhere just to access them.
> One option would be to start with ARRAY by default and see how it goes > based on feedback. If it turns out to be too cumbersome for simple XMLs, we > could add an opt-out parameter later, that would be a backward-compatible > addition so it doesn't need to be solved now. > Does this make sense, or is there a better way to approach this? Sounds good to me, thanks! Best, Fabian On Thu, Sep 17, 2026 at 5:30 PM Krishna Sai <[email protected]> wrote: > Hi Moritz, > > Nice FLIP. The # field is a neat solution, and I checked on master that > Variant does sort object keys, so it is doing necessary work. > > One small question: what does PARSE_XML do with an xmlns declaration in > the input? The parser is not namespace-aware, so xmlns:xsi is an > attribute, and by the @name rule it would land as "@xmlns:xsi". If > XML_STRING writes that back and also adds its own xmlns:xsi declaration > for xsi:nil, the root would carry that attribute twice, which is not > well-formed, so that Variant would not round trip. > > <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> > <a xsi:nil="true"/> > </root> > > Dropping xmlns at parse time would instead leave other prefixes > undeclared. Maybe I am misreading the mapping, but a line either way > might help. > > Best, > Sai Krishna Sepuri > > On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <[email protected]> > wrote: > > > > Hi Fabian, > > > > Good point, and I agree this is a real problem. A consistent VARIANT > > structure across documents with the same schema is important, especially > > for streaming SQL where you don't want CASE WHEN everywhere, or a job > > failing because it tries to access an array that doesn't exist. > Initially, > > the structure optimized for queryability, i.e. a single element stays a > > single node instead of an array. But I think consistency should probably > > win here. > > > > The idea would be to make ARRAY the default for all elements, so the > > structure stays the same no matter how many elements are actually there. > > The downside is that for simple XMLs, where most elements only ever > appear > > once, you'd need to add [1] everywhere just to access them. > > > > One option would be to start with ARRAY by default and see how it goes > > based on feedback. If it turns out to be too cumbersome for simple XMLs, > we > > could add an opt-out parameter later, that would be a backward-compatible > > addition so it doesn't need to be solved now. > > Does this make sense, or is there a better way to approach this? > > > > Best, > > Moritz > > > > On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev < > [email protected]> > > wrote: > > > > > Hi everyone and thanks for the proposal Moritz, > > > > > > The FLIP reads very well and I agree (mostly) with the design. > > > > > > I have only one concern, which is about the representation of repeated > > > elements as arrays. > > > I think it is very important that a batch of XML documents with the > same > > > structure converts into an identical VARIANT structure. > > > If I understand the current proposal correctly, this would not be the > case > > > for two documents: one with a single element and one with a repeated > > > element. > > > The first document would have a single node, while the second document > > > would have an array. > > > If you want to access the nodes, you would need a cumbersome CASE WHEN > > > construct to handle both cases. > > > It would be better to convert both documents into a VARIANT containing > an > > > ARRAY (one only holding a single element). > > > > > > Since deriving the right conversion strategy from a single XML document > > > isn't possible, could we add an optional argument to pass conversion > hints? > > > There might be other solutions to this problem. > > > > > > IMO, it would also be fine to continue with this FLIP and address this > > > issue later, but having a proposal for it would be good. > > > Do you have any thoughts on this, Moritz? > > > > > > Best, Fabian > > > > > > > > > On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <[email protected]> > wrote: > > > > > > > Hi Moritz, > > > > > > > > this is an excellent design document with the right level of detail. > I > > > > really like the design, it is a nice mixture of concerns. The result > of > > > > PARSE_XML looks and feels like a VARIANT while the XML content is > > > > preserved in a lossless fashion and all attributes and text is > > > > accessible. It is a nice combination of what other vendors offer > (modulo > > > > the historical legacy they have to deal with). > > > > > > > > Looking at various UDF implementations, XML handling is by far the > most > > > > important reason for the need of a custom function. So +1 for this > FLIP. > > > > > > > > Thanks, > > > > Timo > > > > > > > > On 11.09.26 14:54, Moritz Manner wrote: > > > > > Hey everyone, > > > > > > > > > > I'd like to start a discussion on FLIP-612: Native XML Functions > for > > > > Flink > > > > > SQL [1]. > > > > > > > > > > The goal is native XML parsing in Flink SQL, parsing XML into a > VARIANT > > > > > instead of requiring custom UDFs. > > > > > > > > > > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML > turn an > > > > XML > > > > > string into a VARIANT, and XML_STRING turns a VARIANT back into an > XML > > > > > string. > > > > > This mirrors the existing JSON functions PARSE_JSON, > TRY_PARSE_JSON, > > > and > > > > > JSON_STRING. > > > > > > > > > > Once XML is a VARIANT, you query it the same way as the output of > > > > PARSE_JSON: > > > > > with the variant accessors (variant.key, variant['key'], > > > variant[index]) > > > > > and a CAST to the type you need. > > > > > > > > > > The main part is the XML→VARIANT mapping: attributes and text go > into > > > @/$ > > > > > fields, and since Variant objects store fields sorted by key rather > > > than > > > > in > > > > > document order, a # field is used to recover the original order > between > > > > > differently-named siblings. Details and examples are in the FLIP. > > > > > > > > > > It's a small, additive API, no new SQL grammar or Table API methods > > > > needed. > > > > > > > > > > Looking forward to your thoughts! > > > > > > > > > > Best, > > > > > Moritz > > > > > > > > > > [1] > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$ > > > > > > > > > > > >
