Hi 2016-09-23 10:05 GMT+02:00 Craig Ringer <cr...@2ndquadrant.com>:
> On 22 September 2016 at 02:31, Pavel Stehule <pavel.steh...@gmail.com>
> wrote:
>
> > another small update - fix XMLPath parser - support multibytes characters
>
> I'm returning for another round of review.
>
> The code doesn't handle the 5 XML built-in entities correctly in
> text-typed output. It processes ' and " but not &, < or
> > . See added test. I have not fixed this, but I think it's clearly
> broken:
>
>
> + -- XML builtin entities
> + SELECT * FROM xmltable('/x/a' PASSING
> '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><
> ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>'
> COLUMNS ent text);
> + ent
> + -------
> + '
> + "
> + &
> + <
> + >
> + (5 rows)
>
> so I've adjusted the docs to claim that they're expanded. The code
> needs fixing to avoid entity-escaping when the output column type is
> not 'xml'.
>
>
fixed
>
> ' and " entities in xml-typed output are expanded, not
> preserved. I don't know if this is intended but I suspect it is:
>
> + SELECT * FROM xmltable('/x/a' PASSING
> '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><
> ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>'
> COLUMNS ent xml);
> + ent
> + ------------------
> + <ent>'</ent>
> + <ent>"</ent>
> + <ent>&</ent>
> + <ent><</ent>
> + <ent>></ent>
> + (5 rows)
>
>
> For the docs changes relevant to the above search for "The five
> predefined XML entities". Adjust that bit of docs if I guessed wrong
> about the intended behaviour.
>
> The tests don't cover CDATA or PCDATA . I didn't try to add that, but
> they should.
>
>
appended
> Did some docs copy-editing and integrated some examples. Explained how
> nested elements work, that multiple top level elements is an error,
> etc. Explained the time-of-evaluation stuff. Pointed out that you can
> refer to prior output columns in PATH and DEFAULT, since that's weird
> and unusual compared to normal SQL. Documented handling of multiple
> node matches, including the surprising results of somepath/text() on
> <somepath>x<!--blah-->y</somepath>. Documented handling of nested
> elements. Documented that xmltable works only on XML documents, not
> fragments/forests.
>
I don't understand to this sentence: "It is possible for a PATH expression
to reference output columns that appear before it in the column-list, so
paths may be dynamically constructed based on other parts of the XML
document:"
> Regarding evaluation time, it struck me that evaluating path
> expressions once per row means the xpath must be parsed and processed
> once per row. Isn't it desirable to store and re-use the preparsed
> xpath? I don't think this is a major problem, since we can later
> detect stable/immutable expressions including constants, evaluate only
> once in that case, and cache. It's just worth thinking about.
>
Probably it is possible - it is exactly how you wrote - it needs to check
the change. We can try do some possible performance optimizations later -
without compatibility issues. Now, I prefer the most simple code.
a note: PATH expression is evaluated for any **input** row. In same moment
is processed row path expression and man XML document DOM parsing. So
overhead of PATH expression and PATH parsing should not be dominant.
>
> The docs and tests don't seem to cover XML entities. What's the
> behaviour there? Core XML only defines one entity, but if a schema
> defines more how are they processed? The tests need to cover the
> predefined entities " & ' < and > at least.
>
I don't understand, what you are propose here. ?? Please, can you send some
examples.
>
> I have no idea whether the current code can fetch a DTD and use any
> <!ENTITY > declarations to expand entities, but I'm guessing not? If
> not, external DTDs, and internal DTDs with external entities should be
> documented as unsupported.
>
> It doesn't seem to cope with internal DTDs at all (libxml2 limitation?):
>
> SELECT * FROM xmltable('/' PASSING $XML$<?xml version="1.0"
> standalone="yes" ?>
> <!DOCTYPE foo [
> <!ELEMENT foo (#PCDATA)>
> <!ENTITY pg "PostgreSQL">
> ]>
> <foo>Hello &pg;.</foo>
> $XML$ COLUMNS foo text);
>
> + ERROR: invalid XML content
> + LINE 1: SELECT * FROM xmltable('/' PASSING $XML$<?xml version="1.0" ...
> + ^
> + DETAIL: line 2: StartTag: invalid element name
> + <!DOCTYPE foo [
> + ^
> + line 3: StartTag: invalid element name
> + <!ELEMENT foo (#PCDATA)>
> + ^
> + line 4: StartTag: invalid element name
> + <!ENTITY pg "PostgreSQL">
> + ^
> + line 6: Entity 'pg' not defined
> + <foo>Hello &pg;.</foo>
> + ^
>
>
It is rejected before XMLTABLE function call
postgres=# select $XML$<?xml version="1.0" standalone="yes" ?>
postgres$# <!DOCTYPE foo [
postgres$# <!ELEMENT foo (#PCDATA)>
postgres$# <!ENTITY pg "PostgreSQL">
postgres$# ]>
postgres$# <foo>Hello &pg;.</foo>
postgres$# $XML$::xml;
ERROR: invalid XML content
LINE 1: select $XML$<?xml version="1.0" standalone="yes" ?>
^
DETAIL: line 2: StartTag: invalid element name
<!DOCTYPE foo [
^
line 3: StartTag: invalid element name
<!ELEMENT foo (#PCDATA)>
^
line 4: StartTag: invalid element name
<!ENTITY pg "PostgreSQL">
^
line 6: Entity 'pg' not defined
<foo>Hello &pg;.</foo>
^
It is disabled by default in libxml2. I found a function
xmlSubstituteEntitiesDefault http://www.xmlsoft.org/entities.html
http://www.xmlsoft.org/html/libxml-parser.html#xmlSubstituteEntitiesDefault
The default behave should be common for all PostgreSQL's libxml2 based
function - and then it is different topic - maybe part for PostgreSQL ToDo?
But I don't remember any user requests related to this issue.
>
> libxml seems to support documents with internal DTDs:
>
> $ xmllint --valid /tmp/x
> <?xml version="1.0" standalone="yes"?>
> <!DOCTYPE foo [
> <!ELEMENT foo (#PCDATA)>
> <!ENTITY pg "PostgreSQL">
> ]>
> <foo>Hello &pg;.</foo>
>
I removed this tests - it is not related to XMLTABLE function, but to
generic XML processing/validation.
>
>
> so presumably the issue lies in the xpath stuff? Note that it's not
> even ignoring the DTD and choking on the undefined entity, it's
> choking on the DTD its self.
>
>
> OK, code comments:
>
>
> In +ExecEvalTableExpr, shouldn't you be using PG_ENSURE_ERROR_CLEANUP
> instead of a PG_TRY() / PG_CATCH() block?
>
If I understand to doc, the PG_ENSURE_ERROR_CLEANUP should be used, when
you want to catch FATAL errors (and when you want to clean shared memory).
XMLTABLE doesn't use shared memory, and doesn't need to catch fatal errors.
>
> I think the new way you handle the type stuff is much, much better,
> and with comments to explain too. Thanks very much.
>
>
> There's an oversight in tableexpr vs xmltable separation here:
>
> + case T_TableExpr:
> + *name = "xmltable";
> + return 2;
>
> presumably you need to look at the node and decide what kind of table
> expression it is or just use a generic "tableexpr".
>
> Same problem here:
>
> + case T_TableExpr:
> + {
> + TableExpr *te = (TableExpr *) node;
> +
> + /* c_expr shoud be closed in brackets */
> + appendStringInfoString(buf, "XMLTABLE(");
>
>
commented
>
>
> I don't have the libxml knowledge or remaining brain to usefully
> evaluate the xpath and xml specifics in xpath.c today. It does strike
> me that the new xpath parser should probably live in its own file,
> though.
>
moved
>
> I think this is all a big improvement. Barring the notes above and my
> lack of review of the guts of the xml.c parts of it, I'm pretty happy
> with what I see now.
>
new version is attached
Regards
Pavel
>
>
> --
> Craig Ringer http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>
xmltable-10.patch.gz
Description: application/gzip
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index f6bead6..92259fd 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -10368,7 +10368,7 @@ SELECT xml_is_well_formed_document('<pg:foo xmlns:pg="http://postgresql.org/stuf
</ROWS>
]]></screen>
- following query produces result:
+ the following query produces the result:
<screen><![CDATA[
SELECT xmltable.*
@@ -10438,7 +10438,8 @@ SELECT *
<literal>BY REF</literal> is required, the second is optional.
Passing <literal>BY VALUE</> is not supported. Multiple
comma-separated terms in the PASSING clause are not supported.
- <literal>AS</> aliases are not supported.
+ <literal>AS</> aliases are not supported. The argument must be
+ a well-formed XML document; fragments/forests are not accepted.
</para>
<para>
@@ -10473,16 +10474,73 @@ SELECT *
<para>
The <literal>PATH</> for a column is an xpath expression that is
evaluated for each row, relative to the result of the
- <replaceable>rowexpr</>, to find the value of the column. If no
- <literal>PATH</> is given then the column name is used as an
- implicit path.
+ <replaceable>rowexpr</>, to find the value of the column.
+ If no <literal>PATH</> is given then the column name is used as an
+ implicit path. It is possible for a <literal>PATH</> expression to
+ reference output columns that appear before it in the column-list, so
+ paths may be dynamically constructed based on other parts of the XML
+ document:
+ <programlisting>
+
+ </programlisting>
+ </para>
+
+ <para>
+ The text body of the XML matched by the <literal>PATH</> expression is
+ used as the column value. Multiple <literal>text()</literal> nodes within
+ an element are concatenated in order. Any child elements, processing
+ instructions, and comments are ignored, but the text contents of child
+ elements are concatenated to the result:
+ <programlisting><![CDATA[
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--x--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
+element
+-----------------
+ a1aa2a bbbbcccc
+ (1 row)
+]]></programlisting>
+ Note that the whitespace-only <literal>text()</> node between two non-text
+ elements is preserved, and that leading whitespace on a <literal>text()</>
+ node is not flattened.
+ </para>
+
+ <para>
+ A <literal>PATH</> expression that matches multiple top-level elements
+ results in an error:
+ <programlisting><![CDATA[
+SELECT * FROM xmltable('/root' passing '<root><element>1</element><element>2</element></root>' COLUMNS element text);
+ERROR: more than one value returned by column XPath expression
+]]></programlisting>
+ This applies to <literal>text()</> nodes too, so avoid using explicit text nodes
+ in your path expressions unless this behaviour is desired:
+ <programlisting><![CDATA[
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a</element></root>' COLUMNS element text PATH 'element/text()');
+ERROR: more than one value returned by column XPath expression
+]]></programlisting>
+ If the <literal>PATH</> matches an empty tag the result is an empty string
+ (not <literal>NULL</>). Any <literal>xsi:nil</> attributes are ignored.
+ </para>
+
+ <para>
+ The five predefined XML entities <literal>'apos;</>,
+ <literal>'quot;</>, <literal>'amp;</>, <literal>'lt;</>
+ and <literal>'gt;</> are expanded to their corresponding characters
+ <literal>'</>, <literal>"</>, <literal>&</>, <literal><</>
+ and <literal>></> on processing. If the output column format is
+ <type>xml</> then the characters <literal><</>, <literal>></> and
+ <literal>&</> are converted to XML entities, but <literal>"</> and
+ <literal>'</> are left unchanged. This means that <literal>'apos;</>
+ and <literal>'quot;</> entities in XML input get expanded in xml
+ output columns. There is no way to prevent their expansion.
</para>
<para>
If the path expression does not match for a given row but a
<literal>DEFAULT</> expression is specified, the resulting
default value is used. If no <literal>DEFAULT</> is given the
- field will be <literal>NULL</>.
+ field will be <literal>NULL</>. Unlike normal queries, it is possible for
+ a <literal>DEFAULT</> expression to reference the value of output columns
+ that appear prior to it in the column-list, so the default of one
+ column may be based on the value of another column.
</para>
<para>
@@ -10492,22 +10550,29 @@ SELECT *
to null then the function terminates with an ERROR.
</para>
+ <important>
+ <para>
+ Unlike normal PostgreSQL functions, the <literal>PATH</> and
+ <literal>DEFAULT</> arguments to <function>xmltable</> are not evaluated
+ to a simple value before calling the function. <literal>PATH</>
+ expressions are normally evaluated <emphasis>exactly once per input row
+ </emphasis>, and <literal>DEFAULT</> expressions each time a default is
+ needed for a field. If the expression qualifies as stable or immutable
+ the repeat evaluation may be skipped. Effectively <function>xmltable</>
+ behaves more like a subquery than a function call. This means that you
+ can usefully use volatile functions like <function>nextval</> in
+ <literal>DEFAULT</> expressions, your <literal>PATH</> expressions may
+ depend on other parts of the XML document, etc.
+ </para>
+ </important>
+
<para>
A column marked with the
<literal>FOR ORDINALITY</literal> keyword will be populated with
row numbers that match the order in which the the output rows appeared
in the original input XML document. Only one column should be
marked <literal>FOR ORDINALITY</literal>.
- </para>
-
- <note>
- <para>
- Empty tag is translated as empty string (possible attribute xsi:nil
- has not any effect. The XPath expression in PATH clause is evaluated
- for any input row. The expression in DEFAULT expression is evaluated for
- any missing value (for any output row).
- </para>
- </note>
+ </para>
</sect3>
<sect3 id="functions-xpath" xreflabel="XPATH">
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 7ce209d..4a345c4 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -1838,6 +1838,10 @@ FigureColnameInternal(Node *node, char **name)
*name = "xmlserialize";
return 2;
case T_TableExpr:
+ /*
+ * Make TableExpr act like a regular function. Only
+ * XMLTABLE expr is supported in this moment.
+ */
*name = "xmltable";
return 2;
default:
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 0f51275..5a3715c 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -29,7 +29,7 @@ OBJS = acl.o amutils.o arrayfuncs.o array_expanded.o array_selfuncs.o \
tsquery_op.o tsquery_rewrite.o tsquery_util.o tsrank.o \
tsvector.o tsvector_op.o tsvector_parser.o \
txid.o uuid.o varbit.o varchar.o varlena.o version.o \
- windowfuncs.o xid.o xml.o
+ windowfuncs.o xid.o xml.o xpath_parser.o
like.o: like.c like_match.c
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3930452..7065415 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -8285,6 +8285,11 @@ get_rule_expr(Node *node, deparse_context *context,
{
TableExpr *te = (TableExpr *) node;
+ /*
+ * Deparse TableExpr - the is only one TableExpr producer,
+ * the function XMLTABLE.
+ */
+
/* c_expr shoud be closed in brackets */
appendStringInfoString(buf, "XMLTABLE(");
diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c
index c931cee..aacbfe4 100644
--- a/src/backend/utils/adt/xml.c
+++ b/src/backend/utils/adt/xml.c
@@ -90,7 +90,7 @@
#include "utils/rel.h"
#include "utils/syscache.h"
#include "utils/xml.h"
-
+#include "utils/xpath_parser.h"
/* GUC variables */
int xmlbinary;
@@ -4102,310 +4102,6 @@ xml_is_well_formed_content(PG_FUNCTION_ARGS)
#ifdef USE_LIBXML
/*
- * We need to work with XPath expression tokens. When expression
- * starting or finishing with nodenames, then we can use prefix
- * and suffix. When default namespace is defined, then we should to
- * enhance any nodename and attribute without namespace by default
- * namespace. The procession of XPath expression is linear.
- */
-
-typedef enum
-{
- XPATH_TOKEN_NONE,
- XPATH_TOKEN_NAME,
- XPATH_TOKEN_STRING,
- XPATH_TOKEN_NUMBER,
- XPATH_TOKEN_OTHER
-} XPathTokenType;
-
-typedef struct TokenInfo
-{
- XPathTokenType ttype;
- char *start;
- int length;
-} XPathTokenInfo;
-
-#define TOKEN_STACK_SIZE 10
-
-typedef struct ParserData
-{
- char *str;
- char *cur;
- XPathTokenInfo stack[TOKEN_STACK_SIZE];
- int stack_length;
-} XPathParserData;
-
-/* Any high-bit-set character is OK (might be part of a multibyte char) */
-#define NODENAME_FIRSTCHAR(c) ((c) == '_' || (c) == '-' || \
- ((c) >= 'A' && (c) <= 'Z') || \
- ((c) >= 'a' && (c) <= 'z') || \
- (IS_HIGHBIT_SET(c)))
-
-#define IS_NODENAME_CHAR(c) (NODENAME_FIRSTCHAR(c) || (c) == '.' || \
- ((c) >= '0' && (c) <= '9'))
-
-
-/*
- * Returns next char after last char of token
- */
-static char *
-getXPathToken(char *str, XPathTokenInfo * ti)
-{
- /* skip initial spaces */
- while (*str == ' ')
- str++;
-
- if (*str != '\0')
- {
- char c = *str;
-
- ti->start = str++;
-
- if (c >= '0' && c <= '9')
- {
- while (*str >= '0' && *str <= '9')
- str++;
- if (*str == '.')
- {
- str++;
- while (*str >= '0' && *str <= '9')
- str++;
- }
- ti->ttype = XPATH_TOKEN_NUMBER;
- }
- else if (NODENAME_FIRSTCHAR(c))
- {
- while (IS_NODENAME_CHAR(*str))
- str++;
-
- ti->ttype = XPATH_TOKEN_NAME;
- }
- else if (c == '"')
- {
- while (*str != '\0')
- if (*str++ == '"')
- break;
-
- ti->ttype = XPATH_TOKEN_STRING;
- }
- else
- ti->ttype = XPATH_TOKEN_OTHER;
-
- ti->length = str - ti->start;
- }
- else
- {
- ti->start = NULL;
- ti->length = 0;
-
- ti->ttype = XPATH_TOKEN_NONE;
- }
-
- return str;
-}
-
-/*
- * reset XPath parser stack
- */
-static void
-initXPathParser(XPathParserData * parser, char *str)
-{
- parser->str = str;
- parser->cur = str;
- parser->stack_length = 0;
-}
-
-/*
- * Returns token from stack or read token
- */
-static void
-nextXPathToken(XPathParserData * parser, XPathTokenInfo * ti)
-{
- if (parser->stack_length > 0)
- memcpy(ti, &parser->stack[--parser->stack_length],
- sizeof(XPathTokenInfo));
- else
- parser->cur = getXPathToken(parser->cur, ti);
-}
-
-/*
- * Push token to stack
- */
-static void
-pushXPathToken(XPathParserData * parser, XPathTokenInfo * ti)
-{
- if (parser->stack_length == TOKEN_STACK_SIZE)
- elog(ERROR, "internal error");
- memcpy(&parser->stack[parser->stack_length++], ti,
- sizeof(XPathTokenInfo));
-}
-
-/*
- * Write token to output string
- */
-static void
-writeXPathToken(StringInfo str, XPathTokenInfo * ti)
-{
- Assert(ti->ttype != XPATH_TOKEN_NONE);
-
- if (ti->ttype != XPATH_TOKEN_OTHER)
- appendBinaryStringInfo(str, ti->start, ti->length);
- else
- appendStringInfoChar(str, *ti->start);
-}
-
-/*
- * Working horse for XPath transformation. When XPath starting by node name,
- * then prefix have to be applied. Any unqualified node name should be
- * qualified by default namespace. inside_predicate is true, when
- * _transformXPath is recursivly called because the predicate expression
- * was found.
- */
-static void
-_transformXPath(StringInfo str, XPathParserData * parser,
- bool inside_predicate,
- char *prefix, char *def_namespace_name)
-{
- XPathTokenInfo t1,
- t2;
- bool is_first_token = true;
- bool last_token_is_name = false;
-
- nextXPathToken(parser, &t1);
-
- while (t1.ttype != XPATH_TOKEN_NONE)
- {
- switch (t1.ttype)
- {
- case XPATH_TOKEN_NUMBER:
- case XPATH_TOKEN_STRING:
- last_token_is_name = false;
- is_first_token = false;
- writeXPathToken(str, &t1);
- nextXPathToken(parser, &t1);
- break;
-
- case XPATH_TOKEN_NAME:
- {
- bool is_qual_name = false;
-
- /* inside predicate ignore keywords "and" "or" */
- if (inside_predicate)
- {
- if ((strncmp(t1.start, "and", 3) == 0 && t1.length == 3) ||
- (strncmp(t1.start, "or", 2) == 0 && t1.length == 2))
- {
- writeXPathToken(str, &t1);
- nextXPathToken(parser, &t1);
- break;
- }
- }
-
- last_token_is_name = true;
- nextXPathToken(parser, &t2);
- if (t2.ttype == XPATH_TOKEN_OTHER)
- {
- if (*t2.start == '(')
- last_token_is_name = false;
- else if (*t2.start == ':')
- is_qual_name = true;
- }
-
- if (is_first_token && last_token_is_name && prefix != NULL)
- appendStringInfoString(str, prefix);
-
- if (last_token_is_name && !is_qual_name && def_namespace_name != NULL)
- appendStringInfo(str, "%s:", def_namespace_name);
-
- writeXPathToken(str, &t1);
- is_first_token = false;
-
- if (is_qual_name)
- {
- writeXPathToken(str, &t2);
- nextXPathToken(parser, &t1);
- if (t1.ttype == XPATH_TOKEN_NAME)
- writeXPathToken(str, &t1);
- else
- pushXPathToken(parser, &t1);
- }
- else
- pushXPathToken(parser, &t2);
-
- nextXPathToken(parser, &t1);
- }
- break;
-
- case XPATH_TOKEN_OTHER:
- {
- char c = *t1.start;
-
- is_first_token = false;
-
- writeXPathToken(str, &t1);
-
- if (c == '[')
- _transformXPath(str, parser, true, NULL, def_namespace_name);
- else
- {
- last_token_is_name = false;
-
- if (c == ']' && inside_predicate)
- return;
-
- else if (c == '@')
- {
- nextXPathToken(parser, &t1);
- if (t1.ttype == XPATH_TOKEN_NAME)
- {
- bool is_qual_name = false;
-
- nextXPathToken(parser, &t2);
- if (t2.ttype == XPATH_TOKEN_OTHER && *t2.start == ':')
- is_qual_name = true;
-
- if (!is_qual_name && def_namespace_name != NULL)
- appendStringInfo(str, "%s:", def_namespace_name);
-
- writeXPathToken(str, &t1);
- if (is_qual_name)
- {
- writeXPathToken(str, &t2);
- nextXPathToken(parser, &t1);
- if (t1.ttype == XPATH_TOKEN_NAME)
- writeXPathToken(str, &t1);
- else
- pushXPathToken(parser, &t1);
- }
- else
- pushXPathToken(parser, &t2);
- }
- else
- pushXPathToken(parser, &t1);
- }
- }
- nextXPathToken(parser, &t1);
- }
- break;
-
- case XPATH_TOKEN_NONE:
- elog(ERROR, "should not be here");
- }
- }
-}
-
-static void
-transformXPath(StringInfo str, char *xpath,
- char *prefix, char *def_namespace_name)
-{
- XPathParserData parser;
-
- initStringInfo(str);
- initXPathParser(&parser, xpath);
- _transformXPath(str, &parser, false, prefix, def_namespace_name);
-}
-
-/*
* Functions for XmlTableBuilder
*
*/
@@ -4840,7 +4536,8 @@ XmlTableGetValue(void *tcontext, int colnum, bool *isnull)
{
PG_TRY();
{
- cstr = escape_xml((char *) str);
+ /* Copy string to PostgreSQL controlled memory */
+ cstr = pstrdup((char *) str);
}
PG_CATCH();
{
@@ -4848,14 +4545,14 @@ XmlTableGetValue(void *tcontext, int colnum, bool *isnull)
PG_RE_THROW();
}
PG_END_TRY();
+
+ xmlFree(str);
}
else
{
/* Return empty string when tag is empty */
cstr = pstrdup("");
}
-
- xmlFree(str);
}
else
{
diff --git a/src/backend/utils/adt/xpath_parser.c b/src/backend/utils/adt/xpath_parser.c
new file mode 100644
index 0000000..a38798b
--- /dev/null
+++ b/src/backend/utils/adt/xpath_parser.c
@@ -0,0 +1,340 @@
+/*-------------------------------------------------------------------------
+ *
+ * xpath_parser.c
+ * XML XPath parser.
+ *
+ *
+ * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/utils/adt/xpath_parser.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+/*
+ * All PostgreSQL xml related functionality is based on libxml2 library.
+ * XPath support is not a exception. But libxml2 doesn't provide all functions
+ * necessary for implementation of XMLTABLE function. There is not API for
+ * access to XPath expression AST (abstract syntax tree), and there is not
+ * support for default namespaces in XPath expressions.
+ *
+ * These requests are implemented with simple XPath parser/preprocessor.
+ * This XPath parser transform XPath expression to another XPath expression
+ * used in libxml2 XPath evaluation. It doesn't replace libxml2 XPath parser
+ * or libxml2 XPath expression evaluation. Currently libxml2 is stable, but
+ * without any movement in implementation new or missing features.
+ */
+#include "postgres.h"
+#include "utils/xpath_parser.h"
+
+/*
+ * support functions for XMLTABLE function
+ *
+ */
+#ifdef USE_LIBXML
+
+/*
+ * We need to work with XPath expression tokens. When expression
+ * starting with nodename, then we can use prefix. When default
+ * namespace is defined, then we should to enhance any nodename
+ * and attribute without namespace by default namespace.
+ */
+
+typedef enum
+{
+ XPATH_TOKEN_NONE,
+ XPATH_TOKEN_NAME,
+ XPATH_TOKEN_STRING,
+ XPATH_TOKEN_NUMBER,
+ XPATH_TOKEN_OTHER
+} XPathTokenType;
+
+typedef struct TokenInfo
+{
+ XPathTokenType ttype;
+ char *start;
+ int length;
+} XPathTokenInfo;
+
+#define TOKEN_STACK_SIZE 10
+
+typedef struct ParserData
+{
+ char *str;
+ char *cur;
+ XPathTokenInfo stack[TOKEN_STACK_SIZE];
+ int stack_length;
+} XPathParserData;
+
+/* Any high-bit-set character is OK (might be part of a multibyte char) */
+#define NODENAME_FIRSTCHAR(c) ((c) == '_' || (c) == '-' || \
+ ((c) >= 'A' && (c) <= 'Z') || \
+ ((c) >= 'a' && (c) <= 'z') || \
+ (IS_HIGHBIT_SET(c)))
+
+#define IS_NODENAME_CHAR(c) (NODENAME_FIRSTCHAR(c) || (c) == '.' || \
+ ((c) >= '0' && (c) <= '9'))
+
+
+/*
+ * Returns next char after last char of token
+ */
+static char *
+getXPathToken(char *str, XPathTokenInfo * ti)
+{
+ /* skip initial spaces */
+ while (*str == ' ')
+ str++;
+
+ if (*str != '\0')
+ {
+ char c = *str;
+
+ ti->start = str++;
+
+ if (c >= '0' && c <= '9')
+ {
+ while (*str >= '0' && *str <= '9')
+ str++;
+ if (*str == '.')
+ {
+ str++;
+ while (*str >= '0' && *str <= '9')
+ str++;
+ }
+ ti->ttype = XPATH_TOKEN_NUMBER;
+ }
+ else if (NODENAME_FIRSTCHAR(c))
+ {
+ while (IS_NODENAME_CHAR(*str))
+ str++;
+
+ ti->ttype = XPATH_TOKEN_NAME;
+ }
+ else if (c == '"')
+ {
+ while (*str != '\0')
+ if (*str++ == '"')
+ break;
+
+ ti->ttype = XPATH_TOKEN_STRING;
+ }
+ else
+ ti->ttype = XPATH_TOKEN_OTHER;
+
+ ti->length = str - ti->start;
+ }
+ else
+ {
+ ti->start = NULL;
+ ti->length = 0;
+
+ ti->ttype = XPATH_TOKEN_NONE;
+ }
+
+ return str;
+}
+
+/*
+ * reset XPath parser stack
+ */
+static void
+initXPathParser(XPathParserData * parser, char *str)
+{
+ parser->str = str;
+ parser->cur = str;
+ parser->stack_length = 0;
+}
+
+/*
+ * Returns token from stack or read token
+ */
+static void
+nextXPathToken(XPathParserData * parser, XPathTokenInfo * ti)
+{
+ if (parser->stack_length > 0)
+ memcpy(ti, &parser->stack[--parser->stack_length],
+ sizeof(XPathTokenInfo));
+ else
+ parser->cur = getXPathToken(parser->cur, ti);
+}
+
+/*
+ * Push token to stack
+ */
+static void
+pushXPathToken(XPathParserData * parser, XPathTokenInfo * ti)
+{
+ if (parser->stack_length == TOKEN_STACK_SIZE)
+ elog(ERROR, "internal error");
+ memcpy(&parser->stack[parser->stack_length++], ti,
+ sizeof(XPathTokenInfo));
+}
+
+/*
+ * Write token to output string
+ */
+static void
+writeXPathToken(StringInfo str, XPathTokenInfo * ti)
+{
+ Assert(ti->ttype != XPATH_TOKEN_NONE);
+
+ if (ti->ttype != XPATH_TOKEN_OTHER)
+ appendBinaryStringInfo(str, ti->start, ti->length);
+ else
+ appendStringInfoChar(str, *ti->start);
+}
+
+/*
+ * Working horse for XPath transformation. When XPath starting by node name,
+ * then prefix have to be applied. Any unqualified node name should be
+ * qualified by default namespace. inside_predicate is true, when
+ * _transformXPath is recursivly called because the predicate expression
+ * was found.
+ */
+static void
+_transformXPath(StringInfo str, XPathParserData * parser,
+ bool inside_predicate,
+ char *prefix, char *def_namespace_name)
+{
+ XPathTokenInfo t1,
+ t2;
+ bool is_first_token = true;
+ bool last_token_is_name = false;
+
+ nextXPathToken(parser, &t1);
+
+ while (t1.ttype != XPATH_TOKEN_NONE)
+ {
+ switch (t1.ttype)
+ {
+ case XPATH_TOKEN_NUMBER:
+ case XPATH_TOKEN_STRING:
+ last_token_is_name = false;
+ is_first_token = false;
+ writeXPathToken(str, &t1);
+ nextXPathToken(parser, &t1);
+ break;
+
+ case XPATH_TOKEN_NAME:
+ {
+ bool is_qual_name = false;
+
+ /* inside predicate ignore keywords "and" "or" */
+ if (inside_predicate)
+ {
+ if ((strncmp(t1.start, "and", 3) == 0 && t1.length == 3) ||
+ (strncmp(t1.start, "or", 2) == 0 && t1.length == 2))
+ {
+ writeXPathToken(str, &t1);
+ nextXPathToken(parser, &t1);
+ break;
+ }
+ }
+
+ last_token_is_name = true;
+ nextXPathToken(parser, &t2);
+ if (t2.ttype == XPATH_TOKEN_OTHER)
+ {
+ if (*t2.start == '(')
+ last_token_is_name = false;
+ else if (*t2.start == ':')
+ is_qual_name = true;
+ }
+
+ if (is_first_token && last_token_is_name && prefix != NULL)
+ appendStringInfoString(str, prefix);
+
+ if (last_token_is_name && !is_qual_name && def_namespace_name != NULL)
+ appendStringInfo(str, "%s:", def_namespace_name);
+
+ writeXPathToken(str, &t1);
+ is_first_token = false;
+
+ if (is_qual_name)
+ {
+ writeXPathToken(str, &t2);
+ nextXPathToken(parser, &t1);
+ if (t1.ttype == XPATH_TOKEN_NAME)
+ writeXPathToken(str, &t1);
+ else
+ pushXPathToken(parser, &t1);
+ }
+ else
+ pushXPathToken(parser, &t2);
+
+ nextXPathToken(parser, &t1);
+ }
+ break;
+
+ case XPATH_TOKEN_OTHER:
+ {
+ char c = *t1.start;
+
+ is_first_token = false;
+
+ writeXPathToken(str, &t1);
+
+ if (c == '[')
+ _transformXPath(str, parser, true, NULL, def_namespace_name);
+ else
+ {
+ last_token_is_name = false;
+
+ if (c == ']' && inside_predicate)
+ return;
+
+ else if (c == '@')
+ {
+ nextXPathToken(parser, &t1);
+ if (t1.ttype == XPATH_TOKEN_NAME)
+ {
+ bool is_qual_name = false;
+
+ nextXPathToken(parser, &t2);
+ if (t2.ttype == XPATH_TOKEN_OTHER && *t2.start == ':')
+ is_qual_name = true;
+
+ if (!is_qual_name && def_namespace_name != NULL)
+ appendStringInfo(str, "%s:", def_namespace_name);
+
+ writeXPathToken(str, &t1);
+ if (is_qual_name)
+ {
+ writeXPathToken(str, &t2);
+ nextXPathToken(parser, &t1);
+ if (t1.ttype == XPATH_TOKEN_NAME)
+ writeXPathToken(str, &t1);
+ else
+ pushXPathToken(parser, &t1);
+ }
+ else
+ pushXPathToken(parser, &t2);
+ }
+ else
+ pushXPathToken(parser, &t1);
+ }
+ }
+ nextXPathToken(parser, &t1);
+ }
+ break;
+
+ case XPATH_TOKEN_NONE:
+ elog(ERROR, "should not be here");
+ }
+ }
+}
+
+void
+transformXPath(StringInfo str, char *xpath,
+ char *prefix, char *def_namespace_name)
+{
+ XPathParserData parser;
+
+ initStringInfo(str);
+ initXPathParser(&parser, xpath);
+ _transformXPath(str, &parser, false, prefix, def_namespace_name);
+}
+
+#endif
diff --git a/src/include/utils/xpath_parser.h b/src/include/utils/xpath_parser.h
new file mode 100644
index 0000000..c6fc532
--- /dev/null
+++ b/src/include/utils/xpath_parser.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * xpath_parser.h
+ * Declarations for XML XPath transformation.
+ *
+ *
+ * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/xml.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef XPATH_PARSER_H
+#define XPATH_PARSER_H
+
+#include "postgres.h"
+#include "lib/stringinfo.h"
+
+void transformXPath(StringInfo str, char *xpath,
+ char *prefix, char *def_namespace_name);
+
+#endif /* XPATH_PARSER_H */
diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out
index 8ea477f..c8ac437 100644
--- a/src/test/regress/expected/xml.out
+++ b/src/test/regress/expected/xml.out
@@ -1266,14 +1266,43 @@ SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan"
5 | Japan | 3 | <COUNTRY_ID>JP</COUNTRY_ID><COUNTRY_NAME>Japan</COUNTRY_NAME><REGION_ID>3</REGION_ID><PREMIER_NAME>Sinzo Abe</PREMIER_NAME>
(2 rows)
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
- element
-----------------
- a1aa2abbbbcccc
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
+ element
+-------------------
+ a1aa2a bbbbcccc
(1 row)
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
ERROR: more than one value returned by column XPath expression
+-- CDATA test
+select * from xmltable('r' passing '<d><r><c><![CDATA[<hello> &"<>!<a>foo</a>]]></c></r><r><c>2</c></r></d>' columns c text);
+ c
+-------------------------
+ <hello> &"<>!<a>foo</a>
+ 2
+(2 rows)
+
+-- XML builtin entities
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent text);
+ ent
+-----
+ '
+ "
+ &
+ <
+ >
+(5 rows)
+
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent xml);
+ ent
+------------------
+ <ent>'</ent>
+ <ent>"</ent>
+ <ent>&</ent>
+ <ent><</ent>
+ <ent>></ent>
+(5 rows)
+
EXPLAIN (VERBOSE, COSTS OFF)
SELECT xmltable.*
FROM (SELECT data FROM xmldata) x,
diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out
index a185ef7..83909b9 100644
--- a/src/test/regress/expected/xml_1.out
+++ b/src/test/regress/expected/xml_1.out
@@ -1058,18 +1058,38 @@ SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan"
----+--------------+-----------+---------
(0 rows)
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
ERROR: unsupported XML feature
LINE 1: SELECT * FROM xmltable('/root' passing '<root><element>a1a<!...
^
DETAIL: This functionality requires the server to be built with libxml support.
HINT: You need to rebuild PostgreSQL using --with-libxml.
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
ERROR: unsupported XML feature
LINE 1: SELECT * FROM xmltable('/root' passing '<root><element>a1a<!...
^
DETAIL: This functionality requires the server to be built with libxml support.
HINT: You need to rebuild PostgreSQL using --with-libxml.
+-- CDATA test
+select * from xmltable('r' passing '<d><r><c><![CDATA[<hello> &"<>!<a>foo</a>]]></c></r><r><c>2</c></r></d>' columns c text);
+ERROR: unsupported XML feature
+LINE 1: select * from xmltable('r' passing '<d><r><c><![CDATA[<hello...
+ ^
+DETAIL: This functionality requires the server to be built with libxml support.
+HINT: You need to rebuild PostgreSQL using --with-libxml.
+-- XML builtin entities
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent text);
+ERROR: unsupported XML feature
+LINE 1: SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</en...
+ ^
+DETAIL: This functionality requires the server to be built with libxml support.
+HINT: You need to rebuild PostgreSQL using --with-libxml.
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent xml);
+ERROR: unsupported XML feature
+LINE 1: SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</en...
+ ^
+DETAIL: This functionality requires the server to be built with libxml support.
+HINT: You need to rebuild PostgreSQL using --with-libxml.
EXPLAIN (VERBOSE, COSTS OFF)
SELECT xmltable.*
FROM (SELECT data FROM xmldata) x,
diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out
index 6e522a6..358fed4 100644
--- a/src/test/regress/expected/xml_2.out
+++ b/src/test/regress/expected/xml_2.out
@@ -1245,14 +1245,43 @@ SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan"
5 | Japan | 3 | <COUNTRY_ID>JP</COUNTRY_ID><COUNTRY_NAME>Japan</COUNTRY_NAME><REGION_ID>3</REGION_ID><PREMIER_NAME>Sinzo Abe</PREMIER_NAME>
(2 rows)
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
- element
-----------------
- a1aa2abbbbcccc
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
+ element
+-------------------
+ a1aa2a bbbbcccc
(1 row)
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
ERROR: more than one value returned by column XPath expression
+-- CDATA test
+select * from xmltable('r' passing '<d><r><c><![CDATA[<hello> &"<>!<a>foo</a>]]></c></r><r><c>2</c></r></d>' columns c text);
+ c
+-------------------------
+ <hello> &"<>!<a>foo</a>
+ 2
+(2 rows)
+
+-- XML builtin entities
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent text);
+ ent
+-----
+ '
+ "
+ &
+ <
+ >
+(5 rows)
+
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent xml);
+ ent
+------------------
+ <ent>'</ent>
+ <ent>"</ent>
+ <ent>&</ent>
+ <ent><</ent>
+ <ent>></ent>
+(5 rows)
+
EXPLAIN (VERBOSE, COSTS OFF)
SELECT xmltable.*
FROM (SELECT data FROM xmldata) x,
diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql
index b67af31..49f6bc4 100644
--- a/src/test/regress/sql/xml.sql
+++ b/src/test/regress/sql/xml.sql
@@ -393,8 +393,15 @@ SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan"
SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan" or COUNTRY_NAME="India"]' PASSING data COLUMNS id int PATH '@id', "COUNTRY_NAME" text, "REGION_ID" int, rawdata xml PATH '.');
SELECT xmltable.* FROM xmldata, LATERAL xmltable('/ROWS/ROW[COUNTRY_NAME="Japan" or COUNTRY_NAME="India"]' PASSING data COLUMNS id int PATH '@id', "COUNTRY_NAME" text, "REGION_ID" int, rawdata xml PATH './*');
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
-SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?>bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text);
+SELECT * FROM xmltable('/root' passing '<root><element>a1a<!-- aaaa -->a2a<?aaaaa?> <!--z--> bbbb<x>xxx</x>cccc</element></root>' COLUMNS element text PATH 'element/text()'); -- should fail
+
+-- CDATA test
+select * from xmltable('r' passing '<d><r><c><![CDATA[<hello> &"<>!<a>foo</a>]]></c></r><r><c>2</c></r></d>' columns c text);
+
+-- XML builtin entities
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent text);
+SELECT * FROM xmltable('/x/a' PASSING '<x><a><ent>'</ent></a><a><ent>"</ent></a><a><ent>&</ent></a><a><ent><</ent></a><a><ent>></ent></a></x>' COLUMNS ent xml);
EXPLAIN (VERBOSE, COSTS OFF)
SELECT xmltable.*
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
