range bug in resolve_generic_type?

[Paul A Jungwirth]

Hello,

I was looking at resolve_generic_type to add anymultirange support,
and the range logic doesn't seem correct to me. This function takes 3
type Oids:

- declared_type is the declared type of a function parameter whose
actual type it would like to deduce.
- context_{declared,actual}_type are the {declared,actual} types of
another parameter, as a hint.

Here is the logic in pseudocode:

    if (declared_type == ANYARRAYOID)
    {
        if (context_declared_type == ANYARRAYOID)
            return the array type
        else if (context_declared_type == ANYELEMENTOID ||
                 context_declared_type == ANYNONARRAYOID ||
                 context_declared_type == ANYENUMOID ||
                 context_declared_type == ANYRANGEOID)
                 context_declared_type == ANYMULTIRANGEOID)
            return an array of those things
        }
    else if (declared_type == ANYELEMENTOID ||
               declared_type == ANYNONARRAYOID ||
               declared_type == ANYENUMOID ||
               declared_type == ANYRANGEOID)
        {
        if (context_declared_type == ANYARRAYOID)
            return the element type
        else if (context_declared_type == ANYRANGEOID)
            return the element type
        else if (context_declared_type == ANYELEMENTOID ||
                 context_declared_type == ANYNONARRAYOID ||
                 context_declared_type == ANYENUMOID)
            return the actual type
    }
    else {
        return declared_type  // since it's not polymorphic....
    }

It seems to me that these inputs would give invalid results:

resolve_generic_type(ANYARRAYOID, x, ANYRANGEOID) - this will return
an array of the *range type*, but that contracts the normal
relationship between anyelement and anyrange. It should return an
array of the range's element type.

resolve_generic_type(ANYRANGEOID, x, ANYRANGEOID) - this will return
the known range's *element* type, but it should return the known
range.

Fortunately we never call the function in either of those ways. The
only call site I could find is utils/fmgr/funcapi.c, and it only calls
it like this:

    resolve_generic_type(ANYELEMENTOID, anyarray_type, ANYARRAYOID)
    resolve_generic_type(ANYELEMENTOID, anyrange_type, ANYRANGEOID)
    resolve_generic_type(ANYARRAYOID, anyelement_type, ANYELEMENTOID)

So I'm curious what I should do about all that, if anything. I'm happy
to fix it (although I'm not sure how I'd test my changes), but it
seems worth asking first. The first case in particular we *could* use
to guess an anyarray's type if we wanted to, so I could add that to
funcapi.c and then probably invent some scenario to test it. For the
second case, I'm guessing if a function has the *same* polymorphic
parameter we probably make an inference without using
resolve_generic_type, since they should obviously match. But does
anyone here have a suggestion?

Thanks!
Paul