autophagy commented on code in PR #29267: URL: https://github.com/apache/flink/pull/29267#discussion_r4182410155
########## flink-python/pyflink/table/typehints.py: ########## @@ -0,0 +1,160 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +"""Shared inference from Python type hints to table :class:`DataType`s. + +This is the neutral core used to resolve a standard Python annotation into a +:mod:`pyflink.table.types` ``DataType``. It is consumed by the DataFrame API and +by UDF type-hint inference so both derive types from a single mapping. +""" + +import collections.abc +import datetime +import decimal +import types +from functools import partial +from typing import Any, Callable, Dict, Union, get_args, get_origin, get_type_hints + +from pyflink.table.types import DataType, DataTypes + +_PEP_604_UNION_TYPE = getattr(types, "UnionType", None) +_AWAITABLE_ORIGINS = (collections.abc.Coroutine, collections.abc.Awaitable) + +_BASIC_TYPE_HINT_FACTORIES: Dict[Any, Callable[[], DataType]] = { + bool: DataTypes.BOOLEAN, + int: DataTypes.BIGINT, + float: DataTypes.DOUBLE, + str: DataTypes.STRING, + bytes: DataTypes.BYTES, + bytearray: DataTypes.BYTES, + decimal.Decimal: partial(DataTypes.DECIMAL, 38, 18), + datetime.date: DataTypes.DATE, + # TIME is stored at runtime as an int number of milliseconds of the day, so 3 is + # the highest fractional-second precision that survives a round trip. + datetime.time: partial(DataTypes.TIME, 3), Review Comment: I dug into this a little deeper and I think keeping it as `TIME(3)` would make more sense, particularly because changing the precision does actually change behaviour for pandas/arrow udfs. PyFlink choses the arrow column type from the time precision, 1-3 to `time32('ms')`, 7-9 to `time64('ns')`, so changing it to `TIME(9)` would make those UDFs carry nanosecond columns when they can only ever hold ms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
