Hi Zhenghua, I think it's not just about precision of type. Connectors not validate the types either. Now there is "SchemaValidator", this validator is just used to validate type properties. But not for connector type support. I think we can have something like "DataTypeValidator" to help connectors validating their type support.
Consider current validator design, validator is called by connector itself. it's more like a util class than a mechanism. Best, Jingsong Lee On Fri, Jan 10, 2020 at 11:47 AM Zhenghua Gao <doc...@gmail.com> wrote: > Hi dev, > > I'd like to kick off a discussion on a mechanism to validate the precision > of columns for some connectors. > > We come to an agreement that the user should be informed if the connector > does not support the desired precision. And from the connector developer's > view, there are 3-levels information to be considered: > > - the ability of external systems (e.g. Apache Derby support > TIMESTAMP(9), Mysql support TIMESTAMP(6), etc) > > Connector developers should use this information to validate user's DDL and > make sure throw an exception if concrete column is out of range. > > > - schema of referenced tables in external systems > > If the schema information of referenced tables is available in Compile > Time, connector developers could use it to find the mismatch between DDL. > But in most cases, the schema information is unavailable because of network > isolation or authority management. We should use it with caution. > > > - schema-less external systems (e.g. HBase) > > If the external systems is schema-less like HBase, the connector developer > should make sure the connector doesn't cause precision loss (e.g. > flink-hbase serializes java.sql.Timestamp to long in bytes which only keep > millisecond's precision.) > > To make it more specific, some scenarios of JDBC Connector are list as > following: > > - The underlying DB supports DECIMAL(65, 30), which is out of the range > of Flink's Decimal > - The underlying DB supports TIMESTAMP(6), and user want to define a > table with TIMESTAMP(9) in Flink > - User defines a table with DECIMAL(10, 4) in underlying DB, and want to > define a table with DECIMAL(5, 2) in Flink > - The precision of the underlying DB varies between different versions > > > What do you think about this? any feedback are appreciates. > > *Best Regards,* > *Zhenghua Gao* > -- Best, Jingsong Lee
