Hi dev,
I'd like to kick off a discussion on the improvement of TableSourceFactory
and TableSinkFactory.
Motivation:
Now the main needs and problems are:
1.Connector can't get TableConfig [1], and some behaviors really need to be
controlled by the user's table configuration. In the era of catalog, we
can't put these config in connector properties, which is too inconvenient.
2.Connector can't know if this is batch or stream execution mode. But the
sink implementation of batch and stream is totally different. I understand
there is an update mode property now, but it splits the batch and stream in
the catalog dimension. In fact, this information can be obtained through
the current TableEnvironment.
3.No interface to call validation. Now our validation is more util classes.
It depends on whether or not the connector calls. Now we have some new
validations to add, such as [2], which is really confuse uses, even
developers. Another problem is that our SQL update (DDL) does not have
validation [3]. It is better to report an error when executing DDL,
otherwise it will confuse the user.
Proposed change draft for 1 and 2:
interface CatalogTableContext {
ObjectPath getTablePath();
CatalogTable getTable();
ReadableConfig getTableConfig();
boolean isStreamingMode();
}
public interface TableSourceFactory<T> extends TableFactory {
default TableSource<T> createTableSource(CatalogTableContext context) {
return createTableSource(context.getTablePath(), context.getTable());
}
......
}
Proposed change draft for 3:
public interface TableFactory {
TableValidators validators();
interface TableValidators {
ConnectorDescriptorValidator connectorValidator();
TableSchemaValidator schemaValidator();
FormatDescriptorValidator formatValidator();
}
}
What do you think?
[1] https://issues.apache.org/jira/browse/FLINK-15290
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-A-mechanism-to-validate-the-precision-of-columns-for-connectors-td36552.html#a36556
[3] https://issues.apache.org/jira/browse/FLINK-15509
Best,
Jingsong Lee
