Hi everyone,
Currently, the data gen table source only supports a subset of Flink SQL
types. One missing type in particular is TIMESTAMP(3). The reason, I
suspect, it was not added originally is that it doesn't really make sense
to have random timestamps. What you really want is for them to be
ascending. In the use cases of data generation, users typically don't care
about late data. The workaround proposed in the docs is to create your
event time attribute using a computed column.
CREATE TABLE t (
ts AS LOCALTIMESTAMP
) WITH (
'connector' = 'datagen'
)
The problem is that this does not play well with the LIKE clause. Many
users do not create datagen backed tables from scratch but using the LIKE
clause to shadow a physical table in their catalog - such as Kafka.
The problem is the LIKE clause does not allow redefining columns so there
is no way to do this for a table with an event time attribute. The below
will fail.
CREATE TABLE Orders (
order_id BIGINT,
order_time TIMESTAMP(3)
quantity INT,
cost AS price * quantity,
WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND,
PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'topic' = 'orders',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'orderGroup',
'format' = 'csv'
)
CREATE TEMPORARY TABLE Orders WITH (
'connector' = 'datagen'
) LIKE Orders (EXCLUDING ALL)
I see two solutions to this and would like to hear what people think.
1) Support TIMESTAMP in datagen tables but always supply strictly ascending
timestamps. The above would now "just work". This semantic makes sense
given the way event time attributes are used in streaming applications and
we can clearly document the behavior.
2) Relax the constraints of the LIKE clause to allow overriding physical
columns with computed columns. This would make it clearer to the user what
is happening but would require substantially higher development effort and
I don't know if this feature would add value beyond this one use case. In
practice, this would allow the following.
Please let me know what you think.
CREATE TEMPORARY TABLE Orders (
order_time AS LOCALTIMESTAMP
) WITH (
'connector' = 'datagen'
) LIKE Orders (EXCLUDING ALL)
Seth
