Hi Yubin,
Thanks for the update! +1 for keeping the default length of String and
Bytes unchange.
Best,
Lincoln Lee
lixin58...@163.com <lixin58...@163.com> 于2023年11月24日周五 12:04写道：
> Thank Jane for providing examples to make discussions clearer.
> Thank Lincoln and Xuyang for your feedback,I agree with you wholeheartedly
> that it is better to throw an error instead of ignoring it directly.
> Extending datagen to generate variable length values is really an excelent
> idea, I will create another jira to follow up.
>
> Taking the example provided,
>
> 1. For fixed-length data types (char, binary), two DDLs which custom
> length should throw exception like 'User-defined length of the fixed-length
> field f0 is not supported.'
>
> 1.
> CREATE TABLE foo (
> f0 CHAR(5)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
> CREATE TABLE bar (
> f0 CHAR(5)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '1');
>
> 1. For variable-length data types (varchar, varbinary)，the first DDL
> can be executed legally, if illegal user-defined length configured, will
> throw exception like 'User-defined length of the VARCHAR field %s should be
> shorter than the schema definition.'
>
> 1.
> CREATE TABLE meow (
> f0 VARCHAR(20)
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
> 1. For special variable-length data types, since the length of String
> and Bytes is very large (2^31 - 1), when users does not specify a smaller
> field length, Fields that occupy a huge amount of memory (estimated to be
> more than 2GB) will be generated by default, which can easily lead to
> "java.lang.OutOfMemoryError: Java heap space", so I recommend that the
> default length of these two fields is 100 just like before, but the length
> can be configured to less than 2^31-1.
>
> 1.
> CREATE TABLE purr (
> f0 STRING
> ) WITH ('connector' = 'datagen', 'fields.f0.length' = '10');
>
> Updates have been synchronized to the merge request [1]
>
> WDYT?
>
> [1] https://github.com/apache/flink/pull/23678
>
>
> Best!
> Yubin
>
