Hi Zhenghua,

Jark is right. The reason why we haven't updated those interfaces yet is because we are actually would like to introduce new interfaces. We should target new interfaces in this release. Even a short-term fix as you proposed with `getRecordDataType` does actually not help as Jingsong pointed out because we cannot represent tuples in DataType and are also not planning to support them natively but only as a structured type in the future.

In my envisioned design, the new sink interface should just always get a `ChangeRow` which is never serialized and just a data structure for communicating between the wrapping sink function and the returned sink function by the table sink.

Let me sketch a rough design document that I will share with you shortly. Then we could also discuss alternatives.

Thanks,
Timo


On 04.02.20 04:18, Zhenghua Gao wrote:
Hi Jark, thanks for your comments.
IIUC, the framework will only recognize getRecordDataType and
ignore getConsumedDataType for UpsertStreamTableSink, is that right?
Your are right.

getRecordDataType is little confused as UpsertStreamTableSink already has
three getXXXType().
the getRecordType and getOutputType is deprecated and mainly for backward
compatibility.

*Best Regards,*
*Zhenghua Gao*


On Mon, Feb 3, 2020 at 10:11 PM Jark Wu <imj...@gmail.com> wrote:

Thanks Zhenghua for starting this discussion.

Currently, all the UpsertStreamTableSinks can't upgrade to the new type
system which affects usability a lot.
I hope we can fix that in 1.11.

I'm find with *getRecordDataType* for a temporary solution.
IIUC, the framework will only recognize getRecordDataType and
ignore getConsumedDataType for UpsertStreamTableSink, is that right?

I guess Timo are planning to design a new source/sink interface which will
also fix this problem, but I'm not sure the timelines. cc @Timo
It would be better if we can have a new and complete interface, because
getRecordDataType is little confused as UpsertStreamTableSink already has
three getXXXType().

Best,
Jark


On Mon, 3 Feb 2020 at 17:48, Zhenghua Gao <doc...@gmail.com> wrote:

Hi Jingsong,  For now, only UpsertStreamTableSink and
RetractStreamTableSink consumes JTuple2
So the 'getConsumedDataType' interface is not necessary in validate &
codegen phase.
See


https://github.com/apache/flink/pull/10880/files#diff-137d090bd719f3a99ae8ba6603ed81ebR52
  and


https://github.com/apache/flink/pull/10880/files#diff-8405c17e5155aa63d16e497c4c96a842R304

What about stay the same to use RAW type?

*Best Regards,*
*Zhenghua Gao*


On Mon, Feb 3, 2020 at 4:59 PM Jingsong Li <jingsongl...@gmail.com>
wrote:

Hi Zhenghua,

The *getRecordDataType* looks good to me.

But the main problem is how to represent the tuple type in DataType. I
understand that it is necessary to use StructuredType, but at present,
planner does not support StructuredType, so the other way is to support
StructuredType.

Best,
Jingsong Lee

On Mon, Feb 3, 2020 at 4:49 PM Kurt Young <ykt...@gmail.com> wrote:

Would overriding `getConsumedDataType` do the job?

Best,
Kurt


On Mon, Feb 3, 2020 at 3:52 PM Zhenghua Gao <doc...@gmail.com>
wrote:

Hi all,

FLINK-12254[1] [2] updated TableSink and related interfaces to new
type
system which
allows connectors use the new type system based on DataTypes.

But FLINK-12911 port UpsertStreamTableSink and
RetractStreamTableSink
to
flink-api-java-bridge and returns TypeInformation of the requested
record
type which
can't support types with precision and scale, e.g. TIMESTAMP(p),
DECIMAL(p,s).

/**
  * Returns the requested record type.
  */
TypeInformation<T> getRecordType();


A proposal is deprecating the *getRecordType* API and adding a
*getRecordDataType* API instead to return the data type of the
requested
record. I have filed the issue FLINK-15469 and
an initial PR to verify it.

What do you think about this API changes? Any feedback are
appreciated.
[1] https://issues.apache.org/jira/browse/FLINK-12254
[2] https://github.com/apache/flink/pull/8596
[3] https://issues.apache.org/jira/browse/FLINK-15469

*Best Regards,*
*Zhenghua Gao*



--
Best, Jingsong Lee





Reply via email to