Hi everyone,
thanks for all the feedback I received so far. I had very healthy
discussions with various people both online and offline at Current and
Flink Forward Berlin. The general user responses were also very
positive. The FLIP should be ready to start a VOTE thread.
This is the last call for feedback. I would start a VOTE tomorrow if
there are no objections. Happy to take further feedback during
implementation as well.
Thanks,
Timo
On 30.10.24 14:34, Timo Walther wrote:
Hi Jim,
3. Multiple output tables
> Does the target_table need to be specified in the SELECT clause?
No. Similar to reading from a regular table. The filter column must not
be part of SELECT part.
> It seems like the two target_table could have separate schemas defined.
That is true. The SELECT is responsible to transforms the columns into
the target table's schema. The output row of the PTF might be a union of
various columns in this case.
10. Support for State TTL
> I'd be strongly in favor of doing any interface / base work we need in
> the initial implementation so that state size can be managed.
I agree, State TTL is crucial. I updated the FLIP and added interfaces
to StateTypeStrategy and @StateHint.
Cheers,
Timo
On 23.10.24 17:59, Jim Hughes wrote:
Hi Timo,
Thank you for the answers. I have a few clarifications inlined.
On Mon, Oct 14, 2024 at 8:07 AM Timo Walther <twal...@apache.org> wrote:
3. Change of interfaces for multiple output tables
Currently, I think using a STATEMENT SET should be enough for side
output semantics. I have added an example in section 5.2.3.2 for that.
We are still free to add more methods to Context, let the function
implement additional interfaces or use more code generation together
with @ArgumentHints.
Does the target_table need to be specified in the SELECT clause? Or
could
it read
EXECUTE STATEMENT SET BEGIN
INSERT INTO main SELECT a, b FROM FunctionWithSideOutput(input =>
data,
uid = 'only_once') WHERE target_table = 'main';
INSERT INTO side SELECT a, b FROM FunctionWithSideOutput(input =>
data,
uid = 'only_once') WHERE target_table = 'side';
END;
Separately, for clarity, it seems like the two target_table could have
separate schemas defined.
10. Support for State TTL
Supporting state TTL will be easy. We just need to add a parameter to
@StateHint and pass it through.
If PTFs can have state, I'd be strongly in favor of doing any interface /
base work we need in the initial implementation so that state size can be
managed. If it is just sufficient to have hints in the interface,
awesome!
Cheers,
Jim
