Very much thank you for reviewing FLIP-255.
## Aggregate functions
I partly refer to Apache Druid[1]. In Druid, each aggregate column is
designated by an Json/map string, such as ??{"type": count, "name":
<output_name>}??. The "type" is aggregate function and the "name" is
column name. In FLIP-255, I plan to use an map to save all column names, whose
aggregate function is given by user, and they aggregate functions. I have
thought the way like `'fields.sum_field1.function'='sum'`, but I think it is
less neat than using map when the number of aggregate columns gets larger.
## Default function
I do mean `replace`.
There is not the same case.
In Drois, columns are either aggregate keys (like primary key in our case) or
the one to be aggregated[2].
In Druid, aggregate results are put in a new table which means all columns have
aggregate functions.
I think it is possible that an column has no aggregate functions. So I use
`replace` as default function.
## Supported functions
`replace` refers to Doris [2]. To be honest, `replace_if_not_null/concatenate`
are proposed according to imaginary scenarios but without other systems can be
referred to.
Aforementioned responses are personal idea. Looking forward to your precious
advice.
[1] https://druid.apache.org/docs/latest/querying/aggregations.html
[2] https://doris.apache.org/zh-CN/docs/data-table/data-model/
------------------ ???????? ------------------
??????:
"dev"
<jingsongl...@gmail.com>;
????????: 2022??8??2??(??????) ????12:23
??????: "dev"<dev@flink.apache.org>;
????: Re: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce
pre-aggregated merge to Table Store
Thanks Nathan for starting this discussion.
This [1] is a very good requirement to build the materialized view on Flink
Table Store.
## Aggregate Functions
For 'aggregate-function' = '{sum_field1:sum,max_field2:max}'.
Do you refer to any other systems? For us at Flink, a viable approach is
something like the Datagen connector [2]. Something like
`'fields.sum_field1.function'='sum'`.
## Default function
>> Tips: Columns which do not have designated aggregate functions using
newest value to overwrite old value.
Do you mean `replace`?
Is there anything about default functions that other systems can refer to?
## Supported functions
I'm not quite sure that the names of these functions are standard enough:
`replace_if_not_null/replace/concatenate`. Can you look at other systems?
You can also specify whether they support retraction messages.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-255+Introduce+pre-aggregated+merge+to+Table+Store
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/datagen/#connector-options
Best,
Jingsong
On Tue, Aug 2, 2022 at 11:09 AM ?????? <liguo...@bytedance.com> wrote:
> Hi Nathan,
>
> Seems a great proposal for table store aggregation.
> In the example, I think the 'max_field1' should be 1 instead of 2 after
> the max aggregation in the output result.
> And there may be a minor typo in the WITH clause, 'max_field2' ->
> 'max_field1'.
>
> Best,
> Guojun
>
> From: "Hannan Kan"<hannan...@foxmail.com>
> Date: Mon, Aug 1, 2022, 11:35 PM
> Subject: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce
> pre-aggregated merge to Table Store
> To: "dev"<dev@flink.apache.org>
> Cc: "lzljs3620320"<lzljs3620...@apache.org>
> Hi everyone, I would like to open a discussion on&nbsp;FLIP-255
Introduce
> pre-aggregated merge to table store&nbsp;[1]. Pre-aggregation
mechanism has
> been adopted by ma?6?7ny bi?6?7g d?6?7ata system?6?7?6?7s (such as Apache
Doris,?6?7&nbsp;
> Apache Kylin?6?7 , Druid etc.)&nbsp;to save storage and accelerate the
> aggregate query.?6?7 FLIP-255 proposes to introduce pre-aggregated merge
into
> Flink Table Store to acquire the same benefit.&nbsp; ?6?7Supported
aggregate
> functions include&nbsp;sum, max/min, count, replace_if_not_null,
> replace,&nbsp; concatenate, or/and?6?7. ?6?7 Looking forward to your
feedback.
>
[1]&nbsp;https://cwiki.apache.org/confluence/display?6?7/FLINK/FLIP-2?6?755+Introduce+pre-aggregated+merge+to+table+store
> Best, Nathan Kan (?6?7Hongnan Gan)?6?7?6?7?6?7
>
