carp84 commented on code in PR #601: URL: https://github.com/apache/flink-web/pull/601#discussion_r1067958425
########## _posts/2023-01-15-release-table-store-0.3.0.md: ########## @@ -0,0 +1,238 @@ +--- +layout: post +title: "Apache Flink Table Store 0.3.0 Release Announcement" +date: 2023-01-15T08:00:00.000Z +categories: news +authors: +- JingsongLi: + name: "Jingsong Lee" + +excerpt: The Apache Flink Community is pleased to announce the release for Flink Table Store 0.3.0! + +--- + +The Apache Flink community is pleased to announce the release of the +[Apache Flink Table Store](https://github.com/apache/flink-table-store) (0.3.0). + +We highly recommend all users upgrade to Flink Table Store 0.3.0. 0.3.0 completed 150+ issues, which were completed by nearly 30 contributors. + +Please check out the full [documentation]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/) for detailed information and user guides. + +<br/> + +Flink Table Store 0.3 completes many exciting features, Enhance its ability as a data lake and greatly +improve the availability of stream pipeline. Some important features are described below. Review Comment: ```suggestion Flink Table Store 0.3 completes many exciting features, enhances its ability as a data lake storage and greatly improves the availability of its stream pipeline. Some important features are described below. ``` ########## _posts/2023-01-15-release-table-store-0.3.0.md: ########## @@ -0,0 +1,238 @@ +--- +layout: post +title: "Apache Flink Table Store 0.3.0 Release Announcement" +date: 2023-01-15T08:00:00.000Z +categories: news +authors: +- JingsongLi: + name: "Jingsong Lee" + +excerpt: The Apache Flink Community is pleased to announce the release for Flink Table Store 0.3.0! + +--- + +The Apache Flink community is pleased to announce the release of the +[Apache Flink Table Store](https://github.com/apache/flink-table-store) (0.3.0). + +We highly recommend all users upgrade to Flink Table Store 0.3.0. 0.3.0 completed 150+ issues, which were completed by nearly 30 contributors. + +Please check out the full [documentation]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/) for detailed information and user guides. + +<br/> + +Flink Table Store 0.3 completes many exciting features, Enhance its ability as a data lake and greatly +improve the availability of stream pipeline. Some important features are described below. + +## Changelog Producer: Full-Compaction + +If: +- You are using `partial-update` or `aggregation` table, at the time of writing, table store can't know what the + result is after merging, so table store can't generate the corresponding changelog. +- Your input can’t produce a complete changelog but you still want to get rid of the costly normalized operator, + +You may consider using the [Full compaction changelog producer]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/features/table-types/#full-compaction). + +By specifying `'changelog-producer' = 'full-compaction'`, Table Store will compare the results between full compactions +and produce the differences as changelog. The latency of changelog is affected by the frequency of full compactions. +By specifying `changelog-producer.compaction-interval` table property (default value 30min), users can define the +maximum interval between two full compactions to ensure latency. + +<center> +<img src="{{site.baseurl}}/img/blog/table-store/changelog-producer-full-compaction.png" width="100%"/> +</center> + +<br/> + +## Dedicated Compaction Job && Multiple Writers + +By default, Table Store writers will perform compaction as needed when writing records. This is sufficient for most use cases, but there are two downsides: + +* This may result in unstable write throughput because throughput might temporarily drop when performing a compaction. +* Compaction will mark some data files as "deleted". If multiple writers mark the same file a conflict will occur when + committing the changes. Table Store will automatically resolve the conflict, but this may result in job restarts. + +To avoid these downsides, users can also choose to skip compactions in writers, and run a +[dedicated job only for compaction]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/maintenance/write-performance/#dedicated-compaction-job). +As compactions are performed only by the dedicated job, writers can continuously write records without pausing and no conflicts will ever occur. + +To skip compactions in writers, set `write-only` to `true`. + +To run a dedicated job for compaction, follow these instructions. + +Flink SQL currently does not support statements related to compactions, so we have to submit the compaction job through `flink run`. + +Run the following command to submit a compaction job for the table. + +```bash +<FLINK_HOME>/bin/flink run \ + -c org.apache.flink.table.store.connector.action.FlinkActions \ + /path/to/flink-table-store-dist-{{< version >}}.jar \ + compact \ + --warehouse <warehouse-path> \ + --database <database-name> \ + --table <table-name> +``` + +## Aggregation Table + +Sometimes users only care about aggregated results. The +[aggregation merge engine]({{site.DOCS_BASE_URL}}flink-table-store-docs-release-0.3/docs/features/table-types/#aggregation) +aggregates each value field with the latest data one by one under the same primary key according to the aggregate function. + +Each field not part of the primary keys must be given an aggregate function, specified by the Review Comment: ```suggestion Each field that is not part of the primary keys must be given an aggregate function, specified by the ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
