This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13122-f3f9baf39d4fecb901103cbf187c54119aeb3f3e in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 72e71f8c1e5b2c76939216b674601aab5a378168 Author: Nikhil kumar <[email protected]> AuthorDate: Sat Nov 7 06:36:03 2026 +0000 [Docs][Transform-V2] Rewrite FilterRowKind documentation (#13122) Co-authored-by: Nikhil <[email protected]> --- docs/en/transforms/filter-rowkind.md | 240 +++++++++++++++++++++++++++++++---- docs/zh/transforms/filter-rowkind.md | 240 +++++++++++++++++++++++++++++++---- 2 files changed, 430 insertions(+), 50 deletions(-) diff --git a/docs/en/transforms/filter-rowkind.md b/docs/en/transforms/filter-rowkind.md index 14946b2a1bd..3d0b9930a48 100644 --- a/docs/en/transforms/filter-rowkind.md +++ b/docs/en/transforms/filter-rowkind.md @@ -1,54 +1,110 @@ # FilterRowKind -> FilterRowKind transform plugin +## Support Those Engines + +> SeaTunnel Zeta<br/> +> Flink<br/> +> Spark ## Description -Filter the data by RowKind +FilterRowKind keeps or drops rows according to their change kind. Use it to select events such as inserts or deletes from an upstream changelog. It does not convert one row kind into another: retained rows keep their values, schema and row kind. Filtering out a row removes that event from the downstream stream. + +:::caution +Dropping update or delete events changes the changelog received by the sink. Choose a rule that matches the downstream use case; filtering is not a substitute for applying a complete CDC changelog. +::: ## Options -| name | type | required | default value | -|---------------|-------|----------|---------------| -| include_kinds | array | no | | -| exclude_kinds | array | no | | +| Name | Type | Required | Default | Description | +|------|------|----------|---------|-------------| +| include_kinds | Array<Enum> | Conditional | - | Keep only the listed row kinds. Mutually exclusive with `exclude_kinds`; the list must not be empty. | +| exclude_kinds | Array<Enum> | Conditional | - | Drop the listed row kinds. Mutually exclusive with `include_kinds`; the list must not be empty. | +| table_match_regex | String | No | `.*` | Match upstream table paths to which the transform-level rule applies. | +| table_transform | Array<Map> | No | `[]` | Complete rules for individual tables, selected by an exact `table_path`. | +| rule_match_mode | Enum | No | `FIRST_MATCH` | Select the first matching per-table rule or apply all matching rules in order. See the duplicate-path behavior below. | +| common options | - | No | - | Common transform options such as `plugin_input` and `plugin_output`. | + +### include_kinds [Array<Enum>] + +Keep a row only when its row kind is in this list, for example `include_kinds = ["INSERT", "UPDATE_AFTER"]`. Accepted values are: + +| Value | Meaning | +|-------|---------| +| `INSERT` | An inserted row. | +| `UPDATE_BEFORE` | The previous contents of an updated row. | +| `UPDATE_AFTER` | The new contents of an updated row. | +| `DELETE` | A deleted row. | + +Without a nonempty `table_transform` list, configure exactly one of `include_kinds` and `exclude_kinds` at the transform level. Configuring both, neither, or an empty selected list fails validation. An empty `table_transform = []` does not remove this requirement. + +With per-table entries, the transform-level include/exclude rule may be absent. If supplied, it must still contain exactly one nonempty list. Each per-table entry must independently satisfy the same rule. + +### exclude_kinds [Array<Enum>] + +Drop rows whose row kind is in this list and keep all other rows. It accepts the same four values as `include_kinds` and has no default. For example, `exclude_kinds = ["DELETE", "UPDATE_BEFORE"]` keeps inserts and after-update rows. + +Do not configure both options in the same rule. A per-table rule can use a different option from the transform-level rule because it replaces that rule for its table. + +### table_match_regex [String] + +A Java regular expression matched against the whole upstream table path, such as `test.abc` or `database.schema.table`. The default `.*` matches all table paths. This is the actual table path, not the dataset name in `plugin_input` or `plugin_output`. + +This option selects tables for the transform-level rule. It does not restrict explicit `table_transform` entries, and it does not create a filtering rule when the transform-level include/exclude options are absent. + +### table_transform [Array<Map>] + +A list of per-table rules; the default is an empty list. Each entry must have a nonblank `table_path` and exactly one nonempty `include_kinds` or `exclude_kinds` list. `table_path` is an exact path, not a regular expression. + +An entry is a complete rule and does not inherit the transform-level include/exclude list. A matching entry takes precedence even when its table does not match `table_match_regex`. + +### rule_match_mode [Enum] + +Controls multiple entries with the same exact `table_path`: + +- Explicit `FIRST_MATCH`: use the first matching entry in list order. +- Explicit `ALL_MATCH`: apply all matching entries in list order. A row must survive every filter; this is not a union of the retained kinds. -Exactly one of `include_kinds` and `exclude_kinds` must be configured; configuring both or neither fails validation. +Although the option's default is `FIRST_MATCH`, omitting the option rejects duplicate `table_path` entries during transform initialization. Set it explicitly if repeated paths are intentional. With unique paths, the default is sufficient. -### include_kinds [array] +### common options -The row kinds to include +Use `plugin_input` and `plugin_output` to connect this transform to upstream and downstream plugins. See [Transform Plugin](common-options/common-options.md) for common parameters. -### exclude_kinds [array] +## Multi-Table Configuration -The row kinds to exclude. +For each input table, the effective rule is selected in this order: -You must configure exactly one of `include_kinds` and `exclude_kinds`; configuring both is not allowed. +1. Use the matching `table_transform` entry or entries, according to `rule_match_mode`. +2. Otherwise, apply the transform-level rule if one is configured and the table path matches `table_match_regex`. +3. Otherwise, pass the table's rows through unchanged. -### common options [string] +In a `table_transform`-only configuration, unlisted tables pass through unchanged. FilterRowKind filters rows, not catalog table definitions; a table can remain in the output schema even when all its rows are dropped. -Transform plugin common parameters, please refer to [Transform Plugin](common-options/common-options.md) for details +The following examples cover all three configuration shapes. See [Multi-Table Transform](transform-multi-table.md) for the shared matching rules. ## Examples -The RowKind of the data generated by FakeSource is `INSERT`. If we use the `FilterRowKind` transform and exclude the `INSERT` data, zero rows will be written into the sink. +These complete batch configurations use FakeSource and Console. Install those connectors in your SeaTunnel distribution before running a saved configuration with `bin/seatunnel.sh --config <file> -e local`. The generated rows in these examples have kind `INSERT`; their random field values do not affect the expected row counts. -```yaml +### Transform-level rule only +Input: three `INSERT` rows from `test.abc`. Output: zero rows, because the rule excludes `INSERT`. + +```hocon env { job.mode = "BATCH" + parallelism = 1 } source { FakeSource { plugin_output = "fake" - row.num = 100 - schema = { - fields { - id = "int" - name = "string" - age = "int" - } + parallelism = 1 + row.num = 3 + schema { + table = "test.abc" + fields { id = "int" } } } } @@ -56,14 +112,148 @@ source { transform { FilterRowKind { plugin_input = "fake" - plugin_output = "fake1" + plugin_output = "filtered" + exclude_kinds = ["INSERT"] + } +} + +sink { + Console { + plugin_input = "filtered" + } +} +``` + +### Transform-level rule with per-table rules + +Each input table contains three `INSERT` rows. The explicit entry for `test.xyz` overrides the transform-level exclusion. `test.www` does not match the regex and passes through unchanged. + +| Table | Input rows | Output rows | Effective rule | +|-------|------------|-------------|----------------| +| `test.abc` | 3 | 0 | Transform-level `exclude_kinds = ["INSERT"]`. | +| `test.xyz` | 3 | 3 | Per-table `include_kinds = ["INSERT"]`. | +| `test.www` | 3 | 3 | No matching rule; unchanged. | + +```hocon +env { + job.mode = "BATCH" + parallelism = 1 +} + +source { + FakeSource { + plugin_output = "fake" + parallelism = 1 + tables_configs = [ + { + row.num = 3 + schema { + table = "test.abc" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.xyz" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.www" + columns = [{ name = "id", type = "int" }] + } + } + ] + } +} + +transform { + FilterRowKind { + plugin_input = "fake" + plugin_output = "filtered" + table_match_regex = "test[.](abc|xyz)" exclude_kinds = ["INSERT"] + table_transform = [ + { + table_path = "test.xyz" + include_kinds = ["INSERT"] + } + ] + } +} + +sink { + Console { + plugin_input = "filtered" + } +} +``` + +### Per-table rules only + +There is no transform-level include/exclude rule. Only `test.xyz` drops its inserts; the other tables pass through unchanged. + +| Table | Input rows | Output rows | +|-------|------------|-------------| +| `test.abc` | 3 | 3 | +| `test.xyz` | 3 | 0 | +| `test.www` | 3 | 3 | + +```hocon +env { + job.mode = "BATCH" + parallelism = 1 +} + +source { + FakeSource { + plugin_output = "fake" + parallelism = 1 + tables_configs = [ + { + row.num = 3 + schema { + table = "test.abc" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.xyz" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.www" + columns = [{ name = "id", type = "int" }] + } + } + ] + } +} + +transform { + FilterRowKind { + plugin_input = "fake" + plugin_output = "filtered" + table_transform = [ + { + table_path = "test.xyz" + exclude_kinds = ["INSERT"] + } + ] } } sink { Console { - plugin_input = "fake1" + plugin_input = "filtered" } } ``` diff --git a/docs/zh/transforms/filter-rowkind.md b/docs/zh/transforms/filter-rowkind.md index 47ef0ded085..1b3af3f79ea 100644 --- a/docs/zh/transforms/filter-rowkind.md +++ b/docs/zh/transforms/filter-rowkind.md @@ -1,54 +1,110 @@ # 行类型过滤 -> FilterRowKind:按插入、更新、删除等行类型筛选数据 +## 支持的引擎 + +> SeaTunnel Zeta<br/> +> Flink<br/> +> Spark ## 描述 -FilterRowKind 转换插件用于按 RowKind 过滤数据。 +FilterRowKind 根据行的变更类型保留或丢弃数据。可以用它从上游变更流中选择插入、删除等事件。插件不会把一种行类型转换成另一种:保留的行仍具有原来的字段值、模式和行类型。被过滤的行不会继续传递到下游。 + +:::caution +丢弃更新或删除事件会改变 Sink 收到的变更流。请根据下游用途选择规则;行过滤不能替代对完整 CDC 变更流的处理。 +::: ## 属性 -| 名称 | 类型 | 是否必须 | 默认值 | -|---------------|-------|------|-----| -| include_kinds | array | no | | -| exclude_kinds | array | no | | +| 名称 | 类型 | 是否必须 | 默认值 | 描述 | +|------|------|----------|--------|------| +| include_kinds | Array<Enum> | 条件必需 | - | 只保留列表中的行类型。与 `exclude_kinds` 互斥,列表不能为空。 | +| exclude_kinds | Array<Enum> | 条件必需 | - | 丢弃列表中的行类型。与 `include_kinds` 互斥,列表不能为空。 | +| table_match_regex | String | 否 | `.*` | 匹配要应用转换级规则的上游表路径。 | +| table_transform | Array<Map> | 否 | `[]` | 通过精确的 `table_path` 为各表指定完整规则。 | +| rule_match_mode | Enum | 否 | `FIRST_MATCH` | 选择第一条匹配的表级规则,或按顺序执行所有匹配的规则。重复路径的行为见下文。 | +| common options | - | 否 | - | `plugin_input`、`plugin_output` 等转换插件通用参数。 | + +### include_kinds [Array<Enum>] + +仅保留行类型在列表中的数据,例如 `include_kinds = ["INSERT", "UPDATE_AFTER"]`。支持以下值: + +| 值 | 含义 | +|----|------| +| `INSERT` | 插入的行。 | +| `UPDATE_BEFORE` | 更新前的行内容。 | +| `UPDATE_AFTER` | 更新后的行内容。 | +| `DELETE` | 删除的行。 | + +没有非空 `table_transform` 列表时,转换级配置必须且只能指定 `include_kinds` 和 `exclude_kinds` 中的一个。两者都配置、两者都不配置或所选列表为空,都会导致校验失败。`table_transform = []` 不能免除这个要求。 + +配置了表级条目时,可以不设置转换级的包含/排除规则;如果设置了,仍必须且只能包含一个非空列表。每个表级条目也必须独立满足相同要求。 + +### exclude_kinds [Array<Enum>] + +丢弃行类型在列表中的数据,保留其他行。支持的四个值与 `include_kinds` 相同,没有默认值。例如 `exclude_kinds = ["DELETE", "UPDATE_BEFORE"]` 会保留插入行和更新后的行。 + +同一条规则不能同时配置这两个选项。表级规则会替代该表的转换级规则,因此可以选择与转换级规则不同的选项。 + +### table_match_regex [String] + +针对完整上游表路径匹配的 Java 正则表达式,例如 `test.abc` 或 `database.schema.table`。默认值 `.*` 匹配所有表路径。这里匹配的是实际表路径,不是 `plugin_input` 或 `plugin_output` 中的数据集名称。 + +此选项为转换级规则选择表,不限制显式的 `table_transform` 条目。未配置转换级包含/排除选项时,仅设置此正则表达式不会产生过滤规则。 + +### table_transform [Array<Map>] + +表级规则列表,默认为空列表。每个条目必须包含非空白的 `table_path`,并且必须且只能包含一个非空的 `include_kinds` 或 `exclude_kinds` 列表。`table_path` 是精确路径,不是正则表达式。 + +每个条目都是完整规则,不继承转换级的包含/排除列表。匹配的条目具有更高优先级,即使该表不匹配 `table_match_regex` 也会使用该条目。 + +### rule_match_mode [Enum] + +控制多个条目具有相同精确 `table_path` 时的行为: + +- 显式设置 `FIRST_MATCH`:按列表顺序使用第一条匹配的条目。 +- 显式设置 `ALL_MATCH`:按列表顺序执行所有匹配的条目。只有通过每个过滤器的行才会保留;这不是对各条目保留类型取并集。 -`include_kinds` 和 `exclude_kinds` 必须且只能配置其中一个,两个都配置或都不配置都会导致校验失败。 +虽然选项的默认值为 `FIRST_MATCH`,但省略此选项时,重复的 `table_path` 会在转换初始化阶段被拒绝。如果确实需要重复路径,请显式设置此选项。路径不重复时,使用默认值即可。 -### include_kinds [array] +### common options -要包含的行类型 +使用 `plugin_input` 和 `plugin_output` 连接上游和下游插件。通用参数详见[转换插件](common-options/common-options.md)。 -### exclude_kinds [array] +## 多表配置 -要排除的行类型。 +对于每个输入表,按照以下顺序选择有效规则: -您必须且只能配置 `include_kinds` 和 `exclude_kinds` 中的一个,不能同时配置。 +1. 根据 `rule_match_mode` 使用匹配的一个或多个 `table_transform` 条目。 +2. 否则,如果配置了转换级规则且表路径匹配 `table_match_regex`,使用转换级规则。 +3. 否则,原样传递该表的行。 -### common options [string] +仅配置 `table_transform` 时,未列出的表保持不变。FilterRowKind 过滤的是行,而不是目录表定义;即使某个表的所有行都被丢弃,其模式仍可保留在输出中。 -转换插件的常见参数, 请参考 [Transform Plugin](common-options/common-options.md) 了解详情 +下面的示例涵盖三种配置方式。共享匹配规则详见[多表转换](transform-multi-table.md)。 ## 示例 -FakeSource 生成的数据的行类型是 `INSERT`。如果我们使用 `FilterRowKind` 转换并排除 `INSERT` 数据,我们将不会向接收器写入任何行。 +以下完整批处理配置使用 FakeSource 和 Console。在 SeaTunnel 发行包中安装这两个连接器后,可以将配置保存为文件,并通过 `bin/seatunnel.sh --config <file> -e local` 运行。这些示例生成的行类型为 `INSERT`,随机字段值不影响预期行数。 -```yaml +### 仅使用转换级规则 +输入:`test.abc` 的三条 `INSERT` 行。输出:零行,因为规则排除了 `INSERT`。 + +```hocon env { job.mode = "BATCH" + parallelism = 1 } source { FakeSource { plugin_output = "fake" - row.num = 100 - schema = { - fields { - id = "int" - name = "string" - age = "int" - } + parallelism = 1 + row.num = 3 + schema { + table = "test.abc" + fields { id = "int" } } } } @@ -56,14 +112,148 @@ source { transform { FilterRowKind { plugin_input = "fake" - plugin_output = "fake1" + plugin_output = "filtered" + exclude_kinds = ["INSERT"] + } +} + +sink { + Console { + plugin_input = "filtered" + } +} +``` + +### 转换级规则与表级规则组合 + +每个输入表都有三条 `INSERT` 行。`test.xyz` 的显式条目覆盖转换级排除规则。`test.www` 不匹配正则表达式,因此保持不变。 + +| 表 | 输入行数 | 输出行数 | 有效规则 | +|----|----------|----------|----------| +| `test.abc` | 3 | 0 | 转换级 `exclude_kinds = ["INSERT"]`。 | +| `test.xyz` | 3 | 3 | 表级 `include_kinds = ["INSERT"]`。 | +| `test.www` | 3 | 3 | 没有匹配的规则,保持不变。 | + +```hocon +env { + job.mode = "BATCH" + parallelism = 1 +} + +source { + FakeSource { + plugin_output = "fake" + parallelism = 1 + tables_configs = [ + { + row.num = 3 + schema { + table = "test.abc" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.xyz" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.www" + columns = [{ name = "id", type = "int" }] + } + } + ] + } +} + +transform { + FilterRowKind { + plugin_input = "fake" + plugin_output = "filtered" + table_match_regex = "test[.](abc|xyz)" exclude_kinds = ["INSERT"] + table_transform = [ + { + table_path = "test.xyz" + include_kinds = ["INSERT"] + } + ] + } +} + +sink { + Console { + plugin_input = "filtered" + } +} +``` + +### 仅使用表级规则 + +未配置转换级包含/排除规则。只有 `test.xyz` 的插入行被丢弃,其他表保持不变。 + +| 表 | 输入行数 | 输出行数 | +|----|----------|----------| +| `test.abc` | 3 | 3 | +| `test.xyz` | 3 | 0 | +| `test.www` | 3 | 3 | + +```hocon +env { + job.mode = "BATCH" + parallelism = 1 +} + +source { + FakeSource { + plugin_output = "fake" + parallelism = 1 + tables_configs = [ + { + row.num = 3 + schema { + table = "test.abc" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.xyz" + columns = [{ name = "id", type = "int" }] + } + }, + { + row.num = 3 + schema { + table = "test.www" + columns = [{ name = "id", type = "int" }] + } + } + ] + } +} + +transform { + FilterRowKind { + plugin_input = "fake" + plugin_output = "filtered" + table_transform = [ + { + table_path = "test.xyz" + exclude_kinds = ["INSERT"] + } + ] } } sink { Console { - plugin_input = "fake1" + plugin_input = "filtered" } } ```
