This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12443-c2e900d973400d47405d0dd7e4d3081c89ac80c2 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit f54bb5876bd282b88181b6d93c9f59ec5dd38621 Author: Tiefang Hu <[email protected]> AuthorDate: Sat Nov 7 14:21:51 2026 +0000 [Fix][Core] Preserve literal config keys across REST and file loading (#12443) Signed-off-by: hutiefang76 <[email protected]> --- docs/en/engines/zeta/rest-api-v2.md | 2 + docs/en/introduction/concepts/config.md | 4 + docs/zh/engines/zeta/rest-api-v2.md | 2 + docs/zh/introduction/concepts/config.md | 4 + .../api/configuration/ReadonlyConfig.java | 4 +- .../api/configuration/util/ConfigMapPathUtils.java | 72 ++++++++ .../api/configuration/ReadableConfigTest.java | 9 + .../core/starter/utils/ConfigBuilder.java | 30 ++-- .../core/starter/utils/ConfigShadeUtils.java | 3 +- .../core/starter/utils/ConfigBuilderTest.java | 200 +++++++++++++++++++++ .../org/apache/seatunnel/engine/e2e/RestApiIT.java | 71 +++++++- .../RestApiSubmitJobConfigShadeDecryptTest.java | 101 +++++++++++ .../engine/server/utils/RestUtilTest.java | 153 ++++++++++++++++ 13 files changed, 638 insertions(+), 17 deletions(-) diff --git a/docs/en/engines/zeta/rest-api-v2.md b/docs/en/engines/zeta/rest-api-v2.md index 09d2d5053f9..c05f7ef575a 100644 --- a/docs/en/engines/zeta/rest-api-v2.md +++ b/docs/en/engines/zeta/rest-api-v2.md @@ -961,6 +961,8 @@ Each member is asked in parallel and awaited against one shared deadline (`seatu #### Body You can choose json, hocon or sql to pass request body. +For JSON submissions, every key is parsed as a SeaTunnel config path expression: `a->b` creates nested config, while a dot such as in `job.mode` is not a path separator. Keys whose path expression cannot be parsed (for example regex field names such as `^t_nova_.*$`, or keys containing reserved characters such as `$`, `:` or `${...}`) are kept as literal keys instead of failing the request. Parsable keys keep the existing semantics: quotes are removed and surrounding unquoted whitespace [...] +For HOCON submissions the body is parsed by the HOCON parser first, so a key containing a reserved character must be quoted in the body: write `"$systemId" = "string"` instead of `$systemId = "string"`. Quoting a key in HOCON does not override the path semantics used during decryption. Keys rejected by the path parser are kept literal; valid expressions such as `a->b` are parsed as paths. The json format example: ``` json { diff --git a/docs/en/introduction/concepts/config.md b/docs/en/introduction/concepts/config.md index ed9471be8b5..6017b69bfa5 100644 --- a/docs/en/introduction/concepts/config.md +++ b/docs/en/introduction/concepts/config.md @@ -141,6 +141,10 @@ When a job contains multiple sources, transforms, or sinks, SeaTunnel needs a wa In simple one-source jobs, you can often omit them because SeaTunnel uses a default convention and passes the previous module's output forward automatically. +### Config Keys And Path Expressions + +SeaTunnel's config path separator is `->`, so `a->b = 1` defines the nested `a.b`, while the dot in `job.mode` is part of the key name; every config file load runs keys through that path parser, including configs read by a config adapter and the internal re-parse that serves `-i` variable substitution even when no `-i` value is passed. A key that is a valid path expression is still parsed as a path (quotes are dropped and surrounding unquoted whitespace is trimmed), so a quoted `"a->b" = [...] + ## Multi-line Support In `hocon`, multiline strings are supported, which allows you to include extended passages of text without worrying about newline characters or special formatting. This is achieved by enclosing the text within triple quotes **`"""`** . For example: diff --git a/docs/zh/engines/zeta/rest-api-v2.md b/docs/zh/engines/zeta/rest-api-v2.md index 30c37f5e2f6..910b90d78fe 100644 --- a/docs/zh/engines/zeta/rest-api-v2.md +++ b/docs/zh/engines/zeta/rest-api-v2.md @@ -947,6 +947,8 @@ seatunnel: #### 请求体 你可以选择用json、hocon或者sql的方式来传递请求体。 +JSON 请求中,每个键都会按 SeaTunnel 配置路径表达式解析:`a->b` 会生成嵌套配置,而 `job.mode` 中的点号不是路径分隔符。无法解析为路径表达式的键(例如正则字段名 `^t_nova_.*$`,或包含 `$`、`:`、`${...}` 等保留字符的键)会作为字面键保留,而不会导致请求失败;能够解析的键仍遵循原有语义:引号会被去除,未加引号的首尾空白会被裁剪。批量提交也遵循同一规则。 +HOCON 请求体会先由 HOCON 解析器解析,因此包含保留字符的键必须在请求体中加引号:应写 `"$systemId" = "string"`,而不是 `$systemId = "string"`。HOCON 中的引号不会覆盖解密时的路径语义。路径解析器拒绝的键会保持为字面键;`a->b` 等合法表达式仍按路径解析。 Json请求示例: ```json { diff --git a/docs/zh/introduction/concepts/config.md b/docs/zh/introduction/concepts/config.md index 6befacb32fd..13b99c4e4b6 100644 --- a/docs/zh/introduction/concepts/config.md +++ b/docs/zh/introduction/concepts/config.md @@ -141,6 +141,10 @@ sink { 如果只是单一 source 的简单链路,很多时候可以省略它们,因为 SeaTunnel 会按默认约定自动把上一个模块的输出继续往下传递。 +### 配置键与路径表达式 + +SeaTunnel 的配置路径分隔符是 `->`,因此 `a->b = 1` 表示嵌套的 `a.b`,而 `job.mode` 中的点号属于键名本身;加载配置文件时所有键都会经过该路径解析器,包括通过配置适配器读取的配置,以及为 `-i` 变量替换服务的内部重新解析(即使没有传入任何 `-i` 值也会执行)。合法的路径表达式仍会按路径解析(引号会被去除、未加引号的首尾空白会被裁剪),因此加引号的 `"a->b" = 1` 会被存为嵌套的 `a.b`;解析器拒绝的键则会作为单个字面键保留,而不会导致加载失败,例如正则字段名 `^t_nova_.*$`,或包含 `$`、`:`、`${...}` 的键。HOCON 本身也会把其中部分字符当作语法,因此这类键必须在文件中加引号:应写 `"$systemId" = "string"`,而不是 `$systemId = "string"`。REST 提交遵循同一规则,详见 [REST API V2](../../engines/zeta/rest-api-v2.md)。 + ## 多行文本支持 `hocon`支持多行字符串,这样就可以包含较长的文本段落,而不必担心换行符或特殊格式。这可以通过将文本括在三层引号 **`"""`** 中来实现。例如: diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java index 0ac179ae764..45043aadb8f 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java @@ -24,6 +24,8 @@ import org.apache.seatunnel.shade.com.typesafe.config.Config; import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; import org.apache.seatunnel.shade.com.typesafe.config.ConfigRenderOptions; +import org.apache.seatunnel.api.configuration.util.ConfigMapPathUtils; + import lombok.extern.slf4j.Slf4j; import java.io.Serializable; @@ -74,7 +76,7 @@ public class ReadonlyConfig implements Serializable { */ @Deprecated public Config toConfig() { - return ConfigFactory.parseMap(confData); + return ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(confData)); } public Map<String, String> toMap() { diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigMapPathUtils.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigMapPathUtils.java new file mode 100644 index 00000000000..65d328cd992 --- /dev/null +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigMapPathUtils.java @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.api.configuration.util; + +import org.apache.seatunnel.shade.com.typesafe.config.ConfigException; +import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; +import org.apache.seatunnel.shade.com.typesafe.config.ConfigUtil; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** Prepares JSON-derived maps for config parsing without changing valid path expressions. */ +public final class ConfigMapPathUtils { + + private ConfigMapPathUtils() {} + + /** + * {@link ConfigFactory#parseMap(Map)} parses every map key as a path, including nested maps and + * maps inside lists. Quote only keys rejected by the same path parser, so regex and other JSON + * literal keys survive each map-to-config conversion while valid path expressions retain their + * existing meaning. A rendered Config loses this quoting, so each later parse must call this + * method again. + */ + public static Map<String, Object> quoteInvalidPathKeys(Map<String, Object> objectMap) { + Map<String, Object> result = new LinkedHashMap<>(); + objectMap.forEach( + (key, value) -> + result.put(quoteInvalidPathKey(key), quoteInvalidPathKeysInValue(value))); + return result; + } + + @SuppressWarnings("unchecked") + private static Object quoteInvalidPathKeysInValue(Object value) { + if (value instanceof Map<?, ?>) { + return quoteInvalidPathKeys((Map<String, Object>) value); + } + if (value instanceof List<?>) { + List<Object> result = new ArrayList<>(); + for (Object item : (List<?>) value) { + result.add(quoteInvalidPathKeysInValue(item)); + } + return result; + } + return value; + } + + private static String quoteInvalidPathKey(String key) { + try { + ConfigUtil.splitPath(key); + return key; + } catch (ConfigException.BadPath ignored) { + return ConfigUtil.quoteString(key); + } + } +} diff --git a/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/ReadableConfigTest.java b/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/ReadableConfigTest.java index 24bd843ee77..a5c4a6b1bc6 100644 --- a/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/ReadableConfigTest.java +++ b/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/ReadableConfigTest.java @@ -40,6 +40,15 @@ public class ReadableConfigTest { private static ReadonlyConfig config; private static Map<String, Object> map; + @Test + public void testToConfigPreservesInvalidPathKey() { + Map<String, Object> fields = new HashMap<>(); + fields.put("^t_nova_.*$", "string"); + + Config converted = ReadonlyConfig.fromMap(fields).toConfig(); + Assertions.assertEquals("string", converted.root().unwrapped().get("^t_nova_.*$")); + } + @BeforeAll public static void prepare() throws URISyntaxException { Config rawConfig = diff --git a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigBuilder.java b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigBuilder.java index 754695826d2..083bcc7fabd 100644 --- a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigBuilder.java +++ b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigBuilder.java @@ -26,6 +26,7 @@ import org.apache.seatunnel.shade.com.typesafe.config.ConfigSyntax; import org.apache.seatunnel.shade.com.typesafe.config.impl.Parseable; import org.apache.seatunnel.api.configuration.ConfigAdapter; +import org.apache.seatunnel.api.configuration.util.ConfigMapPathUtils; import org.apache.seatunnel.api.sink.TablePlaceholder; import org.apache.seatunnel.common.utils.ConfigValueUtils; import org.apache.seatunnel.common.utils.JsonUtils; @@ -107,7 +108,7 @@ public class ConfigBuilder { public static Config of(@NonNull Map<String, Object> objectMap) { log.info("Loading config file from objectMap"); Config config = - ConfigFactory.parseMap(objectMap) + ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(objectMap)) .resolve(ConfigResolveOptions.defaults().setAllowUnresolved(true)) .resolveWith( ConfigFactory.systemProperties(), @@ -238,7 +239,8 @@ public class ConfigBuilder { log.info("With config adapter spi {}", configAdapter.getClass().getName()); try { Map<String, Object> flattenedMap = configAdapter.loadConfig(filePath); - Config config = ConfigFactory.parseMap(flattenedMap); + Config config = + ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(flattenedMap)); return ConfigShadeUtils.decryptConfig(backfillUserVariables(config, variables)); } catch (ParserException | IllegalArgumentException e) { throw e; @@ -258,21 +260,22 @@ public class ConfigBuilder { processVariablesMap(resolvedConfigMap, userConfigMap); - return ConfigFactory.parseMap(resolvedConfigMap); + // unwrapped() drops the quoting of literal keys, so re-quote before parsing the map. + return ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(resolvedConfigMap)); } public static Config substituteUserConfig( Config originalConfig, Map<String, String> userConfigMap) { + Map<String, Object> userVariableMap = + userConfigMap.entrySet().stream() + .collect( + Collectors.toMap( + Map.Entry::getKey, + entry -> + ConfigValueUtils.parseValue(entry.getValue()) + .unwrapped())); Config userConfig = - ConfigFactory.parseMap( - userConfigMap.entrySet().stream() - .collect( - Collectors.toMap( - Map.Entry::getKey, - entry -> - ConfigValueUtils.parseValue( - entry.getValue()) - .unwrapped()))); + ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(userVariableMap)); Config systemConfig = Parseable.newProperties( @@ -285,7 +288,8 @@ public class ConfigBuilder { // Priority: userConfig > systemProperties Map<String, Object> mergedMap = new LinkedHashMap<>(systemConfig.root().unwrapped()); mergedMap.putAll(userConfig.root().unwrapped()); - Config sourceConfig = ConfigFactory.parseMap(mergedMap); + Config sourceConfig = + ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(mergedMap)); return originalConfig.resolveWith( sourceConfig, ConfigResolveOptions.defaults().setAllowUnresolved(true)); diff --git a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java index 8ec4b875a35..74b5a815971 100644 --- a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java +++ b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigShadeUtils.java @@ -24,6 +24,7 @@ import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; import org.apache.seatunnel.shade.com.typesafe.config.ConfigRenderOptions; import org.apache.seatunnel.api.configuration.ConfigShade; +import org.apache.seatunnel.api.configuration.util.ConfigMapPathUtils; import org.apache.seatunnel.common.Constants; import org.apache.seatunnel.common.config.TypesafeConfigUtils; import org.apache.seatunnel.common.utils.JsonUtils; @@ -225,7 +226,7 @@ public final class ConfigShadeUtils { configMap.put(Constants.SOURCE, sources); configMap.put(Constants.SINK, sinks); configMap.put(Constants.TRANSFORM, transforms); - return ConfigFactory.parseMap(configMap); + return ConfigFactory.parseMap(ConfigMapPathUtils.quoteInvalidPathKeys(configMap)); } catch (Exception e) { // Log desensitized error information log.error( diff --git a/seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigBuilderTest.java b/seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigBuilderTest.java index 9a86c43b98b..1b51d4c3f3b 100644 --- a/seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigBuilderTest.java +++ b/seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigBuilderTest.java @@ -17,19 +17,153 @@ package org.apache.seatunnel.core.starter.utils; +import org.apache.seatunnel.shade.com.typesafe.config.Config; +import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; +import org.apache.seatunnel.shade.com.typesafe.config.ConfigUtil; + +import org.apache.seatunnel.api.configuration.ConfigAdapter; import org.apache.seatunnel.core.starter.exception.ConfigCheckException; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + public class ConfigBuilderTest { + @Test + public void testInvalidPathKeysSurviveConfigShadeRoundTrip() { + Map<String, Object> fields = new LinkedHashMap<>(); + fields.put("^t_nova_.*$", "string"); + fields.put("a:b", "long"); + fields.put("a\"b\\c", "string"); + fields.put("${FOO}", "boolean"); + fields.put("", "int"); + Map<String, Object> schema = new LinkedHashMap<>(); + schema.put("fields", fields); + Map<String, Object> source = new LinkedHashMap<>(); + source.put("schema", schema); + Map<String, Object> configMap = new LinkedHashMap<>(); + configMap.put("source", Arrays.asList(source)); + configMap.put("sink", Arrays.asList(new LinkedHashMap<>())); + + Config config = ConfigShadeUtils.decryptConfig(ConfigBuilder.of(configMap)); + Map<?, ?> actualFields = + config.getConfigList("source") + .get(0) + .getConfig("schema") + .getConfig("fields") + .root() + .unwrapped(); + Assertions.assertEquals(fields, actualFields); + } + + @Test + public void testWhitespacePaddedKeysTrimmedUnlessGuardQuotesWholeKey() { + Map<String, Object> fields = new LinkedHashMap<>(); + // Parsable keys keep path semantics: surrounding unquoted whitespace is trimmed. + fields.put(" padded ", "trimmed"); + fields.put(" job.mode ", "dotted"); + // Unparsable keys: the guard quotes each whole key, so padding stays literal. + fields.put(" ^t_nova_.*$ ", "regex"); + fields.put(" ${FOO} ", "literal"); + Map<String, Object> schema = new LinkedHashMap<>(); + schema.put("fields", fields); + Map<String, Object> source = new LinkedHashMap<>(); + source.put("schema", schema); + Map<String, Object> configMap = new LinkedHashMap<>(); + configMap.put("source", Arrays.asList(source)); + configMap.put("sink", Arrays.asList(new LinkedHashMap<>())); + + assertWhitespacePaddedKeys(ConfigBuilder.of(configMap)); + assertWhitespacePaddedKeys(ConfigShadeUtils.decryptConfig(ConfigBuilder.of(configMap))); + } + + @Test + public void testQuotedLiteralKeysSurviveConfigFileLoad(@TempDir Path tempDir) + throws IOException { + Config config = ConfigBuilder.of(writeLiteralKeyConfigFile(tempDir, false)); + + Assertions.assertEquals("BATCH", config.getConfig("env").getString("job.mode")); + assertLiteralSchemaFields(config); + } + + @Test + public void testQuotedLiteralKeysSurviveConfigFileLoadWithUserVariables(@TempDir Path tempDir) + throws IOException { + Config config = + ConfigBuilder.of( + writeLiteralKeyConfigFile(tempDir, true), + Arrays.asList("jobName=quoted_key_job")); + + Assertions.assertEquals("quoted_key_job", config.getConfig("env").getString("job.name")); + assertLiteralSchemaFields(config); + } + + /** The distinct fallback value exposes an unintended fallback after a valid adapter result. */ + @Test + public void testConfigAdapterPathKeepsLiteralKeysAndValues(@TempDir Path tempDir) + throws IOException { + Path configFile = tempDir.resolve("fallback.conf"); + Files.write( + configFile, + "origin = \"fallback\"\nsource { FakeSource {} }\nsink { Console {} }\n" + .getBytes(StandardCharsets.UTF_8)); + + Map<String, Object> adapterConfig = new LinkedHashMap<>(); + adapterConfig.put("origin", "adapter"); + adapterConfig.put( + "source", + Arrays.asList( + Collections.singletonMap( + "schema", + Collections.singletonMap("fields", literalSchemaFields())))); + adapterConfig.put( + "sink", Arrays.asList(Collections.singletonMap("plugin_name", "Console"))); + + ConfigAdapter adapter = mock(ConfigAdapter.class); + when(adapter.loadConfig(configFile)).thenReturn(adapterConfig); + + Config config = ConfigBuilder.of(adapter, configFile, Collections.<String>emptyList()); + + Assertions.assertEquals("adapter", config.getString("origin")); + assertLiteralSchemaFields(config); + } + + /** Quoted values remain literal for ConfigBuilder's variable substitution. */ + @Test + public void testQuotedPlaceholderIsSubstitutedFromUserVariables(@TempDir Path tempDir) + throws IOException { + String configText = + "env { job.name = \"prefix-${jobName$}\" }\nsource { FakeSource {} }\nsink { Console {} }\n"; + + // HOCON leaves quoted placeholders for ConfigBuilder to resolve. + Assertions.assertEquals( + "prefix-${jobName$}", + ConfigFactory.parseString(configText).getConfig("env").getString("job.name")); + + Path configFile = tempDir.resolve("variables.conf"); + Files.write(configFile, configText.getBytes(StandardCharsets.UTF_8)); + + Config config = ConfigBuilder.of(configFile, Arrays.asList("jobName$=quoted_key_job")); + + Assertions.assertEquals( + "prefix-quoted_key_job", config.getConfig("env").getString("job.name")); + } + @Test public void testConfigDesensitizationSort() { Map<String, Object> config = new LinkedHashMap<>(); @@ -236,6 +370,72 @@ public class ConfigBuilderTest { Assertions.assertEquals("string", desensitizedFields.get("user-password")); } + private static void assertWhitespacePaddedKeys(Config config) { + Map<?, ?> fields = + config.getConfigList("source") + .get(0) + .getConfig("schema") + .getConfig("fields") + .root() + .unwrapped(); + + Assertions.assertEquals(4, fields.size()); + Assertions.assertEquals("trimmed", fields.get("padded")); + Assertions.assertFalse(fields.containsKey(" padded ")); + Assertions.assertEquals("dotted", fields.get("job.mode")); + Assertions.assertEquals("regex", fields.get(" ^t_nova_.*$ ")); + Assertions.assertFalse(fields.containsKey("^t_nova_.*$")); + Assertions.assertEquals("literal", fields.get(" ${FOO} ")); + Assertions.assertFalse(fields.containsKey("${FOO}")); + } + + private static void assertLiteralSchemaFields(Config config) { + Assertions.assertEquals( + literalSchemaFields(), + config.getConfigList("source") + .get(0) + .getConfig("schema") + .getConfig("fields") + .root() + .unwrapped()); + } + + private static Map<String, Object> literalSchemaFields() { + Map<String, Object> fields = new LinkedHashMap<>(); + fields.put("^t_nova_.*$", "string"); + fields.put("a:b", "long"); + fields.put("${FOO}", "boolean"); + fields.put("a\"b\\c", "int"); + return fields; + } + + /** Quotes literal field keys while leaving the optional value substitution active. */ + private static Path writeLiteralKeyConfigFile(Path tempDir, boolean withJobNamePlaceholder) + throws IOException { + StringBuilder content = new StringBuilder(); + content.append("env {\n"); + content.append(" job.mode = BATCH\n"); + if (withJobNamePlaceholder) { + content.append(" job.name = ${jobName}\n"); + } + content.append("}\n"); + content.append("source {\n FakeSource {\n schema {\n"); + content.append(" fields {\n"); + for (Map.Entry<String, Object> field : literalSchemaFields().entrySet()) { + content.append(literalKeyLine(field.getKey(), (String) field.getValue())); + } + content.append(" }\n"); + content.append(" }\n }\n}\nsink { Console {} }\n"); + + Path configFile = tempDir.resolve("literal_path_keys.conf"); + Files.write(configFile, content.toString().getBytes(StandardCharsets.UTF_8)); + return configFile; + } + + private static String literalKeyLine(String key, String type) { + return " " + ConfigUtil.quoteString(key) + " = " + type + "\n"; + } + @Test void testDuplicateKeyThrowsException() { List<String> variables = Arrays.asList("jobName=value1", "jobName=value2"); diff --git a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/RestApiIT.java b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/RestApiIT.java index 381fe8cabf5..8bf2c3c768d 100644 --- a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/RestApiIT.java +++ b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/RestApiIT.java @@ -1370,6 +1370,8 @@ public class RestApiIT { @Test public void testSubmitJobWithJsonFormat() { + // The nested schema field name and the top-level key cannot be parsed as config paths, + // so both must stay literal instead of failing RestUtil.buildConfig. String jsonConfig = "{\n" + " \"env\": {\n" @@ -1384,7 +1386,8 @@ public class RestApiIT { + " \"schema\": {\n" + " \"fields\": {\n" + " \"name\": \"string\",\n" - + " \"age\": \"int\"\n" + + " \"age\": \"int\",\n" + + " \"^t_nova_.*$\": \"string\"\n" + " }\n" + " }\n" + " }\n" @@ -1394,7 +1397,8 @@ public class RestApiIT { + " \"plugin_name\": \"Console\",\n" + " \"plugin_input\": [\"fake\"]\n" + " }\n" - + " ]\n" + + " ],\n" + + " \"^t_nova_.*$\": \"literal\"\n" + "}"; Arrays.asList(node2, node1) @@ -1457,6 +1461,69 @@ public class RestApiIT { }); } + /** + * HOCON bodies are parsed by the HOCON parser before the submit flow decrypts them. A key such + * as {@code "$systemId"} has to be quoted in the body, because {@code $} is a reserved HOCON + * character, and it has to stay a literal key when the decrypted config is re-parsed so that + * the FieldMapper mapping keeps working. + */ + @Test + public void testSubmitJobWithHoconFormatAndQuotedLiteralKey() { + String hoconConfig = + "env {\n" + + " parallelism = 1\n" + + " job.mode = \"BATCH\"\n" + + "}\n" + + "\n" + + "source {\n" + + " FakeSource {\n" + + " plugin_output = \"fake\"\n" + + " row.num = 2\n" + + " schema = {\n" + + " fields {\n" + + " name = \"string\"\n" + + " age = \"int\"\n" + + " \"$systemId\" = \"string\"\n" + + " }\n" + + " }\n" + + " }\n" + + "}\n" + + "\n" + + "transform {\n" + + " FieldMapper {\n" + + " plugin_input = \"fake\"\n" + + " plugin_output = \"mapped\"\n" + + " field_mapper = {\n" + + " \"$systemId\" = \"nova_system_id\"\n" + + " }\n" + + " }\n" + + "}\n" + + "\n" + + "sink {\n" + + " Console {\n" + + " plugin_input = \"mapped\"\n" + + " }\n" + + "}"; + + Arrays.asList(node2, node1) + .forEach( + instance -> { + ports.forEach( + (key, value) -> { + given().body(hoconConfig) + .queryParam("format", "hocon") + .queryParam("jobName", "test-hocon-literal-key-job") + .post(HOST + key + CONTEXT_PATH + "/submit-job") + .then() + .statusCode(200) + .body("jobId", notNullValue()) + .body( + "jobName", + equalTo("test-hocon-literal-key-job")); + }); + }); + } + @Test public void testCheckpointOverviewAndHistoryApi() { long jobId = clientJobProxy.getJobId(); diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/rest/RestApiSubmitJobConfigShadeDecryptTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/rest/RestApiSubmitJobConfigShadeDecryptTest.java index 9d6d33eef38..b16b5fb20ba 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/rest/RestApiSubmitJobConfigShadeDecryptTest.java +++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/rest/RestApiSubmitJobConfigShadeDecryptTest.java @@ -17,11 +17,13 @@ package org.apache.seatunnel.engine.server.rest; +import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; import org.apache.seatunnel.shade.org.eclipse.jetty.server.Connector; import org.apache.seatunnel.shade.org.eclipse.jetty.server.ServerConnector; import org.apache.seatunnel.common.utils.ExceptionUtils; import org.apache.seatunnel.common.utils.FileUtils; +import org.apache.seatunnel.core.starter.utils.ConfigShadeUtils; import org.apache.seatunnel.engine.common.config.ConfigProvider; import org.apache.seatunnel.engine.common.config.SeaTunnelConfig; import org.apache.seatunnel.engine.common.config.server.HttpConfig; @@ -49,6 +51,7 @@ import java.net.HttpURLConnection; import java.net.URL; import java.nio.charset.StandardCharsets; import java.nio.file.Paths; +import java.util.Map; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; @@ -130,6 +133,77 @@ public class RestApiSubmitJobConfigShadeDecryptTest { "Response should contain jobId, got: " + response.body); } + /** + * The HOCON branch of {@code JobInfoService.submitJob} parses the body and then runs it through + * {@link ConfigShadeUtils#decryptConfig}, which renders the config to JSON and re-parses the + * resulting map. A quoted key such as {@code "$systemId"} is a literal key in HOCON, so the + * re-parse must not interpret it as a path expression. + */ + @Test + public void testHoconDecryptPreservesQuotedFieldMapperKey() { + String body = buildQuotedLiteralKeyHoconBody(true); + + org.apache.seatunnel.shade.com.typesafe.config.Config decrypted = + ConfigShadeUtils.decryptConfig(ConfigFactory.parseString(body)); + + Map<String, Object> fields = + decrypted + .getConfigList("source") + .get(0) + .getConfig("schema") + .getConfig("fields") + .root() + .unwrapped(); + Assertions.assertEquals("string", fields.get("$systemId")); + + Map<String, Object> fieldMapper = + decrypted + .getConfigList("transform") + .get(0) + .getConfig("field_mapper") + .root() + .unwrapped(); + Assertions.assertEquals("nova_system_id", fieldMapper.get("$systemId")); + } + + /** + * Regression for the REST {@code format=hocon} flow: the quoted {@code "$systemId"} key must + * survive the decrypt step of the submit path. The repository's engine-server test classpath + * only ships FakeSource/Console, so the literal key is exercised as a schema field here and as + * a FieldMapper key in {@link #testHoconDecryptPreservesQuotedFieldMapperKey()}. + */ + @Test + public void testSubmitJobWithHoconFormatPreservesQuotedLiteralKey() throws Exception { + String requestUrl = + "http://localhost:" + + restPort + + "/submit-job?format=hocon&jobName=hocon_literal_key_test"; + + HttpResponse response = + post(requestUrl, "text/plain", buildQuotedLiteralKeyHoconBody(false)); + Assertions.assertEquals(200, response.code, () -> "responseBody=" + response.body); + Assertions.assertTrue( + response.body.contains("jobId"), + "Response should contain jobId, got: " + response.body); + } + + @Test + public void testSubmitJobWithRegexFieldInJson() throws Exception { + String body = + "{\"env\":{\"job.mode\":\"BATCH\"}," + + "\"source\":[{\"plugin_name\":\"FakeSource\",\"row.num\":2," + + "\"schema\":{\"fields\":{\"^t_nova_.*$\":\"string\"}}}]," + + "\"sink\":[{\"plugin_name\":\"Console\"}]}"; + + HttpResponse response = + post( + "http://localhost:" + restPort + "/submit-job?jobName=regex_json_test", + "application/json", + body); + Assertions.assertEquals(200, response.code, () -> "responseBody=" + response.body); + Assertions.assertTrue(response.body.contains("jobId")); + } + @Test public void testSubmitJobWithHoconFormatMissingShadeIdentifier() throws Exception { String bodyWithoutShade = @@ -314,6 +388,33 @@ public class RestApiSubmitJobConfigShadeDecryptTest { } } + /** + * HOCON body with a quoted {@code "$systemId"} key. The key must be quoted because {@code $} is + * a reserved HOCON character, and because the key is not a SeaTunnel path expression it has to + * stay literal while the config is re-parsed after decryption. + */ + private String buildQuotedLiteralKeyHoconBody(boolean withFieldMapper) { + String body = + "env {\n" + + " job.mode = \"BATCH\"\n" + + "}\n" + + "source {\n" + + " FakeSource {\n" + + " row.num = 2\n" + + " schema { fields { name = \"string\", \"$systemId\" = \"string\" } }\n" + + " }\n" + + "}\n"; + if (withFieldMapper) { + body += + "transform {\n" + + " FieldMapper {\n" + + " field_mapper { \"$systemId\" = \"nova_system_id\" }\n" + + " }\n" + + "}\n"; + } + return body + "sink { Console {} }"; + } + private String buildHoconBody() { return "env {\n" + " job.mode = \"BATCH\"\n" diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/utils/RestUtilTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/utils/RestUtilTest.java new file mode 100644 index 00000000000..2a9220a5801 --- /dev/null +++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/utils/RestUtilTest.java @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.engine.server.utils; + +import org.apache.seatunnel.shade.com.typesafe.config.Config; + +import org.apache.seatunnel.core.starter.utils.ConfigShadeUtils; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import scala.Tuple2; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; + +class RestUtilTest { + + private static final String REGEX_FIELD = "^t_nova_.*$"; + + @Test + void buildConfigShouldPreserveRegexKeysFromJson() throws IOException { + Config config = RestUtil.buildConfig(jsonNode(jobConfigJson())); + + assertJobConfig(config); + assertJobConfig(ConfigShadeUtils.decryptConfig(config)); + } + + @Test + void buildConfigShouldPreserveDottedKeyAsLiteral() throws IOException { + Config config = RestUtil.buildConfig(jsonNode("{\"a.b\":\"value\"}")); + + Assertions.assertEquals("value", config.getString("a.b")); + Assertions.assertEquals("value", config.root().unwrapped().get("a.b")); + } + + @Test + void buildConfigShouldKeepValidPathExpressions() throws IOException { + Config config = RestUtil.buildConfig(jsonNode("{\"a->b\":\"value\"}")); + + Assertions.assertEquals("value", config.getConfig("a").getString("b")); + Assertions.assertFalse(config.root().unwrapped().containsKey("a->b")); + } + + @Test + void buildConfigShouldPreserveTopLevelInvalidPathKey() throws IOException { + String json = + jobConfigJson().substring(0, jobConfigJson().length() - 1) + + ",\"^t_nova_.*$\":\"literal\"}"; + + Config config = ConfigShadeUtils.decryptConfig(RestUtil.buildConfig(jsonNode(json))); + + Assertions.assertEquals("literal", config.root().unwrapped().get(REGEX_FIELD)); + assertJobConfig(config); + } + + @Test + void buildConfigShouldPreserveEscapedAndReservedKeysThroughConfigShade() throws IOException { + String json = + "{\"source\":[{\"schema\":{\"fields\":{" + + "\"$\":\"dollar\",\":\":\"colon\",\"${FOO}\":\"substitution\"," + + "\"a\\\"b\\\\c\":\"escaped\"}}}],\"sink\":[{}]}"; + + Config config = + ConfigShadeUtils.decryptConfig( + ConfigShadeUtils.encryptConfig(RestUtil.buildConfig(jsonNode(json)))); + Map<String, Object> fields = + config.getConfigList("source") + .get(0) + .getConfig("schema") + .getConfig("fields") + .root() + .unwrapped(); + + Assertions.assertEquals("dollar", fields.get("$")); + Assertions.assertEquals("colon", fields.get(":")); + Assertions.assertEquals("substitution", fields.get("${FOO}")); + Assertions.assertEquals("escaped", fields.get("a\"b\\c")); + } + + @Test + void buildConfigListShouldPreserveRegexKeysFromJson() throws IOException { + List<Tuple2<Map<String, String>, Config>> configs = + RestUtil.buildConfigList( + jsonNode("[{\"params\":{}," + jobConfigJson().substring(1) + "]")); + + Assertions.assertEquals(1, configs.size()); + Assertions.assertTrue(configs.get(0)._1.isEmpty()); + assertJobConfig(configs.get(0)._2); + assertJobConfig(ConfigShadeUtils.decryptConfig(configs.get(0)._2)); + } + + @Test + void buildConfigShouldPreserveValuesRequiredForConfigShadeDecryption() throws IOException { + Config decryptedConfig = + ConfigShadeUtils.decryptConfig(RestUtil.buildConfig(jsonNode(configShadeJson()))); + + Config source = decryptedConfig.getConfigList("source").get(0); + Assertions.assertEquals("seatunnel", source.getString("username")); + Assertions.assertEquals("seatunnel_password", source.getString("password")); + } + + private void assertJobConfig(Config config) { + Assertions.assertEquals( + "BATCH", config.getConfig("env").root().unwrapped().get("job.mode")); + Config fields = + config.getConfigList("source").get(0).getConfig("schema").getConfig("fields"); + Assertions.assertEquals("string", fields.root().unwrapped().get(REGEX_FIELD)); + } + + private String jobConfigJson() { + return "{" + + "\"env\":{\"job.mode\":\"BATCH\"}," + + "\"source\":[{\"schema\":{\"fields\":{\"" + + REGEX_FIELD + + "\":\"string\"}}}]," + + "\"transform\":[]," + + "\"sink\":[{}]" + + "}"; + } + + private String configShadeJson() { + return "{" + + "\"env\":{\"shade.identifier\":\"base64\"}," + + "\"source\":[{\"username\":\"c2VhdHVubmVs\"," + + "\"password\":\"c2VhdHVubmVsX3Bhc3N3b3Jk\"}]," + + "\"transform\":[]," + + "\"sink\":[{}]" + + "}"; + } + + private org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode jsonNode(String json) + throws IOException { + return RestUtil.convertByteToJsonNode(json.getBytes(StandardCharsets.UTF_8)); + } +}
