Re: [PR] [INLONG-10362][Manager] Simplify code for InLong group management [inlong]
dockerzhang merged PR #10364: URL: https://github.com/apache/inlong/pull/10364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated (cdab4cd5e8 -> 497f0966bd)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from cdab4cd5e8 [INLONG-10353][Manager] Refactor code for building and submitting flink job (#10354) add 497f0966bd [INLONG-10362][Manager] Simplify code for InLong group management (#10364) No new revisions were added by this update. Summary of changes: .../manager/service/group/InlongGroupService.java | 20 - .../service/group/InlongGroupServiceImpl.java | 51 -- .../openapi/OpenInLongGroupController.java | 4 +- 3 files changed, 2 insertions(+), 73 deletions(-)
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
aloyszhang commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629107348
##
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/TemplateVisibleRange.java:
##
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.common.enums;
+
+/**
+ * TemplateVisibleRange
+ */
+public enum TemplateVisibleRange {
+
+ALL,
+IN_CHARGE,
+TENANT,
Review Comment:
```suggestion
TENANT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [INLONG-10368][Manager] Data preview supports obtaining message attribute information [inlong]
fuweng11 opened a new pull request, #10369: URL: https://github.com/apache/inlong/pull/10369 Fixes #10368 ### Motivation Data preview supports obtaining message attribute information. ### Modifications Data preview supports obtaining message attribute information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
aloyszhang commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629125685
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/TemplateField.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.stream;
+
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelEntity;
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelField;
+import org.apache.inlong.manager.common.tool.excel.annotation.Font;
+import org.apache.inlong.manager.common.tool.excel.annotation.Style;
+import
org.apache.inlong.manager.common.tool.excel.validator.NonEmptyCellValidator;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.poi.ss.usermodel.BorderStyle;
+import org.apache.poi.ss.usermodel.IndexedColors;
+
+import java.io.Serializable;
+
+/**
+ * Template filed, including field name, field type, etc.
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Template field configuration")
+@ExcelEntity(name = "InLong-Template-field")
+public class TemplateField implements Serializable {
+
+@ApiModelProperty("Field index")
+private Integer id;
+
+@ExcelField(name = "Field name", validator = NonEmptyCellValidator.class,
font = @Font(size = 16), style = @Style(bgColor =
IndexedColors.LIGHT_TURQUOISE, width = 9000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field name", required = true)
+private String fieldName;
+
+@ExcelField(name = "Field type", validator =
StreamFieldTypeCellValidator.class, font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 6000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field type", required = true)
+private String fieldType;
+
+@ExcelField(name = "Field comment", font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 1, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty("Field comment")
+private String fieldComment;
+
+@ApiModelProperty(value = "Is predefined field, 1: yes, 0: no")
+private Integer isPredefinedField;
+
+@ApiModelProperty(value = "Field value for constants")
+private String fieldValue;
+
+@ApiModelProperty(value = "Value expression of predefined field")
+private String preExpression;
+
+@ApiModelProperty("Is this field a meta field, 0: no, 1: yes")
Review Comment:
```suggestion
@ApiModelProperty("Is this field a meta field, 0: no, 1: yes, default is
0")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
aloyszhang commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629127394
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/TemplateField.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.stream;
+
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelEntity;
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelField;
+import org.apache.inlong.manager.common.tool.excel.annotation.Font;
+import org.apache.inlong.manager.common.tool.excel.annotation.Style;
+import
org.apache.inlong.manager.common.tool.excel.validator.NonEmptyCellValidator;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.poi.ss.usermodel.BorderStyle;
+import org.apache.poi.ss.usermodel.IndexedColors;
+
+import java.io.Serializable;
+
+/**
+ * Template filed, including field name, field type, etc.
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Template field configuration")
+@ExcelEntity(name = "InLong-Template-field")
+public class TemplateField implements Serializable {
+
+@ApiModelProperty("Field index")
+private Integer id;
+
+@ExcelField(name = "Field name", validator = NonEmptyCellValidator.class,
font = @Font(size = 16), style = @Style(bgColor =
IndexedColors.LIGHT_TURQUOISE, width = 9000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field name", required = true)
+private String fieldName;
+
+@ExcelField(name = "Field type", validator =
StreamFieldTypeCellValidator.class, font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 6000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field type", required = true)
+private String fieldType;
+
+@ExcelField(name = "Field comment", font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 1, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty("Field comment")
+private String fieldComment;
+
+@ApiModelProperty(value = "Is predefined field, 1: yes, 0: no")
+private Integer isPredefinedField;
+
+@ApiModelProperty(value = "Field value for constants")
+private String fieldValue;
+
+@ApiModelProperty(value = "Value expression of predefined field")
+private String preExpression;
+
+@ApiModelProperty("Is this field a meta field, 0: no, 1: yes")
+@Builder.Default
+private Integer isMetaField = 0;
+
+@ApiModelProperty(value = "Meta field name")
+private String metaFieldName;
+
+@ApiModelProperty("Field format, including: MICROSECONDS, MILLISECONDS,
SECONDS, SQL, ISO_8601"
++ " and custom such as '-MM-dd HH:mm:ss'. This is mainly used
for time format")
+private String fieldFormat;
+
+@ApiModelProperty("Origin node name which stream fields belong")
+private String originNodeName;
+
+@ApiModelProperty("Origin field name before transform operation")
+private String originFieldName;
+
+@ApiModelProperty("Extra Param in JSON style")
+private String extParams;
+
+public TemplateField(int index, String fieldType, String fieldName, String
fieldComment, String fieldValue) {
+this.id = index;
+t
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
aloyszhang commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629130917
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/TemplateField.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.stream;
+
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelEntity;
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelField;
+import org.apache.inlong.manager.common.tool.excel.annotation.Font;
+import org.apache.inlong.manager.common.tool.excel.annotation.Style;
+import
org.apache.inlong.manager.common.tool.excel.validator.NonEmptyCellValidator;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.poi.ss.usermodel.BorderStyle;
+import org.apache.poi.ss.usermodel.IndexedColors;
+
+import java.io.Serializable;
+
+/**
+ * Template filed, including field name, field type, etc.
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Template field configuration")
+@ExcelEntity(name = "InLong-Template-field")
+public class TemplateField implements Serializable {
+
+@ApiModelProperty("Field index")
+private Integer id;
+
+@ExcelField(name = "Field name", validator = NonEmptyCellValidator.class,
font = @Font(size = 16), style = @Style(bgColor =
IndexedColors.LIGHT_TURQUOISE, width = 9000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field name", required = true)
+private String fieldName;
+
+@ExcelField(name = "Field type", validator =
StreamFieldTypeCellValidator.class, font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 6000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field type", required = true)
+private String fieldType;
+
+@ExcelField(name = "Field comment", font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 1, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty("Field comment")
+private String fieldComment;
+
+@ApiModelProperty(value = "Is predefined field, 1: yes, 0: no")
+private Integer isPredefinedField;
+
+@ApiModelProperty(value = "Field value for constants")
+private String fieldValue;
+
+@ApiModelProperty(value = "Value expression of predefined field")
+private String preExpression;
+
+@ApiModelProperty("Is this field a meta field, 0: no, 1: yes")
+@Builder.Default
+private Integer isMetaField = 0;
+
+@ApiModelProperty(value = "Meta field name")
+private String metaFieldName;
+
+@ApiModelProperty("Field format, including: MICROSECONDS, MILLISECONDS,
SECONDS, SQL, ISO_8601"
++ " and custom such as '-MM-dd HH:mm:ss'. This is mainly used
for time format")
+private String fieldFormat;
+
+@ApiModelProperty("Origin node name which stream fields belong")
+private String originNodeName;
+
+@ApiModelProperty("Origin field name before transform operation")
+private String originFieldName;
+
+@ApiModelProperty("Extra Param in JSON style")
+private String extParams;
+
+public TemplateField(int index, String fieldType, String fieldName, String
fieldComment, String fieldValue) {
+this.id = index;
+t
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
vernedeng commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629234486
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java:
##
@@ -385,16 +399,14 @@ private void summaryExpiredStatMap(long isolateKey) {
* Summary pre stat map
*/
private void summaryPreStatMap(long isolateKey, ConcurrentHashMap statInfo) {
-List expiredKeys =
this.expiredKeyList.computeIfAbsent(isolateKey, k -> new ArrayList<>());
+HashSet expiredKeys =
this.expiredKeyList.computeIfAbsent(isolateKey, k -> new HashSet<>());
Review Comment:
Use the interface **Set** instead of the specific implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
fuweng11 commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629258369
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/TemplateField.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.stream;
+
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelEntity;
+import org.apache.inlong.manager.common.tool.excel.annotation.ExcelField;
+import org.apache.inlong.manager.common.tool.excel.annotation.Font;
+import org.apache.inlong.manager.common.tool.excel.annotation.Style;
+import
org.apache.inlong.manager.common.tool.excel.validator.NonEmptyCellValidator;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.poi.ss.usermodel.BorderStyle;
+import org.apache.poi.ss.usermodel.IndexedColors;
+
+import java.io.Serializable;
+
+/**
+ * Template filed, including field name, field type, etc.
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+@ApiModel("Template field configuration")
+@ExcelEntity(name = "InLong-Template-field")
+public class TemplateField implements Serializable {
+
+@ApiModelProperty("Field index")
+private Integer id;
+
+@ExcelField(name = "Field name", validator = NonEmptyCellValidator.class,
font = @Font(size = 16), style = @Style(bgColor =
IndexedColors.LIGHT_TURQUOISE, width = 9000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field name", required = true)
+private String fieldName;
+
+@ExcelField(name = "Field type", validator =
StreamFieldTypeCellValidator.class, font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 6000, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty(value = "Field type", required = true)
+private String fieldType;
+
+@ExcelField(name = "Field comment", font = @Font(size = 16), style =
@Style(bgColor = IndexedColors.LIGHT_TURQUOISE, width = 1, allBorderColor =
IndexedColors.BLUE, allBorderStyle = BorderStyle.THIN), headerFont = @Font(size
= 20, color = IndexedColors.WHITE), headerStyle = @Style(bgColor =
IndexedColors.DARK_BLUE, width = 9000, allBorderColor = IndexedColors.BLUE,
allBorderStyle = BorderStyle.THIN))
+@ApiModelProperty("Field comment")
+private String fieldComment;
+
+@ApiModelProperty(value = "Is predefined field, 1: yes, 0: no")
+private Integer isPredefinedField;
+
+@ApiModelProperty(value = "Field value for constants")
+private String fieldValue;
+
+@ApiModelProperty(value = "Value expression of predefined field")
+private String preExpression;
+
+@ApiModelProperty("Is this field a meta field, 0: no, 1: yes")
+@Builder.Default
+private Integer isMetaField = 0;
+
+@ApiModelProperty(value = "Meta field name")
+private String metaFieldName;
+
+@ApiModelProperty("Field format, including: MICROSECONDS, MILLISECONDS,
SECONDS, SQL, ISO_8601"
++ " and custom such as '-MM-dd HH:mm:ss'. This is mainly used
for time format")
+private String fieldFormat;
+
+@ApiModelProperty("Origin node name which stream fields belong")
+private String originNodeName;
+
+@ApiModelProperty("Origin field name before transform operation")
+private String originFieldName;
+
+@ApiModelProperty("Extra Param in JSON style")
+private String extParams;
+
+public TemplateField(int index, String fieldType, String fieldName, String
fieldComment, String fieldValue) {
+this.id = index;
+thi
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
fuweng11 commented on code in PR #10333:
URL: https://github.com/apache/inlong/pull/10333#discussion_r1629258935
##
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/TemplateVisibleRange.java:
##
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.common.enums;
+
+/**
+ * TemplateVisibleRange
+ */
+public enum TemplateVisibleRange {
+
+ALL,
+IN_CHARGE,
+TENANT,
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10368][Manager] Data preview supports obtaining message attribute information [inlong]
dockerzhang merged PR #10369: URL: https://github.com/apache/inlong/pull/10369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10368][Manager] Data preview supports obtaining message attribute information (#10369)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new ce49ad9bc8 [INLONG-10368][Manager] Data preview supports obtaining
message attribute information (#10369)
ce49ad9bc8 is described below
commit ce49ad9bc8b5ea08a51f251a0ad8621c67d7424a
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Thu Jun 6 19:02:59 2024 +0800
[INLONG-10368][Manager] Data preview supports obtaining message attribute
information (#10369)
---
.../java/org/apache/inlong/manager/pojo/consume/BriefMQMessage.java| 3 +++
.../inlong/manager/service/message/InlongMsgDeserializeOperator.java | 1 +
2 files changed, 4 insertions(+)
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/consume/BriefMQMessage.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/consume/BriefMQMessage.java
index d471c23865..6161684b43 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/consume/BriefMQMessage.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/consume/BriefMQMessage.java
@@ -52,6 +52,9 @@ public class BriefMQMessage {
@ApiModelProperty(value = "Client ip")
private String clientIp;
+@ApiModelProperty(value = "Message attribute")
+private String attribute;
+
@ApiModelProperty(value = "Message header")
private Map headers;
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java
index d805ead216..b04931db0f 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/message/InlongMsgDeserializeOperator.java
@@ -94,6 +94,7 @@ public class InlongMsgDeserializeOperator implements
DeserializeOperator {
.dt(msgTime)
.clientIp(attrMap.get(CLIENT_IP))
.headers(headers)
+.attribute(attr)
.body(body)
.fieldList(streamFieldList)
.build();
Re: [PR] [INLONG-10323][Sort] Support Kv deserialization format in sort module [inlong]
dockerzhang merged PR #10349: URL: https://github.com/apache/inlong/pull/10349 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10323][Sort] Support KV deserialization format in sort module (#10349)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c244ec7561 [INLONG-10323][Sort] Support KV deserialization format in
sort module (#10349)
c244ec7561 is described below
commit c244ec75616591ab956e4610ecbc0cf73f6ee394
Author: ZiruiPeng
AuthorDate: Thu Jun 6 19:03:39 2024 +0800
[INLONG-10323][Sort] Support KV deserialization format in sort module
(#10349)
---
.../inlong/sort/protocol/node/format/Format.java | 3 +-
.../inlong/sort/protocol/node/format/KvFormat.java | 131 +
.../sort/protocol/node/format/KvFormatTest.java| 28 +
inlong-sort/sort-dist/pom.xml | 2 +-
.../inlong/sort/formats/base/TableFormatUtils.java | 16 +++
.../inlong/sort/formats/kv/KvFormatFactory.java| 6 +-
6 files changed, 181 insertions(+), 5 deletions(-)
diff --git
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/Format.java
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/Format.java
index 82196600c1..69182d1df2 100644
---
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/Format.java
+++
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/Format.java
@@ -35,7 +35,8 @@ import java.util.Map;
@JsonSubTypes.Type(value = CanalJsonFormat.class, name =
"canalJsonFormat"),
@JsonSubTypes.Type(value = CsvFormat.class, name = "csvFormat"),
@JsonSubTypes.Type(value = InLongMsgFormat.class, name =
"inLongMsgFormat"),
-@JsonSubTypes.Type(value = RawFormat.class, name = "rawFormat")
+@JsonSubTypes.Type(value = RawFormat.class, name = "rawFormat"),
+@JsonSubTypes.Type(value = KvFormat.class, name = "kvFormat")
})
public interface Format extends Serializable {
diff --git
a/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/KvFormat.java
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/KvFormat.java
new file mode 100644
index 00..742d2423e4
--- /dev/null
+++
b/inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/format/KvFormat.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.protocol.node.format;
+
+import lombok.Data;
+import org.apache.commons.lang3.StringUtils;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonIgnore;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonTypeName;
+
+import javax.annotation.Nullable;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_CHARSET;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_ESCAPE_CHARACTER;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_IGNORE_ERRORS;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_KV_DELIMITER;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_KV_ENTRY_DELIMITER;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_NULL_LITERAL;
+import static
org.apache.inlong.sort.formats.base.TableFormatConstants.FORMAT_QUOTE_CHARACTER;
+
+@JsonTypeName("kvFormat")
+@Data
+public class KvFormat implements Format {
+
+private static final String IDENTIFIER = "inlong-kv";
+
+@JsonProperty(value = "entryDelimiter", defaultValue = "&")
+private final String entryDelimiter;
+
+@JsonProperty(value = "kvDelimiter", defaultValue = "=")
+private final String kvDelimiter;
+
+@JsonProperty(value = "ignoreParseErrors", defaultValue = "false")
+@Nullable
+private final Boolean ignoreParseErrors;
+
+@JsonProperty(value = "escapeChar")
+
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
aloyszhang commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629333246
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/util/RequestIdUtils.java:
##
@@ -17,33 +17,22 @@
package org.apache.inlong.audit.util;
-import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicLong;
-public class SenderResult {
+public class RequestIdUtils {
-public final InetSocketAddress addr;
-public boolean result;
+private static final Long MAX_REQUEST_ID = 10L;
+private static final AtomicLong requestIdSeq = new AtomicLong(0L);
/**
- * Constructor
- *
- * @param addr
- * @param result
+ * Next request id
*/
-public SenderResult(InetSocketAddress addr, boolean result) {
-this.addr = addr;
-this.result = result;
-}
-
-/**
- * Constructor
- *
- * @param sendIp
- * @param sendPort
- * @param result
- */
-public SenderResult(String sendIp, int sendPort, boolean result) {
-this.addr = new InetSocketAddress(sendIp, sendPort);
-this.result = result;
+public static Long nextRequestId() {
Review Comment:
Race conditions may happen here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
aloyszhang commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629333246
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/util/RequestIdUtils.java:
##
@@ -17,33 +17,22 @@
package org.apache.inlong.audit.util;
-import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicLong;
-public class SenderResult {
+public class RequestIdUtils {
-public final InetSocketAddress addr;
-public boolean result;
+private static final Long MAX_REQUEST_ID = 10L;
+private static final AtomicLong requestIdSeq = new AtomicLong(0L);
/**
- * Constructor
- *
- * @param addr
- * @param result
+ * Next request id
*/
-public SenderResult(InetSocketAddress addr, boolean result) {
-this.addr = addr;
-this.result = result;
-}
-
-/**
- * Constructor
- *
- * @param sendIp
- * @param sendPort
- * @param result
- */
-public SenderResult(String sendIp, int sendPort, boolean result) {
-this.addr = new InetSocketAddress(sendIp, sendPort);
-this.result = result;
+public static Long nextRequestId() {
Review Comment:
Race conditions may happen here and you will get two same requestId.
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/util/RequestIdUtils.java:
##
@@ -17,33 +17,22 @@
package org.apache.inlong.audit.util;
-import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicLong;
-public class SenderResult {
+public class RequestIdUtils {
-public final InetSocketAddress addr;
-public boolean result;
+private static final Long MAX_REQUEST_ID = 10L;
+private static final AtomicLong requestIdSeq = new AtomicLong(0L);
/**
- * Constructor
- *
- * @param addr
- * @param result
+ * Next request id
*/
-public SenderResult(InetSocketAddress addr, boolean result) {
-this.addr = addr;
-this.result = result;
-}
-
-/**
- * Constructor
- *
- * @param sendIp
- * @param sendPort
- * @param result
- */
-public SenderResult(String sendIp, int sendPort, boolean result) {
-this.addr = new InetSocketAddress(sendIp, sendPort);
-this.result = result;
+public static Long nextRequestId() {
Review Comment:
Race conditions may happen here and you will get two same r`equestId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
aloyszhang commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629333246
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/util/RequestIdUtils.java:
##
@@ -17,33 +17,22 @@
package org.apache.inlong.audit.util;
-import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicLong;
-public class SenderResult {
+public class RequestIdUtils {
-public final InetSocketAddress addr;
-public boolean result;
+private static final Long MAX_REQUEST_ID = 10L;
+private static final AtomicLong requestIdSeq = new AtomicLong(0L);
/**
- * Constructor
- *
- * @param addr
- * @param result
+ * Next request id
*/
-public SenderResult(InetSocketAddress addr, boolean result) {
-this.addr = addr;
-this.result = result;
-}
-
-/**
- * Constructor
- *
- * @param sendIp
- * @param sendPort
- * @param result
- */
-public SenderResult(String sendIp, int sendPort, boolean result) {
-this.addr = new InetSocketAddress(sendIp, sendPort);
-this.result = result;
+public static Long nextRequestId() {
Review Comment:
Race conditions may happen here and you will get two same `requestId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(inlong) 03/05: [INLONG-9822][Manager] Support flink job runtime execution mode configuration (#9823)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch dev-offline-sync
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit ed21473a97181420df50719c64d96200b0bec136
Author: AloysZhang
AuthorDate: Fri Mar 15 15:12:53 2024 +0800
[INLONG-9822][Manager] Support flink job runtime execution mode
configuration (#9823)
---
.../apache/inlong/manager/common/consts/InlongConstants.java | 3 +++
.../org/apache/inlong/manager/plugin/flink/FlinkService.java | 2 ++
.../org/apache/inlong/manager/plugin/flink/dto/FlinkInfo.java | 2 ++
.../inlong/manager/plugin/listener/DeleteSortListener.java| 1 +
.../inlong/manager/plugin/listener/RestartSortListener.java | 8 +++-
.../java/org/apache/inlong/sort/configuration/Constants.java | 8
.../src/main/java/org/apache/inlong/sort/Entrance.java| 11 ++-
7 files changed, 33 insertions(+), 2 deletions(-)
diff --git
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
index 581ebb3098..e84dcc6602 100644
---
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
@@ -90,6 +90,9 @@ public class InlongConstants {
public static final Integer DATASYNC_REALTIME_MODE = 1;
public static final Integer DATASYNC_OFFLINE_MODE = 2;
+public static final String RUNTIME_EXECUTION_MODE_STREAMING = "streaming";
+public static final String RUNTIME_EXECUTION_MODE_BATCH = "batch";
+
public static final Integer DISABLE_ZK = 0;
public static final Integer ENABLE_ZK = 1;
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/FlinkService.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/FlinkService.java
index 129b7aa444..9188c6b73f 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/FlinkService.java
+++
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/FlinkService.java
@@ -258,6 +258,8 @@ public class FlinkService {
list.add(flinkInfo.getLocalConfPath());
list.add("-checkpoint.interval");
list.add("6");
+list.add("-runtime.execution.mode");
+list.add(flinkInfo.getRuntimeExecutionMode());
return list.toArray(new String[0]);
}
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/dto/FlinkInfo.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/dto/FlinkInfo.java
index f7071ceb4b..4c3c75f855 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/dto/FlinkInfo.java
+++
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/flink/dto/FlinkInfo.java
@@ -52,4 +52,6 @@ public class FlinkInfo {
private boolean isException = false;
private String exceptionMsg;
+
+private String runtimeExecutionMode;
}
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/DeleteSortListener.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/DeleteSortListener.java
index 009374e2a6..5e30ad8cb5 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/DeleteSortListener.java
+++
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/DeleteSortListener.java
@@ -119,6 +119,7 @@ public class DeleteSortListener implements
SortOperateListener {
FlinkOperation flinkOperation = FlinkOperation.getInstance();
try {
flinkOperation.delete(flinkInfo);
+// TODO if the job is OFFLINE, should delete the scheduler
information
log.info("job delete success for jobId={}", jobId);
} catch (Exception e) {
flinkInfo.setException(true);
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/RestartSortListener.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/RestartSortListener.java
index 0c6828c984..242138c1e4 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/RestartSortListener.java
+++
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/RestartSortListener.java
@@ -86,7 +86,13 @@ public class RestartSortListener implements
SortOperateListener {
(inlong) 05/05: [INLONG-10247][Manager] Support schedule information management for offline sync (#10254)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch dev-offline-sync
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit b68bf5c5e597bb6870e6b9addb37aa7c3b1d9aab
Author: AloysZhang
AuthorDate: Thu May 23 18:18:31 2024 +0800
[INLONG-10247][Manager] Support schedule information management for
offline sync (#10254)
---
.../client/api/inner/client/ClientFactory.java | 2 +
.../api/inner/client/InLongScheduleClient.java | 67 +
.../client/api/service/InLongScheduleApi.java | 49 +++
.../inlong/manager/common/enums/ErrorCodeEnum.java | 3 +
.../manager/common/enums/OperationTarget.java | 4 +-
.../{OperationTarget.java => ScheduleStatus.java} | 52 ---
.../inlong/manager/dao/entity/ScheduleEntity.java | 61
.../manager/dao/mapper/ScheduleEntityMapper.java} | 34 ++---
.../resources/mappers/ScheduleEntityMapper.xml | 144 +++
.../manager/dao/mapper/ScheduleEntityTest.java | 118
.../inlong/manager/pojo/schedule/ScheduleInfo.java | 82 +++
.../manager/pojo/schedule/ScheduleInfoRequest.java | 76 ++
.../manager/service/schedule/ScheduleService.java | 71 ++
.../service/schedule/ScheduleServiceImpl.java | 155 +
.../main/resources/h2/apache_inlong_manager.sql| 31 +
.../manager-web/sql/apache_inlong_manager.sql | 30
inlong-manager/manager-web/sql/changes-1.13.0.sql | 30
.../web/controller/InLongSchedulerController.java | 88
18 files changed, 1045 insertions(+), 52 deletions(-)
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/ClientFactory.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/ClientFactory.java
index 3c67f1fa71..01c9fb6473 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/ClientFactory.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/ClientFactory.java
@@ -55,6 +55,7 @@ public class ClientFactory {
private final AuditClient auditClient;
private final InlongTenantClient inlongTenantClient;
private final InlongTenantRoleClient inlongTenantRoleClient;
+private final InLongScheduleClient inLongScheduleClient;
public ClientFactory(ClientConfiguration configuration) {
groupClient = new InlongGroupClient(configuration);
@@ -74,5 +75,6 @@ public class ClientFactory {
auditClient = new AuditClient(configuration);
inlongTenantClient = new InlongTenantClient(configuration);
inlongTenantRoleClient = new InlongTenantRoleClient(configuration);
+inLongScheduleClient = new InLongScheduleClient(configuration);
}
}
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InLongScheduleClient.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InLongScheduleClient.java
new file mode 100644
index 00..86638dbae8
--- /dev/null
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InLongScheduleClient.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.client.api.inner.client;
+
+import org.apache.inlong.manager.client.api.ClientConfiguration;
+import org.apache.inlong.manager.client.api.service.InLongScheduleApi;
+import org.apache.inlong.manager.client.api.util.ClientUtils;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.util.Preconditions;
+import org.apache.inlong.manager.pojo.common.Response;
+import org.apache.inlong.manager.pojo.schedule.ScheduleInfo;
+import org.apache.inlong.manager.pojo.schedule.ScheduleInfoRequest;
+
+public class InLongScheduleClient {
+
+private InLongScheduleApi scheduleApi;
+
+public InLongScheduleClient(ClientConfiguration clientConfiguration) {
+scheduleApi =
ClientUtils.createRetrofit(clientConfi
(inlong) 04/05: [INLONG-9862][Manager] Support submit flink job for offline sync (#9865)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch dev-offline-sync
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit b5279c097b53e8bd1ad35cace748fa0646063bf3
Author: AloysZhang
AuthorDate: Fri Mar 22 18:24:26 2024 +0800
[INLONG-9862][Manager] Support submit flink job for offline sync (#9865)
---
.../manager/common/consts/InlongConstants.java | 4 +
.../manager/pojo/sort/util/StreamParseUtils.java | 21 +
.../listener/StreamTaskListenerFactory.java| 28 +++
.../schedule/StreamScheduleResourceListener.java | 95 ++
.../service/listener/sort/SortConfigListener.java | 8 ++
.../stream/CreateStreamWorkflowDefinition.java | 13 ++-
.../workflow/definition/ServiceTaskType.java | 1 +
.../task/ScheduleOperateListener.java} | 36
.../manager/workflow/plugin/ProcessPlugin.java | 5 ++
9 files changed, 188 insertions(+), 23 deletions(-)
diff --git
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
index e84dcc6602..f0156cd0e9 100644
---
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
@@ -141,6 +141,10 @@ public class InlongConstants {
*/
public static final String DATAFLOW = "dataflow";
+public static final String REGISTER_SCHEDULE_STATUS =
"register.schedule.status";
+
+public static final String REGISTERED = "registered";
+
public static final String STREAMS = "streams";
public static final String RELATIONS = "relations";
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/util/StreamParseUtils.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/util/StreamParseUtils.java
index 92470968ac..203bcc6778 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/util/StreamParseUtils.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/util/StreamParseUtils.java
@@ -17,11 +17,14 @@
package org.apache.inlong.manager.pojo.sort.util;
+import org.apache.inlong.manager.common.consts.InlongConstants;
import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
import org.apache.inlong.manager.common.enums.TransformType;
import org.apache.inlong.manager.common.util.Preconditions;
import org.apache.inlong.manager.pojo.sink.StreamSink;
import org.apache.inlong.manager.pojo.source.StreamSource;
+import org.apache.inlong.manager.pojo.stream.InlongStreamExtInfo;
+import org.apache.inlong.manager.pojo.stream.InlongStreamInfo;
import org.apache.inlong.manager.pojo.stream.StreamNode;
import org.apache.inlong.manager.pojo.stream.StreamPipeline;
import org.apache.inlong.manager.pojo.stream.StreamTransform;
@@ -38,6 +41,8 @@ import
org.apache.inlong.manager.pojo.transform.splitter.SplitterDefinition;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
/**
* Utils of stream parse.
@@ -154,4 +159,20 @@ public class StreamParseUtils {
return GSON.fromJson(tempView, StreamPipeline.class);
}
+public static String getStreamExtProperty(String key, InlongStreamInfo
streamInfo) {
+if (StringUtils.isNotBlank(key) && streamInfo != null &&
CollectionUtils.isNotEmpty(streamInfo.getExtList())) {
+for (InlongStreamExtInfo ext : streamInfo.getExtList()) {
+if (key.equalsIgnoreCase(ext.getKeyName())) {
+return ext.getKeyValue();
+}
+}
+}
+return null;
+}
+
+public static boolean isRegisterScheduleSuccess(InlongStreamInfo
streamInfo) {
+return InlongConstants.REGISTERED
+
.equalsIgnoreCase(getStreamExtProperty(InlongConstants.REGISTER_SCHEDULE_STATUS,
streamInfo));
+}
+
}
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/StreamTaskListenerFactory.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/StreamTaskListenerFactory.java
index 43fdfe4666..87c3a867c2 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/StreamTaskListenerFactory.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/StreamTaskListenerFactory.java
@@ -20,12 +20,14 @@ package org.apache.inlong.manager.service.listener;
import org.apache.inlong.manager.common.plugin.Plugin;
import org.apache.inlong.manager.common.plugin.PluginBinder
(inlong) 01/05: [INLONG-9781][Manager] Add offline sync task type definition (#9787)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch dev-offline-sync
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit a23ad3234b9b3cbb91d10e8965f75664a3a5e308
Author: AloysZhang
AuthorDate: Thu Mar 7 11:44:20 2024 +0800
[INLONG-9781][Manager] Add offline sync task type definition (#9787)
---
.../java/org/apache/inlong/manager/client/ut/BaseTest.java | 2 +-
.../inlong/manager/common/consts/InlongConstants.java | 3 ++-
.../org/apache/inlong/manager/common/enums/GroupMode.java | 14 +++---
.../manager/plugin/listener/StartupSortListener.java | 2 +-
.../apache/inlong/manager/pojo/group/InlongGroupInfo.java | 3 ++-
.../inlong/manager/pojo/group/InlongGroupPageRequest.java | 3 ++-
.../inlong/manager/pojo/group/InlongGroupRequest.java | 5 +++--
.../inlong/manager/service/core/impl/AuditServiceImpl.java | 6 --
.../manager/service/group/InlongGroupServiceImpl.java | 2 +-
.../service/listener/group/InitGroupCompleteListener.java | 2 +-
.../listener/group/UpdateGroupCompleteListener.java| 2 +-
.../service/listener/queue/QueueResourceListener.java | 2 +-
.../listener/stream/InitStreamCompleteListener.java| 2 +-
.../manager/service/source/AbstractSourceOperator.java | 2 +-
.../manager/service/source/StreamSourceServiceImpl.java| 2 +-
15 files changed, 33 insertions(+), 19 deletions(-)
diff --git
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
index 4d218918f5..9961c16ef8 100644
---
a/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
+++
b/inlong-manager/manager-client-examples/src/test/java/org/apache/inlong/manager/client/ut/BaseTest.java
@@ -110,7 +110,7 @@ public class BaseTest {
// set enable zk, create resource, group mode, and cluster tag
pulsarInfo.setEnableZookeeper(InlongConstants.DISABLE_ZK);
pulsarInfo.setEnableCreateResource(InlongConstants.ENABLE_CREATE_RESOURCE);
-pulsarInfo.setInlongGroupMode(InlongConstants.DATASYNC_MODE);
+pulsarInfo.setInlongGroupMode(InlongConstants.DATASYNC_REALTIME_MODE);
pulsarInfo.setInlongClusterTag("default_cluster");
pulsarInfo.setDailyRecords(1000);
diff --git
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
index c3085972fd..581ebb3098 100644
---
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/InlongConstants.java
@@ -87,7 +87,8 @@ public class InlongConstants {
public static final Integer DELETED_STATUS = 10;
public static final Integer STANDARD_MODE = 0;
-public static final Integer DATASYNC_MODE = 1;
+public static final Integer DATASYNC_REALTIME_MODE = 1;
+public static final Integer DATASYNC_OFFLINE_MODE = 2;
public static final Integer DISABLE_ZK = 0;
public static final Integer ENABLE_ZK = 1;
diff --git
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/GroupMode.java
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/GroupMode.java
index f0db2353b6..d4b417f39a 100644
---
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/GroupMode.java
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/GroupMode.java
@@ -31,10 +31,18 @@ public enum GroupMode {
STANDARD("standard"),
/**
- * DataSync mode(only Data Synchronization): group init only with sort in
InLong Cluster
- * StreamSource -> Sort -> StreamSink
+ * DataSync mode(only Data Synchronization): real-time data sync in stream
way, group init only with
+ * sort in InLong Cluster.
+ * StreamSource -> Sort -> Sink
*/
-DATASYNC("datasync");
+DATASYNC("datasync"),
+
+/**
+ * DataSync mode(only Data Synchronization): offline data sync in batch
way, group init only with sort
+ * in InLong Cluster.
+ * BatchSource -> Sort -> Sink
+ */
+DATASYNC_BATCH("datasync_offline");
@Getter
private final String mode;
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
index 038f35543f..9997c2abd2 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListen
(inlong) 02/05: [INLONG-9813][Manager] Support offline data sync management (#9814)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch dev-offline-sync
in repository https://gitbox.apache.org/repos/asf/inlong.git
commit 7b0ba77d0c4545491a045031b96e4924704a4794
Author: AloysZhang
AuthorDate: Wed Mar 13 16:18:04 2024 +0800
[INLONG-9813][Manager] Support offline data sync management (#9814)
---
.../apache/inlong/manager/plugin/listener/StartupSortListener.java | 3 ++-
.../manager/service/listener/group/InitGroupCompleteListener.java | 3 ++-
.../manager/service/listener/group/UpdateGroupCompleteListener.java | 3 ++-
.../manager/service/listener/queue/QueueResourceListener.java | 6 --
.../manager/service/listener/stream/InitStreamCompleteListener.java | 3 ++-
.../inlong/manager/service/source/AbstractSourceOperator.java | 1 +
.../inlong/manager/service/source/StreamSourceServiceImpl.java | 3 ++-
7 files changed, 15 insertions(+), 7 deletions(-)
diff --git
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
index 9997c2abd2..a84d3f5868 100644
---
a/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
+++
b/inlong-manager/manager-plugins/base/src/main/java/org/apache/inlong/manager/plugin/listener/StartupSortListener.java
@@ -61,7 +61,8 @@ public class StartupSortListener implements
SortOperateListener {
}
log.info("add startup group listener for groupId [{}]", groupId);
-return
InlongConstants.DATASYNC_REALTIME_MODE.equals(groupProcessForm.getGroupInfo().getInlongGroupMode());
+return
(InlongConstants.DATASYNC_REALTIME_MODE.equals(groupProcessForm.getGroupInfo().getInlongGroupMode())
+||
InlongConstants.DATASYNC_OFFLINE_MODE.equals(groupProcessForm.getGroupInfo().getInlongGroupMode()));
}
@Override
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
index c96c12c9ff..c8c087320f 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/InitGroupCompleteListener.java
@@ -98,7 +98,8 @@ public class InitGroupCompleteListener implements
ProcessEventListener {
// update status of other related configs
if
(InlongConstants.DISABLE_CREATE_RESOURCE.equals(groupInfo.getEnableCreateResource()))
{
-if
(InlongConstants.DATASYNC_REALTIME_MODE.equals(groupInfo.getInlongGroupMode()))
{
+if
(InlongConstants.DATASYNC_REALTIME_MODE.equals(groupInfo.getInlongGroupMode())
+||
InlongConstants.DATASYNC_OFFLINE_MODE.equals(groupInfo.getInlongGroupMode())) {
sourceService.updateStatus(groupId, null,
SourceStatus.SOURCE_NORMAL.getCode(), operator);
} else {
sourceService.updateStatus(groupId, null,
SourceStatus.TO_BE_ISSUED_ADD.getCode(), operator);
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/UpdateGroupCompleteListener.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/UpdateGroupCompleteListener.java
index a6dd8fc786..1a6e1f7fe1 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/UpdateGroupCompleteListener.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/group/UpdateGroupCompleteListener.java
@@ -96,7 +96,8 @@ public class UpdateGroupCompleteListener implements
ProcessEventListener {
}
// if the inlong group is dataSync mode, the stream source needs to be
processed.
-if
(InlongConstants.DATASYNC_REALTIME_MODE.equals(groupInfo.getInlongGroupMode()))
{
+if
(InlongConstants.DATASYNC_REALTIME_MODE.equals(groupInfo.getInlongGroupMode())
+||
InlongConstants.DATASYNC_OFFLINE_MODE.equals(groupInfo.getInlongGroupMode())) {
changeSource4DataSync(groupId, operateType, operator);
}
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/queue/QueueResourceListener.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/queue/QueueResourceListener.java
index f566c29728..0ce551d1e6 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/listener/queue/QueueResou
(inlong) branch dev-offline-sync updated (a793963543 -> b68bf5c5e5)
This is an automated email from the ASF dual-hosted git repository. aloyszhang pushed a change to branch dev-offline-sync in repository https://gitbox.apache.org/repos/asf/inlong.git discard a793963543 [INLONG-10247][Manager] Support schedule information management for offline sync (#10254) omit 65d3a928ea [INLONG-9862][Manager] Support submit flink job for offline sync (#9865) omit ff3c351401 [INLONG-9822][Manager] Support flink job runtime execution mode configuration (#9823) omit dfb22f4d8e [INLONG-9813][Manager] Support offline data sync management (#9814) omit 1e0fddc0a8 [INLONG-9781][Manager] Add offline sync task type definition (#9787) add 03c2160411 [INLONG-10225][Audit] Audit SDK provides the ability to automatically assign and manage Audit ID (#10232) add 40548ae01b [INLONG-10229][Sort] EsSink support unified configuration (#10237) add 5a331f1c86 [INLONG-10230][Sort] KafkaSink support unified configuration (#10235) add 97e12593e2 [INLONG-10233][Manager] Report heartbeat removal port restriction (#10234) add e4e24def90 [INLONG-10228][Sort] PulsarSink support unified configuration (#10236) add 6645261800 [INLONG-10245][Manager] Support setting audit version for file collection (#10246) add 2acf82d174 [INLONG-10238][Sort] MySQL connector support audit ID (#10239) add 0acdfe9519 [INLONG-10195][Sort] Hudi connector support audit ID (#10231) add 23728055c3 [INLONG-10242][Audit] Audit SDK compatible with InLong Manager to manage audit items (#10248) add 59ea439321 [INLONG-10249][Manager] Fix the problem of duplicate data appears during data preview (#10250) add 85fc8b02e6 [INLONG-9613][TubeMQ] Adjust FATAL type error return content, without carrying class name (#10251) add 6eeec6ee61 [INLONG-10252][Docker] Fix the bug can not get the real audit address (#10253) add d890a1933f [INLONG-10260][Sort] Correct wrong create table statement (#10262) add 2c890c29bb [INLONG-10263][Audit] Solve the conflict between the jdbc.url parameter of Audit Store and the container environment variable (#10264) add d18232b54a [INLONG-10241][Sort] TLog format requires the ability to parse and determine if the first segment is present (#10243) add b5261a3927 [INLONG-10224][Sort] Unified configuration check utils (#10255) add 94a99ac999 [INLONG-10266][Manager] Fix the problem of password is overwritten when adding field information (#10267) add 0113074789 [INLONG-10272][Sort] Unified configuration check utils support check latest config (#10273) add b22a3bdca8 [INLONG-10256][DashBoard] Modify the data source IP item of the file data source form in the data access module (#10258) add ca4bc8dce9 [INLONG-10268][Agent] Get the data version from the auditVersion field (#10269) add ca3fd42542 [INLONG-10270][Manager] Data source tasks allow for multiple IPs (#10271) add 9f829fac3b [INLONG-10274][Audit] The OpenAPI of Audit Service returns the average transmission time (#10275) add 879bf36017 [INLONG-10265][Manager] Fix the problem of correct wrong starrocks create table statement (#10276) add 4f9c4098eb [INLONG-10281][Agent] Real time file collection takes the current time as the data time (#10282) add e0a564b0b2 [INLONG-10277][Manager] Support calling API to refresh cluster config (#10278) add af0cc8255a [INLONG-10277][Manager] Fix the problem of migration failed when migrating multiple groups to the same tenant (#10285) add 863190ea29 [INLONG-10290][Manager] Prohibit groups that have not been successfully configured from obtaining dataproxy addresses (#10293) add c07316253e [INLONG-10292][SDK] Fix panic in connpool.UpdateEndpoints() of Golang SDK (#10295) add d43e1e84ec [INLONG-10291][SDK] Fix incorrect initializing of gnet in Golang SDK (#10294) add 4555ebfe1c [INLONG-10297][Sort] Fix mysql connector cannot submit to flink cluster (#10301) add e5ad544716 [INLONG-10306][Audit] Compatible with scenarios where the Audit Tag is empty (#10308) add 12efb53dae [INLONG-10302][Agent] Add an interface for limiting the number of instances obtained (#10303) add b0a70bda57 [INLONG-10300][Manager] Allow unsubmitted groups to modify mq type (#10304) add 229174675b [INLONG-10298][Agent] Delete useless code (#10299) add 9d94ef72df [INLONG-10305][Manager] Delete k8s related parameters in file collection (#10307) add 83ded2149d [INLONG-10313][DataProxy] Replace audit ID macro with audit API (#10315) add c6892951ed [INLONG-10319][Agent] Get audit Id from audit sdk (#10322) add 8f6ce51cd7 [INLONG-10314][DashBoard] Add an operation time to the operation log table (#10316) add ca0983fceb [INLONG-10318][Agent] Add PostgreSQL data source for Agent (#10320) add 83fc0711b8 [INLONG-10324][Manager] Fix the problem of useExtendedFields in Stream incorrectly overwritten (#10325) add 8171257a62 [IN
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
doleyzi commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629373166
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java:
##
@@ -385,16 +399,14 @@ private void summaryExpiredStatMap(long isolateKey) {
* Summary pre stat map
*/
private void summaryPreStatMap(long isolateKey, ConcurrentHashMap statInfo) {
-List expiredKeys =
this.expiredKeyList.computeIfAbsent(isolateKey, k -> new ArrayList<>());
+HashSet expiredKeys =
this.expiredKeyList.computeIfAbsent(isolateKey, k -> new HashSet<>());
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
doleyzi commented on code in PR #10366:
URL: https://github.com/apache/inlong/pull/10366#discussion_r1629374995
##
inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/util/RequestIdUtils.java:
##
@@ -17,33 +17,22 @@
package org.apache.inlong.audit.util;
-import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicLong;
-public class SenderResult {
+public class RequestIdUtils {
-public final InetSocketAddress addr;
-public boolean result;
+private static final Long MAX_REQUEST_ID = 10L;
+private static final AtomicLong requestIdSeq = new AtomicLong(0L);
/**
- * Constructor
- *
- * @param addr
- * @param result
+ * Next request id
*/
-public SenderResult(InetSocketAddress addr, boolean result) {
-this.addr = addr;
-this.result = result;
-}
-
-/**
- * Constructor
- *
- * @param sendIp
- * @param sendPort
- * @param result
- */
-public SenderResult(String sendIp, int sendPort, boolean result) {
-this.addr = new InetSocketAddress(sendIp, sendPort);
-this.result = result;
+public static Long nextRequestId() {
Review Comment:
It doesn't matter, requestId is just used as a request ID. When the sending
fails, the data that marks the SDK instance fails. In addition, only when it is
0, does global duplication occur, but each SDK instance is obtained by a single
thread, and duplication is impossible.
1. Both agent and dataproxy are single instances and only operate with one
thread.
2. Sort uses multiple instances. Each instance is single-threaded. A single
thread obtains the RequestId serially without duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
fuweng11 opened a new pull request, #10371: URL: https://github.com/apache/inlong/pull/10371 Fixes #10370 ### Motivation Support configuration of kV data format. ### Modifications Support configuration of kV data format. \ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
vernedeng commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629387538
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java:
##
@@ -131,6 +136,21 @@ default Format parsingFormat(
case RAW:
format = new RawFormat();
break;
+case KV:
+if (StringUtils.isNumeric(separatorStr)) {
Review Comment:
Extract to one method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
vernedeng commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629389254
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/kafka/KafkaSource.java:
##
@@ -90,6 +90,9 @@ public class KafkaSource extends StreamSource {
@ApiModelProperty(value = "Data separator")
private String dataSeparator;
+@ApiModelProperty(value = "Kv separator")
Review Comment:
KV
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
vernedeng commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629398698
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/pulsar/PulsarSource.java:
##
@@ -73,6 +73,9 @@ public class PulsarSource extends StreamSource {
@ApiModelProperty(value = "Data separator")
private String dataSeparator;
+@ApiModelProperty(value = "Kv separator")
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
fuweng11 commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629402448
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/kafka/KafkaSource.java:
##
@@ -90,6 +90,9 @@ public class KafkaSource extends StreamSource {
@ApiModelProperty(value = "Data separator")
private String dataSeparator;
+@ApiModelProperty(value = "Kv separator")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
fuweng11 commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629401071
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java:
##
@@ -131,6 +136,21 @@ default Format parsingFormat(
case RAW:
format = new RawFormat();
break;
+case KV:
+if (StringUtils.isNumeric(separatorStr)) {
Review Comment:
Fixed.
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/kafka/KafkaSourceRequest.java:
##
@@ -80,10 +78,13 @@ public class KafkaSourceRequest extends SourceRequest {
private String primaryKey;
@ApiModelProperty(value = "Data encoding format: UTF-8, GBK")
-private String dataEncoding = StandardCharsets.UTF_8.toString();
+private String dataEncoding;
@ApiModelProperty(value = "Data separator")
-private String dataSeparator = String.valueOf((int) '|');
+private String dataSeparator;
+
+@ApiModelProperty(value = "Kv separator")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
fuweng11 commented on code in PR #10371:
URL: https://github.com/apache/inlong/pull/10371#discussion_r1629401535
##
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/source/pulsar/PulsarSource.java:
##
@@ -73,6 +73,9 @@ public class PulsarSource extends StreamSource {
@ApiModelProperty(value = "Data separator")
private String dataSeparator;
+@ApiModelProperty(value = "Kv separator")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data [inlong]
justinwwhuang merged PR #10366: URL: https://github.com/apache/inlong/pull/10366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication of audit data (#10366)
This is an automated email from the ASF dual-hosted git repository.
wenweihuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 4bd806e9fc [INLONG-10365][Audit] Optimizing TCP sticky packets may
lead to duplication of audit data (#10366)
4bd806e9fc is described below
commit 4bd806e9fcd02086ad5719c8ff100eb253bffece
Author: doleyzi <43397300+dole...@users.noreply.github.com>
AuthorDate: Thu Jun 6 20:34:39 2024 +0800
[INLONG-10365][Audit] Optimizing TCP sticky packets may lead to duplication
of audit data (#10366)
* Optimizing TCP sticky packets may lead to duplication of audit data
* Remove useless code
* Added tcp packet return judgment
* update test case
* update test case
* Optimizing TCP sticky packets may lead to duplication of audit data
* use HashSet instead if List
* Use the interface Set instead of the specific implementation
---
.../org/apache/inlong/audit/AuditReporterImpl.java | 63 ++--
.../AuditMetric.java} | 36 ++-
.../org/apache/inlong/audit/send/ProxyManager.java | 65
.../apache/inlong/audit/send/SenderChannel.java| 215
.../org/apache/inlong/audit/send/SenderGroup.java | 270
.../apache/inlong/audit/send/SenderHandler.java| 110 ---
.../apache/inlong/audit/send/SenderManager.java| 360 +
.../org/apache/inlong/audit/util/AuditConfig.java | 36 +--
.../java/org/apache/inlong/audit/util/Decoder.java | 57
.../apache/inlong/audit/util/EventLoopUtil.java| 118 ---
.../{SenderResult.java => RequestIdUtils.java} | 35 +-
...enderManagerTest.java => ProxyManagerTest.java} | 42 ++-
.../apache/inlong/audit/send/SenderGroupTest.java | 53 ---
.../inlong/audit/send/SenderManagerTest.java | 15 +-
.../inlong/audit/util/RequestIdUtilsTest.java} | 26 +-
15 files changed, 338 insertions(+), 1163 deletions(-)
diff --git
a/inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java
b/inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java
index da3a1144e5..745821b512 100644
---
a/inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java
+++
b/inlong-audit/audit-sdk/src/main/java/org/apache/inlong/audit/AuditReporterImpl.java
@@ -18,15 +18,18 @@
package org.apache.inlong.audit;
import org.apache.inlong.audit.entity.AuditInformation;
+import org.apache.inlong.audit.entity.AuditMetric;
import org.apache.inlong.audit.entity.FlowType;
import org.apache.inlong.audit.loader.SocketAddressListLoader;
import org.apache.inlong.audit.protocol.AuditApi;
+import org.apache.inlong.audit.send.ProxyManager;
import org.apache.inlong.audit.send.SenderManager;
import org.apache.inlong.audit.util.AuditConfig;
import org.apache.inlong.audit.util.AuditDimensions;
import org.apache.inlong.audit.util.AuditManagerUtils;
import org.apache.inlong.audit.util.AuditValues;
import org.apache.inlong.audit.util.Config;
+import org.apache.inlong.audit.util.RequestIdUtils;
import org.apache.inlong.audit.util.StatInfo;
import org.apache.commons.lang3.ClassUtils;
@@ -35,12 +38,12 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.Serializable;
-import java.util.ArrayList;
import java.util.Calendar;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.StringJoiner;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
@@ -61,7 +64,7 @@ public class AuditReporterImpl implements Serializable {
private static final int BATCH_NUM = 100;
private static final int PERIOD = 1000 * 60;
// Resource isolation key is used in checkpoint, the default value is 0.
-private static final long DEFAULT_ISOLATE_KEY = 0;
+public static final long DEFAULT_ISOLATE_KEY = 0;
private final ReentrantLock GLOBAL_LOCK = new ReentrantLock();
private final ConcurrentHashMap>
preStatMap =
new ConcurrentHashMap<>();
@@ -69,7 +72,7 @@ public class AuditReporterImpl implements Serializable {
new ConcurrentHashMap<>();
private final ConcurrentHashMap>
expiredStatMap =
new ConcurrentHashMap<>();
-private final ConcurrentHashMap> expiredKeyList = new
ConcurrentHashMap<>();
+private final ConcurrentHashMap> expiredKeyList =
new ConcurrentHashMap<>();
private final ConcurrentHashMap flushTime = new
ConcurrentHashMap<>();
private final Config config = new Config();
private final ScheduledExecutorService timeoutExecutor =
Executors.newSingleThreadScheduledExecutor();
@@ -83,6 +86,8 @@ public class AuditReporterImpl implements Serializable {
private int flushStatThreshold = 100;
Re: [PR] [INLONG-10330][Manager] Support field template management [inlong]
dockerzhang merged PR #10333: URL: https://github.com/apache/inlong/pull/10333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10330][Manager] Support field template management (#10333)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 26927f526f [INLONG-10330][Manager] Support field template management
(#10333)
26927f526f is described below
commit 26927f526f28f227cfb2224a10bd9d1fe042e075
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Fri Jun 7 09:44:48 2024 +0800
[INLONG-10330][Manager] Support field template management (#10333)
---
.../inlong/manager/common/enums/ErrorCodeEnum.java | 8 +-
.../common/enums/TemplateVisibleRange.java}| 17 +-
.../inlong/manager/dao/entity/TemplateEntity.java} | 31 ++-
.../manager/dao/entity/TemplateFieldEntity.java} | 33 ++-
.../manager/dao/mapper/TemplateEntityMapper.java} | 28 +-
.../dao/mapper/TemplateFieldEntityMapper.java} | 32 ++-
.../resources/mappers/TemplateEntityMapper.xml | 108
.../mappers/TemplateFieldEntityMapper.xml | 122 +
.../inlong/manager/pojo/stream/TemplateField.java | 117 +
.../inlong/manager/pojo/stream/TemplateInfo.java | 59 +
.../manager/pojo/stream/TemplatePageRequest.java | 62 +
.../manager/pojo/stream/TemplateRequest.java | 59 +
.../manager/service/stream/TemplateService.java| 83 ++
.../service/stream/TemplateServiceImpl.java| 287 +
.../main/resources/h2/apache_inlong_manager.sql| 41 +++
.../manager-web/sql/apache_inlong_manager.sql | 44
inlong-manager/manager-web/sql/changes-1.13.0.sql | 43 +++
17 files changed, 1132 insertions(+), 42 deletions(-)
diff --git
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/ErrorCodeEnum.java
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/ErrorCodeEnum.java
index 98a7f50bd7..ed7549b699 100644
---
a/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/ErrorCodeEnum.java
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/ErrorCodeEnum.java
@@ -162,9 +162,13 @@ public enum ErrorCodeEnum {
MODULE_INFO_INCORRECT(6002, "Module info was incorrect"),
PACKAGE_NOT_FOUND(7001, "Package does not exist/no operation authority"),
-PACKAGE_INFO_INCORRECT(7002, "Package info was incorrect")
+PACKAGE_INFO_INCORRECT(7002, "Package info was incorrect"),
-;
+TEMPLATE_NOT_FOUND(8001, "Template does not exist/no operation authority"),
+TEMPLATE_NAME_DUPLICATE(8002, "The current template name is exist"),
+TEMPLATE_INFO_INCORRECT(8003, "Template info was incorrect"),
+TEMPLATE_FIELD_UPDATE_NOT_ALLOWED(8004, "Current status not allowed to
modification/delete field"),
+TEMPLATE_PERMISSION_DENIED(8005, "No permission to this inlong template");
private final int code;
private final String message;
diff --git a/inlong-manager/manager-web/sql/changes-1.13.0.sql
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/TemplateVisibleRange.java
similarity index 57%
copy from inlong-manager/manager-web/sql/changes-1.13.0.sql
copy to
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/TemplateVisibleRange.java
index 7013c2990f..9b787b0c46 100644
--- a/inlong-manager/manager-web/sql/changes-1.13.0.sql
+++
b/inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/enums/TemplateVisibleRange.java
@@ -15,14 +15,15 @@
* limitations under the License.
*/
--- This is the SQL change file from version 1.12.0 to the current version
1.13.0.
--- When upgrading to version 1.13.0, please execute those SQLs in the DB (such
as MySQL) used by the Manager module.
+package org.apache.inlong.manager.common.enums;
-SET NAMES utf8mb4;
-SET FOREIGN_KEY_CHECKS = 0;
+/**
+ * TemplateVisibleRange
+ */
+public enum TemplateVisibleRange {
-USE `apache_inlong_manager`;
+ALL,
+IN_CHARGE,
+TENANT
-ALTER TABLE `inlong_cluster_node` ADD COLUMN `username` varchar(256) DEFAULT
NULL COMMENT 'username for ssh';
-ALTER TABLE `inlong_cluster_node` ADD COLUMN `password` varchar(256) DEFAULT
NULL COMMENT 'password for ssh';
-ALTER TABLE `inlong_cluster_node` ADD COLUMN `ssh_port` int(11) DEFAULT NULL
COMMENT 'ssh port';
+}
diff --git a/inlong-manager/manager-web/sql/changes-1.13.0.sql
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TemplateEntity.java
similarity index 57%
copy from inlong-manager/manager-web/sql/changes-1.13.0.sql
copy to
inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TemplateEntity.java
index 7013c2990f..f9156ca190 100644
--- a/inlong-manager/manager-web/sql/changes-1.13.0.sql
+++
b/inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TemplateEntity.java
@@ -
Re: [PR] [INLONG-10370][Manager] Support configuration of kV data format [inlong]
EMsnap merged PR #10371: URL: https://github.com/apache/inlong/pull/10371 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10370][Manager] Support configuration of kV data format (#10371)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new e9cbc3671f [INLONG-10370][Manager] Support configuration of kV data
format (#10371)
e9cbc3671f is described below
commit e9cbc3671fc3ecad4b6ca70e628015d4fbb1e03b
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Fri Jun 7 10:08:23 2024 +0800
[INLONG-10370][Manager] Support configuration of kV data format (#10371)
---
.../org/apache/inlong/common/enums/DataTypeEnum.java | 1 +
.../org/apache/inlong/common/util/StringUtil.java| 9 +
.../pojo/sort/node/base/ExtractNodeProvider.java | 20 ++--
.../pojo/sort/node/provider/KafkaProvider.java | 2 ++
.../pojo/sort/node/provider/PulsarProvider.java | 2 ++
.../pojo/sort/node/provider/TubeMqProvider.java | 2 ++
.../manager/pojo/source/kafka/KafkaSource.java | 3 +++
.../manager/pojo/source/kafka/KafkaSourceDTO.java| 8 ++--
.../pojo/source/kafka/KafkaSourceRequest.java| 9 +
.../manager/pojo/source/pulsar/PulsarSource.java | 3 +++
.../manager/pojo/source/pulsar/PulsarSourceDTO.java | 8 ++--
.../pojo/source/pulsar/PulsarSourceRequest.java | 9 +
.../manager/pojo/source/tubemq/TubeMQSource.java | 9 +
.../manager/pojo/source/tubemq/TubeMQSourceDTO.java | 12 +++-
.../pojo/source/tubemq/TubeMQSourceRequest.java | 9 +
15 files changed, 87 insertions(+), 19 deletions(-)
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/enums/DataTypeEnum.java
b/inlong-common/src/main/java/org/apache/inlong/common/enums/DataTypeEnum.java
index 385c5ed71d..e0dc5c1251 100644
---
a/inlong-common/src/main/java/org/apache/inlong/common/enums/DataTypeEnum.java
+++
b/inlong-common/src/main/java/org/apache/inlong/common/enums/DataTypeEnum.java
@@ -25,6 +25,7 @@ import java.util.Locale;
public enum DataTypeEnum {
CSV("csv"),
+KV("kv"),
AVRO("avro"),
JSON("json"),
CANAL("canal"),
diff --git
a/inlong-common/src/main/java/org/apache/inlong/common/util/StringUtil.java
b/inlong-common/src/main/java/org/apache/inlong/common/util/StringUtil.java
index cddef9dc8f..7df1eb6975 100644
--- a/inlong-common/src/main/java/org/apache/inlong/common/util/StringUtil.java
+++ b/inlong-common/src/main/java/org/apache/inlong/common/util/StringUtil.java
@@ -17,6 +17,8 @@
package org.apache.inlong.common.util;
+import org.apache.commons.lang3.StringUtils;
+
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
@@ -274,4 +276,11 @@ public class StringUtil {
}
}
+public static String parseChar(String charStr) {
+if (StringUtils.isNumeric(charStr)) {
+char numberChar = (char) Integer.parseInt(charStr);
+charStr = Character.toString(numberChar);
+}
+return charStr;
+}
}
diff --git
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java
index 81b9763d79..52c09062d7 100644
---
a/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java
+++
b/inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/node/base/ExtractNodeProvider.java
@@ -19,6 +19,7 @@ package org.apache.inlong.manager.pojo.sort.node.base;
import org.apache.inlong.common.enums.DataTypeEnum;
import org.apache.inlong.common.enums.MessageWrapType;
+import org.apache.inlong.common.util.StringUtil;
import
org.apache.inlong.manager.common.fieldtype.strategy.FieldTypeMappingStrategy;
import org.apache.inlong.manager.pojo.sort.util.FieldInfoUtils;
import org.apache.inlong.manager.pojo.stream.StreamField;
@@ -32,10 +33,10 @@ import
org.apache.inlong.sort.protocol.node.format.DebeziumJsonFormat;
import org.apache.inlong.sort.protocol.node.format.Format;
import org.apache.inlong.sort.protocol.node.format.InLongMsgFormat;
import org.apache.inlong.sort.protocol.node.format.JsonFormat;
+import org.apache.inlong.sort.protocol.node.format.KvFormat;
import org.apache.inlong.sort.protocol.node.format.RawFormat;
import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang3.StringUtils;
import java.util.List;
import java.util.Objects;
@@ -90,8 +91,10 @@ public interface ExtractNodeProvider extends NodeProvider {
* Parse format
*
* @param serializationType data serialization, support: csv, json, canal,
avro, etc
- * @param wrapWithInlongMsg whether wrap content with {@link
InLongMsgFormat}
+ * @param wrapType whether wrap content with {@link InLongMsgFormat}
* @param separatorStr the sep
[PR] [INLONG-10363][Manager] Support template multi tenant management [inlong]
fuweng11 opened a new pull request, #10372: URL: https://github.com/apache/inlong/pull/10372 Fixes #10363 ### Motivation Support template multi tenant management. ### Modifications Support template multi tenant management. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10373][Manager] Manager client support template operation [inlong]
fuweng11 opened a new pull request, #10374: URL: https://github.com/apache/inlong/pull/10374 Fixes #10373 ### Motivation Manager client support template operation. ### Modifications Manager client support template operation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10375][Manager] Add field mapping strategy for CLS, StarRocks and Elasticsearch [inlong]
fuweng11 opened a new pull request, #10376: URL: https://github.com/apache/inlong/pull/10376 Fixes #10375 ### Motivation Add field mapping strategy for CLS, StarRocks and Elasticsearch. ### Modifications Add field mapping strategy for CLS, StarRocks and Elasticsearch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10284][Manager][Sort] Upgrade flink default version to 1.15 [inlong]
dockerzhang commented on code in PR #10310: URL: https://github.com/apache/inlong/pull/10310#discussion_r1630703575 ## inlong-sort/sort-formats/format-row/pom.xml: ## @@ -52,10 +52,18 @@ -v1.13 +all-flink-version Review Comment: ```suggestion flink-all-version ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
