[PR] [INLONG-9594][sort] Need to remove duplicate classes in the same directory under different sort format modules [inlong]
baomingyu opened a new pull request, #9595: URL: https://github.com/apache/inlong/pull/9595 Need to remove duplicate classes in the same directory under different sort format modules - Fixes #9594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-9598][Agent] Modify comments [inlong]
justinwwhuang opened a new pull request, #9599: URL: https://github.com/apache/inlong/pull/9599 [INLONG-9598][Agent] Modify comments - Fixes #9598 ### Motivation Modify comments ### Modifications Modify comments ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-9598][Agent] Modify comments [inlong]
luchunliang merged PR #9599: URL: https://github.com/apache/inlong/pull/9599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9598][Agent] Modify comments (#9599)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 2fe010987c [INLONG-9598][Agent] Modify comments (#9599)
2fe010987c is described below
commit 2fe010987c9023903d1715d245e60119b2c29887
Author: justinwwhuang
AuthorDate: Mon Jan 22 14:05:35 2024 +0800
[INLONG-9598][Agent] Modify comments (#9599)
---
.../main/java/org/apache/inlong/agent/plugin/sinks/AbstractSink.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sinks/AbstractSink.java
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sinks/AbstractSink.java
index f09de34a1d..01612493eb 100644
---
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sinks/AbstractSink.java
+++
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sinks/AbstractSink.java
@@ -64,7 +64,7 @@ public abstract class AbstractSink implements Sink {
protected String sourceName;
protected String jobInstanceId;
protected int batchFlushInterval;
-// key is stream id, value is a batch of messages belong to the same
stream id
+// key is stream id, value is a batch of messages belongs to the same
stream id
protected ConcurrentHashMap cache;
@Override
Re: [PR] [INLONG-9594][sort] Need to remove duplicate classes in the same directory under different sort format modules [inlong]
EMsnap merged PR #9595: URL: https://github.com/apache/inlong/pull/9595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9594][sort] Need to remove duplicate classes in the same directory under different sort format modules (#9595)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new d62115dd6c [INLONG-9594][sort] Need to remove duplicate classes in the
same directory under different sort format modules (#9595)
d62115dd6c is described below
commit d62115dd6cfb09110ca703dcd974fcbed3eeefc3
Author: baomingyu
AuthorDate: Mon Jan 22 14:15:08 2024 +0800
[INLONG-9594][sort] Need to remove duplicate classes in the same directory
under different sort format modules (#9595)
---
.../formats/inlongmsg/IgnoreFailureHandler.java| 2 +-
.../sort/formats/inlongmsg/InLongMsgOptions.java | 27 +
.../sort/formats/inlongmsg/FailureHandler.java | 54 --
.../formats/inlongmsg/IgnoreFailureHandler.java| 52 --
.../sort/formats/inlongmsg/InLongMsgBody.java | 105 ---
.../sort/formats/inlongmsg/InLongMsgHead.java | 113 -
.../formats/inlongmsg/InLongMsgMixedValidator.java | 40
.../sort/formats/inlongmsg/InLongMsgOptions.java | 77 --
.../sort/formats/inlongmsg/InLongMsgValidator.java | 50 -
.../sort/formats/inlongmsg/NoOpFailureHandler.java | 55 --
10 files changed, 28 insertions(+), 547 deletions(-)
diff --git
a/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/IgnoreFailureHandler.java
b/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/IgnoreFailureHandler.java
index 33f46b1c2b..bb57bddfba 100644
---
a/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/IgnoreFailureHandler.java
+++
b/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/IgnoreFailureHandler.java
@@ -34,7 +34,7 @@ public class IgnoreFailureHandler implements FailureHandler {
@Override
public void onParsingBodyFailure(byte[] body, Exception exception) {
-LOG.warn("Cannot properly parse the body.", exception);
+LOG.warn("Cannot properly parse the body: {}.", new String(body),
exception);
}
@Override
diff --git
a/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/InLongMsgOptions.java
b/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/InLongMsgOptions.java
index e07b10..4d6ccf860f 100644
---
a/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/InLongMsgOptions.java
+++
b/inlong-sort/sort-formats/format-common/src/main/java/org/apache/inlong/sort/formats/inlongmsg/InLongMsgOptions.java
@@ -158,4 +158,31 @@ public class InLongMsgOptions {
.defaultValue(DEFAULT_INCLUDE_UPDATE_BEFORE)
.withDescription(
"True if the retain predefined field should be
skip the predefined Field. (true by default).");
+
+public static final ConfigOption CSV_IGNORE_PARSE_ERRORS =
+ConfigOptions.key("csv.ignore-parse-errors")
+.booleanType()
+.defaultValue(false)
+.withDescription("Allows the case that real size exceeds
the expected size.\n "
++ "The extra column will be skipped");
+
+public static final ConfigOption CSV_IGNORE_TRAILING_UNMAPPABLE =
+ConfigOptions.key("csv.ignore-trailing-unmappable")
+.booleanType()
+.defaultValue(false)
+.withDescription("Allows the case that real size exceeds
the expected size.\n "
++ "The extra column will be skipped");
+
+public static final ConfigOption
CSV_INSERT_NULLS_FOR_MISSING_COLUMNS =
+ConfigOptions.key("csv.insert-nulls-for-missing-columns")
+.booleanType()
+.defaultValue(false)
+.withDescription("For missing columns, insert null.");
+
+public static final ConfigOption CSV_EMPTY_STRING_AS_NULL =
+ConfigOptions.key("csv.empty-string-as-null")
+.booleanType()
+.defaultValue(false)
+.withDescription("if the string value is empty, make it as
null");
+
}
diff --git
a/inlong-sort/sort-formats/format-row/format-inlongmsg-base/src/main/java/org/apache/inlong/sort/formats/inlongmsg/FailureHandler.java
b/inlong-sort/sort-formats/format-row/format-inlongmsg-base/src/main/java/org/apache/inlong/sort/formats/inlongmsg/FailureHandler.java
deleted file mode 100644
index 00b0249d51..00
---
a/inlong-sort/sort-formats/format-row/format-inlongmsg-base/src/main/java/org/apache/inlong/sort/formats/inlongmsg/FailureHandler.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licen
[PR] [INLONG-9600][Agent] Adjust the sins directory for code consistency [inlong]
justinwwhuang opened a new pull request, #9601: URL: https://github.com/apache/inlong/pull/9601 [INLONG-9600][Agent] Adjust the sins directory for code consistency - Fixes #9600 ### Motivation Adjust the sins directory for code consistency ### Modifications Adjust the sins directory for code consistency ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
