[GitHub] [inlong] gosonzhang commented on issue #6406: [Improve][DataProxy] DataProxy should support creating sink dynamically after started
gosonzhang commented on issue #6406: URL: https://github.com/apache/inlong/issues/6406#issuecomment-1303054850 Thanks @bruceneenhl, let me see it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #6404: [INLONG-6402][Sort] Modify the metadata field of oracle connector
gong commented on code in PR #6404:
URL: https://github.com/apache/inlong/pull/6404#discussion_r1013689847
##
inlong-sort/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/table/OracleReadableMetaData.java:
##
@@ -47,6 +47,70 @@
/** Defines the supported metadata columns for {@link OracleTableSource}. */
public enum OracleReadableMetaData {
+/**
+ * Name of the table that contain the row.
+ */
+TABLE_NAME(
+"table_name",
+DataTypes.STRING().notNull(),
+new MetadataConverter() {
+private static final long serialVersionUID = 1L;
+
+@Override
+public Object read(SourceRecord record) {
+return StringData.fromString(getMetaData(record,
AbstractSourceInfo.TABLE_NAME_KEY));
+}
+}),
+
+/**
+ * Name of the schema that contain the row.
+ */
+SCHEMA_NAME(
+"schema_name",
+DataTypes.STRING().notNull(),
+new MetadataConverter() {
+private static final long serialVersionUID = 1L;
+
+@Override
+public Object read(SourceRecord record) {
+return StringData.fromString(getMetaData(record,
AbstractSourceInfo.SCHEMA_NAME_KEY));
+}
+}),
+
+/**
+ * Name of the database that contain the row.
+ */
+DATABASE_NAME(
+"database_name",
+DataTypes.STRING().notNull(),
+new MetadataConverter() {
+private static final long serialVersionUID = 1L;
+
+@Override
+public Object read(SourceRecord record) {
+return StringData.fromString(getMetaData(record,
AbstractSourceInfo.DATABASE_NAME_KEY));
+}
+}),
+
+/**
+ * It indicates the time that the change was made in the database. If the
record is read from
+ * snapshot of the table instead of the binlog, the value is always 0.
Review Comment:
binlog is concept of mysql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] thesumery opened a new pull request, #6407: [INLONG-6401][Sort] Schema update stuck in dead loop cause stackoverflow in multiple sink scences
thesumery opened a new pull request, #6407: URL: https://github.com/apache/inlong/pull/6407 [INLONG-6401][Sort] Bugfix:Schema update stuck in dead loop cause stackoverflow in multiple sink scences ### Prepare a Pull Request - [INLONG-6401][Sort] Bugfix:Schema update stuck in dead loop cause stackoverflow in multiple sink scences - Fixes #5401 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve* ### Modifications *If can not handle this schema update, should not push data into next operator, just break the dead loop* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] yunqingmoswu opened a new issue, #583: [Feature][Doc] Support dynamic topic and dynamic partition for KafkaLoadNode
yunqingmoswu opened a new issue, #583: URL: https://github.com/apache/inlong-website/issues/583 ### Description Support dynamic topic and dynamic partition for KafkaLoadNode, describe how to config to support it. ### Use case _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] healchow merged pull request #6405: [INLONG-6399][Manager] Support adding extension params for InLong Group and Stream
healchow merged PR #6405: URL: https://github.com/apache/inlong/pull/6405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (3124201da -> 53d8c0eb1)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
from 3124201da [INLONG-6373][Sort] Fix the time value when data type is
datetime and timestamp (#6403)
add 53d8c0eb1 [INLONG-6399][Manager] Support adding extension params for
InLong Group and Stream (#6405)
No new revisions were added by this update.
Summary of changes:
.../inlong/manager/client/ut/Kafka2HiveTest.java | 85 +-
.../client/api/inner/ClientFactoryTest.java| 116 +--
.../BaseInlongGroup.java} | 15 +--
.../inlong/manager/pojo/group/InlongGroupInfo.java | 8 +-
.../manager/pojo/group/InlongGroupRequest.java | 4 +-
.../manager/pojo/group/kafka/InlongKafkaInfo.java | 4 +-
.../pojo/group/pulsar/InlongPulsarInfo.java| 4 +-
.../BaseInlongStream.java} | 15 +--
.../manager/pojo/stream/InlongStreamInfo.java | 19 +---
.../manager/pojo/stream/InlongStreamRequest.java | 4 +-
.../manager/pojo/stream/InlongStreamResponse.java | 125 -
.../web/controller/InlongStreamController.java | 5 +-
12 files changed, 118 insertions(+), 286 deletions(-)
copy
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/{sink/greenplum/GreenplumColumnInfo.java
=> group/BaseInlongGroup.java} (74%)
copy
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/{sink/greenplum/GreenplumColumnInfo.java
=> stream/BaseInlongStream.java} (74%)
delete mode 100644
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/stream/InlongStreamResponse.java
[GitHub] [inlong] leezng merged pull request #6400: [INLONG-6397][Dashboard] Support management for MySQL data node
leezng merged PR #6400: URL: https://github.com/apache/inlong/pull/6400 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (53d8c0eb1 -> 1ad2d996a)
This is an automated email from the ASF dual-hosted git repository.
leezng pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
from 53d8c0eb1 [INLONG-6399][Manager] Support adding extension params for
InLong Group and Stream (#6405)
add 1ad2d996a [INLONG-6397][Dashboard] Support management for MySQL data
node (#6400)
No new revisions were added by this update.
Summary of changes:
inlong-dashboard/src/locales/cn.json | 4 +++
inlong-dashboard/src/locales/en.json | 4 +++
.../src/metas/nodes/defaults/{Hive.ts => MySQL.ts} | 29 --
inlong-dashboard/src/metas/nodes/defaults/index.ts | 5
4 files changed, 29 insertions(+), 13 deletions(-)
copy inlong-dashboard/src/metas/nodes/defaults/{Hive.ts => MySQL.ts} (71%)
[GitHub] [inlong] gosonzhang commented on issue #6406: [Improve][DataProxy] DataProxy should support creating sink dynamically after started
gosonzhang commented on issue #6406: URL: https://github.com/apache/inlong/issues/6406#issuecomment-1303107201 This should be a problem, I'll fix it, thanks @bruceneenhl -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap opened a new pull request, #6408: [INLONG-6370][Sort] The op type in debezium format should be u or update
EMsnap opened a new pull request, #6408: URL: https://github.com/apache/inlong/pull/6408 - Fixes #6370 ### Motivation The op type in debezium format should be u or update ### Modifications The op type in debezium format should be u or update ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (1ad2d996a -> 9228a22d6)
This is an automated email from the ASF dual-hosted git repository. zirui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 1ad2d996a [INLONG-6397][Dashboard] Support management for MySQL data node (#6400) add 9228a22d6 [INLONG-6379][Sort] Bugfix:iceberg miss metric data in multiple sink (#6381) No new revisions were added by this update. Summary of changes: .../apache/inlong/sort/iceberg/sink/FlinkSink.java | 3 +- .../sink/multiple/IcebergMultipleStreamWriter.java | 59 +- 2 files changed, 60 insertions(+), 2 deletions(-)
[GitHub] [inlong] EMsnap merged pull request #6381: [INLONG-6379][Sort] Iceberg misses metric data in multiple sink scenes
EMsnap merged PR #6381: URL: https://github.com/apache/inlong/pull/6381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #6383: [INLONG-6382][Sort] Iceberg misses data when the source table has no primary key in multiple sink scenes
EMsnap commented on PR #6383: URL: https://github.com/apache/inlong/pull/6383#issuecomment-1303148604 I think the Pr title doesn't match with the code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] thesumery commented on a diff in pull request #6383: [INLONG-6382][Sort] Iceberg misses data when the source table has no primary key in multiple sink scenes
thesumery commented on code in PR #6383:
URL: https://github.com/apache/inlong/pull/6383#discussion_r1013781469
##
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergMultipleStreamWriter.java:
##
@@ -130,12 +129,6 @@ public void processElement(RecordWithSchema
recordWithSchema) throws Exception {
List equalityFieldIds =
recordWithSchema.getPrimaryKeys().stream()
.map(pk ->
recordWithSchema.getSchema().findField(pk).fieldId())
.collect(Collectors.toList());
-// if physical primary key not exist, put all field to logical
primary key
-if (equalityFieldIds.isEmpty()) {
Review Comment:
good proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on issue #6410: [Bug][Manager] The extension parameters of group and stream cannot be saved to ext_params
fuweng11 commented on issue #6410: URL: https://github.com/apache/inlong/issues/6410#issuecomment-1303280323 I will deal with this problem! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 opened a new pull request, #6411: [INLONG-6410][Manager] Ensure that the additional parameters of group and stream can be saved
fuweng11 opened a new pull request, #6411: URL: https://github.com/apache/inlong/pull/6411 ### Prepare a Pull Request - Fixes #6410 ### Motivation Ensure that the additional parameters of group and stream can be saved ### Modifications `InlongGroupDTO` extends `BasegInlongGroup` Ensure that the additional parameters of group and stream can be saved ### Verifying this change - [X] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #6333: [INLONG-6332][Agent] Fix reboot will reset file position error
dockerzhang merged PR #6333: URL: https://github.com/apache/inlong/pull/6333 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-6332][Agent] Fix reboot will reset file position error (#6333)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new d4d4dd5b4 [INLONG-6332][Agent] Fix reboot will reset file position
error (#6333)
d4d4dd5b4 is described below
commit d4d4dd5b4cae8240f03ddf413f2abf13fa979456
Author: ganfengtan
AuthorDate: Fri Nov 4 21:18:58 2022 +0800
[INLONG-6332][Agent] Fix reboot will reset file position error (#6333)
Co-authored-by: healchow
---
.../agent/core/task/TaskPositionManager.java | 2 +
.../sources/reader/file/MonitorTextFile.java | 101 -
2 files changed, 60 insertions(+), 43 deletions(-)
diff --git
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskPositionManager.java
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskPositionManager.java
index c3bfa0abe..c5b944210 100644
---
a/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskPositionManager.java
+++
b/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskPositionManager.java
@@ -139,6 +139,8 @@ public class TaskPositionManager extends AbstractDaemon {
ConcurrentHashMap positionTemp = new
ConcurrentHashMap<>();
ConcurrentHashMap position =
jobTaskPositionMap.putIfAbsent(jobInstanceId, positionTemp);
if (position == null) {
+JobProfile jobProfile = jobConfDb.getJobById(jobInstanceId);
+positionTemp.put(sourcePath, jobProfile.getLong(sourcePath +
POSITION_SUFFIX, 0));
position = positionTemp;
}
Long beforePosition = position.getOrDefault(sourcePath, 0L);
diff --git
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
index d8160bd8b..1f5958cdf 100644
---
a/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
+++
b/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java
@@ -21,10 +21,10 @@ import org.apache.inlong.agent.common.AgentThreadFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.attribute.BasicFileAttributes;
-import java.util.Objects;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
@@ -35,26 +35,28 @@ import static
org.apache.inlong.agent.constant.JobConstants.JOB_FILE_MONITOR_EXP
import static
org.apache.inlong.agent.constant.JobConstants.JOB_FILE_MONITOR_INTERVAL;
/**
- * monitor files
+ * Monitor for text files
*/
public final class MonitorTextFile {
private static final Logger LOGGER =
LoggerFactory.getLogger(MonitorTextFile.class);
-private static volatile MonitorTextFile monitorTextFile = null;
// monitor thread pool
-private final ThreadPoolExecutor runningPool = new ThreadPoolExecutor(
+private static final ThreadPoolExecutor EXECUTOR_SERVICE = new
ThreadPoolExecutor(
0, Integer.MAX_VALUE,
60L, TimeUnit.SECONDS,
new SynchronousQueue<>(),
new AgentThreadFactory("monitor-file"));
+private static volatile MonitorTextFile monitorTextFile = null;
+
private MonitorTextFile() {
}
/**
- * Mode of singleton
- * @return MonitorTextFile instance
+ * Get a singleton instance of MonitorTextFile.
+ *
+ * @return monitor text file instance
*/
public static MonitorTextFile getInstance() {
if (monitorTextFile == null) {
@@ -68,37 +70,35 @@ public final class MonitorTextFile {
}
public void monitor(FileReaderOperator fileReaderOperator, TextFileReader
textFileReader) {
-MonitorEventRunnable monitorEvent = new
MonitorEventRunnable(fileReaderOperator, textFileReader);
-runningPool.execute(monitorEvent);
+EXECUTOR_SERVICE.execute(new MonitorEventRunnable(fileReaderOperator,
textFileReader));
}
/**
- * monitor file event
+ * Runnable for monitor the file event
*/
-private class MonitorEventRunnable implements Runnable {
+private static class MonitorEventRunnable implements Runnable {
private static final int WAIT_TIME = 30;
private final FileReaderOperator fileReaderOperator;
private final TextFileReader textFileReader;
private final Long interval;
private final long startTime = System.currentTimeMillis();
+private long lastFlushTime = System.currentTimeMillis();
private String path;
-/**
-
[GitHub] [inlong] github-actions[bot] commented on issue #4961: [Feature][DataProxy] Golang SDK
github-actions[bot] commented on issue #4961: URL: https://github.com/apache/inlong/issues/4961#issuecomment-1304378228 This issue is stale because it has been open for 60 days with no activity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
