[GitHub] [inlong] leezng commented on a diff in pull request #8205: [INLONG-8204][Dashboard] Support querying audit information by sink id
leezng commented on code in PR #8205: URL: https://github.com/apache/inlong/pull/8205#discussion_r1226218057 ## inlong-dashboard/src/ui/locales/cn.json: ## @@ -623,6 +623,7 @@ "pages.GroupDetail.Audit.Min": "分钟", "pages.GroupDetail.Audit.Hour": "小时", "pages.GroupDetail.Audit.Day": "天", + "pages.GroupDetail.Audit.Sink": "数据目标", Review Comment: It should not be needed at present, this system-level naming is unified -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-8197][Manager] Optimize the ClickHouse query for the Audit interface (#8198)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new cab63a8eea [INLONG-8197][Manager] Optimize the ClickHouse query for
the Audit interface (#8198)
cab63a8eea is described below
commit cab63a8eea6c0f4bf3d30ce245b7e1beee42504d
Author: Hao <1780095+hnrai...@users.noreply.github.com>
AuthorDate: Mon Jun 12 17:44:36 2023 +0800
[INLONG-8197][Manager] Optimize the ClickHouse query for the Audit
interface (#8198)
---
.../service/core/impl/AuditServiceImpl.java| 36 +++---
1 file changed, 25 insertions(+), 11 deletions(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AuditServiceImpl.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AuditServiceImpl.java
index 0cddccb59e..80fc900b01 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AuditServiceImpl.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/core/impl/AuditServiceImpl.java
@@ -67,8 +67,9 @@ import javax.annotation.PostConstruct;
import java.math.BigDecimal;
import java.sql.Connection;
+import java.sql.PreparedStatement;
import java.sql.ResultSet;
-import java.sql.Statement;
+import java.sql.SQLException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
@@ -239,9 +240,9 @@ public class AuditServiceImpl implements AuditService {
}
} else if (AuditQuerySource.CLICKHOUSE == querySource) {
try (Connection connection =
ClickHouseConfig.getCkConnection();
-Statement statement = connection.createStatement();
-ResultSet resultSet = statement.executeQuery(
-toAuditCkSql(groupId, streamId, auditId,
request.getDt( {
+PreparedStatement statement =
+getAuditCkStatement(connection, groupId,
streamId, auditId, request.getDt());
+ResultSet resultSet = statement.executeQuery()) {
List auditSet = new ArrayList<>();
while (resultSet.next()) {
AuditInfo vo = new AuditInfo();
@@ -308,28 +309,41 @@ public class AuditServiceImpl implements AuditService {
}
/**
- * Convert to clickhouse search sql
+ * Get clickhouse Statement
*
+ * @param connection The ClickHouse connection
* @param groupId The groupId of inlong
* @param streamId The streamId of inlong
* @param auditId The auditId of request
* @param dt The datetime of request
- * @return clickhouse sql
+ * @return The clickhouse Statement
*/
-private String toAuditCkSql(String groupId, String streamId, String
auditId, String dt) {
+private PreparedStatement getAuditCkStatement(Connection connection,
String groupId, String streamId,
+String auditId, String dt) throws SQLException {
DateTimeFormatter formatter = DateTimeFormat.forPattern(DAY_FORMAT);
DateTime date = formatter.parseDateTime(dt);
String startDate = date.toString(SECOND_FORMAT);
String endDate = date.plusDays(1).toString(SECOND_FORMAT);
-return new SQL()
+
+String sql = new SQL()
.SELECT("log_ts", "sum(count) as total")
.FROM("audit_data")
-.WHERE("inlong_group_id = '" + groupId + "'",
"inlong_stream_id = '" + streamId + "'",
-"audit_id = '" + auditId + "'")
-.WHERE("log_ts >= '" + startDate + "'", "log_ts < '" + endDate
+ "'")
+.WHERE("inlong_group_id = ?")
+.WHERE("inlong_stream_id = ?")
+.WHERE("audit_id = ?")
+.WHERE("log_ts >= ?")
+.WHERE("log_ts < ?")
.GROUP_BY("log_ts")
.ORDER_BY("log_ts")
.toString();
+
+PreparedStatement statement = connection.prepareStatement(sql);
+statement.setString(1, groupId);
+statement.setString(2, streamId);
+statement.setString(3, auditId);
+statement.setString(4, startDate);
+statement.setString(5, endDate);
+return statement;
}
/**
[GitHub] [inlong] dockerzhang merged pull request #8193: [INLONG-8062][Sort] Add PostgreSQL source connector on flink 1.15
dockerzhang merged PR #8193: URL: https://github.com/apache/inlong/pull/8193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-8062][Sort] Add PostgreSQL source connector on flink 1.15 (#8193)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 6c1494eb30 [INLONG-8062][Sort] Add PostgreSQL source connector on
flink 1.15 (#8193)
6c1494eb30 is described below
commit 6c1494eb308066207658d5da5e758a96bbc38509
Author: Sting
AuthorDate: Mon Jun 12 17:46:05 2023 +0800
[INLONG-8062][Sort] Add PostgreSQL source connector on flink 1.15 (#8193)
---
.../src/main/assemblies/sort-connectors-v1.15.xml | 2 +-
inlong-sort/sort-flink/sort-flink-v1.15/pom.xml| 14 +-
.../sort-flink-v1.15/sort-connectors/pom.xml | 12 +
.../sort-connectors/postgres-cdc/pom.xml | 150 +
.../sort/postgre/PostgreSQLTableFactory.java | 181 ++
.../org.apache.flink.table.factories.Factory | 16 +
licenses/inlong-sort-connectors/LICENSE| 699 +++--
7 files changed, 723 insertions(+), 351 deletions(-)
diff --git a/inlong-distribution/src/main/assemblies/sort-connectors-v1.15.xml
b/inlong-distribution/src/main/assemblies/sort-connectors-v1.15.xml
index d5c3165050..5ef95c3243 100644
--- a/inlong-distribution/src/main/assemblies/sort-connectors-v1.15.xml
+++ b/inlong-distribution/src/main/assemblies/sort-connectors-v1.15.xml
@@ -31,7 +31,7 @@
../inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/jdbc/target
inlong-sort/connectors
-
sort-connector-jdbc-${project.version}-v1.15.jar
+
sort-connector-postgres-cdc-v1.15-${project.version}.jar
0644
diff --git a/inlong-sort/sort-flink/sort-flink-v1.15/pom.xml
b/inlong-sort/sort-flink/sort-flink-v1.15/pom.xml
index 4d01a9bd0b..6bec4ecaab 100644
--- a/inlong-sort/sort-flink/sort-flink-v1.15/pom.xml
+++ b/inlong-sort/sort-flink/sort-flink-v1.15/pom.xml
@@ -38,7 +38,7 @@
${project.parent.parent.parent.basedir}
1.15.4
-2.0.1.Final
+1.6.4.Final
2.3.0
2.3.0
2.3.0
@@ -53,6 +53,8 @@
7.2.2.jre8
0.12.3
0.9.3
+2.7.1
+
2.3.0
@@ -95,6 +97,11 @@
flink-connector-sqlserver-cdc
${flink.connector.sqlserver.cdc.version}
+
+com.ververica
+flink-connector-debezium
+${flink-connector-debezium.version}
+
com.ververica
flink-connector-mysql-cdc
@@ -218,6 +225,11 @@
pulsar-client-all
${pulsar.version}
+
+org.apache.kafka
+kafka-clients
+${kafka-clients.version}
+
diff --git a/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/pom.xml
b/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/pom.xml
index f769edeae2..7b5c1b00a4 100644
--- a/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/pom.xml
+++ b/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/pom.xml
@@ -30,8 +30,20 @@
pom
Apache InLong - Sort Connectors v1.15
+
+postgres-cdc
+
+
${project.parent.parent.parent.parent.basedir}
+
+
+org.apache.inlong
+sort-flink-dependencies-v1.15
+${project.version}
+
+
+
diff --git
a/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/postgres-cdc/pom.xml
b/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/postgres-cdc/pom.xml
new file mode 100644
index 00..6e9debb89d
--- /dev/null
+++
b/inlong-sort/sort-flink/sort-flink-v1.15/sort-connectors/postgres-cdc/pom.xml
@@ -0,0 +1,150 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+org.apache.inlong
+sort-connectors-v1.15
+1.8.0-SNAPSHOT
+
+
+sort-connector-postgres-cdc-v1.15
+jar
+Apache InLong - Sort-connector-postgres-cdc
+
+
+
${project.parent.parent.parent.parent.parent.basedir}
+
+
+
+
+
+
+com.ververica
+flink-connector-postgres-cdc
+
+
+
+com.ververica
+flink-connector-debezium
+
+
+org.apache.kafka
+kafka-log4j-appender
+
+
+
+
+
+org.apache.kafka
+kafka-clients
+${kafka-clients.version}
+
+
+
+io.debezium
+debezium-connector-postgres
+${debezium.version}
+
+
+
[GitHub] [inlong] healchow merged pull request #8205: [INLONG-8204][Dashboard] Support querying audit information by sink id
healchow merged PR #8205: URL: https://github.com/apache/inlong/pull/8205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-8204][Dashboard] Support querying audit information by sink id (#8205)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 8727a9d834 [INLONG-8204][Dashboard] Support querying audit information
by sink id (#8205)
8727a9d834 is described below
commit 8727a9d834b85837afbcba1c02f89a1a4054cda7
Author: Lizhen <88174078+bluew...@users.noreply.github.com>
AuthorDate: Mon Jun 12 17:55:52 2023 +0800
[INLONG-8204][Dashboard] Support querying audit information by sink id
(#8205)
---
inlong-dashboard/src/ui/locales/cn.json| 1 +
inlong-dashboard/src/ui/locales/en.json| 1 +
.../src/ui/pages/GroupDetail/Audit/config.tsx | 27 ++
3 files changed, 29 insertions(+)
diff --git a/inlong-dashboard/src/ui/locales/cn.json
b/inlong-dashboard/src/ui/locales/cn.json
index 3d98c4c22b..d7679e07fb 100644
--- a/inlong-dashboard/src/ui/locales/cn.json
+++ b/inlong-dashboard/src/ui/locales/cn.json
@@ -623,6 +623,7 @@
"pages.GroupDetail.Audit.Min": "分钟",
"pages.GroupDetail.Audit.Hour": "小时",
"pages.GroupDetail.Audit.Day": "天",
+ "pages.GroupDetail.Audit.Sink": "数据目标",
"pages.ApprovalDetail.GroupConfig.DataStorages": "数据存储",
"pages.ApprovalDetail.GroupConfig.ApprovalInformation": "审批信息",
"pages.ApprovalDetail.GroupConfig.DataFlowInformation": "数据流信息",
diff --git a/inlong-dashboard/src/ui/locales/en.json
b/inlong-dashboard/src/ui/locales/en.json
index b982b5f58e..fa104096bd 100644
--- a/inlong-dashboard/src/ui/locales/en.json
+++ b/inlong-dashboard/src/ui/locales/en.json
@@ -623,6 +623,7 @@
"pages.GroupDetail.Audit.Min": "Minute",
"pages.GroupDetail.Audit.Hour": "Hour",
"pages.GroupDetail.Audit.Day": "Day",
+ "pages.GroupDetail.Audit.Sink": "Sink",
"pages.ApprovalDetail.GroupConfig.DataStorages": "DataStorages",
"pages.ApprovalDetail.GroupConfig.ApprovalInformation": "Approval
information",
"pages.ApprovalDetail.GroupConfig.DataFlowInformation": "Data stream
information",
diff --git a/inlong-dashboard/src/ui/pages/GroupDetail/Audit/config.tsx
b/inlong-dashboard/src/ui/pages/GroupDetail/Audit/config.tsx
index 7a7637f0e5..ec6527a9c7 100644
--- a/inlong-dashboard/src/ui/pages/GroupDetail/Audit/config.tsx
+++ b/inlong-dashboard/src/ui/pages/GroupDetail/Audit/config.tsx
@@ -130,6 +130,33 @@ export const getFormContent = (inlongGroupId,
initialValues, onSearch, onDataStr
},
rules: [{ required: true }],
},
+ {
+type: 'select',
+label: i18n.t('pages.GroupDetail.Audit.Sink'),
+name: 'sinkId',
+props: values => ({
+ dropdownMatchSelectWidth: false,
+ options: {
+requestService: {
+ url: '/sink/list',
+ method: 'POST',
+ data: {
+pageNum: 1,
+pageSize: 1000,
+inlongGroupId,
+inlongStreamId: values.inlongStreamId,
+ },
+},
+requestParams: {
+ formatResult: result =>
+result?.list.map(item => ({
+ label: item.sinkName + ` ( ${sinks.find(c => c.value ===
item.sinkType)?.label} )`,
+ value: item.id,
+})) || [],
+},
+ },
+}),
+ },
{
type: 'datepicker',
label: i18n.t('pages.GroupDetail.Audit.Date'),
[GitHub] [inlong] EMsnap commented on a diff in pull request #8182: [INLONG-8180][Agent] Improve the efficiency and safety of log file reading
EMsnap commented on code in PR #8182:
URL: https://github.com/apache/inlong/pull/8182#discussion_r1222385551
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/MonitorTextFile.java:
##
@@ -168,7 +164,7 @@ private void listen() throws IOException {
}
/**
- * Reset the position and bytePosition
+ * reset the position and bytePositionreset the position and
bytePosition
Review Comment:
bytePositionreset typo here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] justinwwhuang opened a new pull request, #8216: [INLONG-8215][DataProxySDK] fix thread leak bug
justinwwhuang opened a new pull request, #8216: URL: https://github.com/apache/inlong/pull/8216 fix thread leak bug ### Prepare a Pull Request *[INLONG-8215][DataProxySDK] thread leak after sdk shut down* - Fixes #[Bug][DataProxySDK] thread leak after sdk shut down #8215 ### Motivation *io threads were not shut down after sdk shut down, cause threads leaking* ### Modifications *shut down the io threads and metrics threads when the sdk shut down* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong commented on pull request #8209: [INLONG-7900][Sort] Support partition by primary key when upsert single table of Kafka
gong commented on PR #8209: URL: https://github.com/apache/inlong/pull/8209#issuecomment-1588616261 @lordcheng10 please review this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] gong opened a new pull request, #8219: [INLONG-8218][Sort] Fix kafka connector reader data byte calculation error
gong opened a new pull request, #8219: URL: https://github.com/apache/inlong/pull/8219 ### Prepare a Pull Request - [INLONG-8218][Sort] Fix kafka connector reader data byte calculation error - Fixes #8218 ### Motivation * Fix kafka connector reader data byte calculation error ### Modifications * modify metrics calculation of `DynamicKafkaDeserializationSchema` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on issue #8091: [Bug] 源和目标都是postgres,报错Unsupported FieldType : INTEGER
dockerzhang commented on issue #8091: URL: https://github.com/apache/inlong/issues/8091#issuecomment-1588650389 @JeffWang79, please use English to describe this issue. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
