[GitHub] [inlong] fuweng11 opened a new pull request, #8089: [INLONG-8088][Manager] Support configure heartbeat timeout interval
fuweng11 opened a new pull request, #8089: URL: https://github.com/apache/inlong/pull/8089 ### Prepare a Pull Request - Fixes #8088 ### Motivation Support configure heartbeat timeout interval. ### Modifications Support configure heartbeat timeout interval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng opened a new pull request, #8090: [INLONG-8087] Add definition of Inlong tenant table
vernedeng opened a new pull request, #8090: URL: https://github.com/apache/inlong/pull/8090 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #8087 - Parent #7914 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8086: [INLONG-8042][CI] Update the CI workflow to avoid the network being unreachable
dockerzhang merged PR #8086: URL: https://github.com/apache/inlong/pull/8086 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (e620e845d -> 94a98be87)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from e620e845d [INLONG-8078][Dashboard] Fix the Source and sink list pagination does not work (#8079) add 94a98be87 [INLONG-8042][CI] Update the CI workflow to avoid the network being unreachable (#8086) No new revisions were added by this update. Summary of changes: .github/workflows/ci_build.yml| 2 +- .github/workflows/ci_ut.yml | 2 +- .github/workflows/codeql_analysis.yml | 2 +- .idea/vcs.xml | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-)
[GitHub] [inlong] dockerzhang commented on pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
dockerzhang commented on PR #8090: URL: https://github.com/apache/inlong/pull/8090#issuecomment-1562427106 please update the `changes-1.8.0.sql` at the same time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8075: [INLONG-8068][Manager] Support repeatable read for http request
dockerzhang merged PR #8075: URL: https://github.com/apache/inlong/pull/8075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-8068][Manager] Support repeatable read for http request (#8075)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 2da89aa15 [INLONG-8068][Manager] Support repeatable read for http
request (#8075)
2da89aa15 is described below
commit 2da89aa157dae5b59417587f04c73fda394e4dd2
Author: vernedeng
AuthorDate: Thu May 25 16:18:02 2023 +0800
[INLONG-8068][Manager] Support repeatable read for http request (#8075)
Co-authored-by: vernedeng
---
.../inlong/manager/web/auth/InlongShiroImpl.java | 4 +
.../manager/web/auth/web/AuthenticationFilter.java | 1 -
.../web/filter/HttpServletRequestFilter.java | 54
.../inlong/manager/web/utils/HttpContextUtils.java | 111
.../manager/web/utils/InlongRequestWrapper.java| 144 +
.../web/filter/HttpServletRequestFilterTest.java | 77 +++
6 files changed, 390 insertions(+), 1 deletion(-)
diff --git
a/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/InlongShiroImpl.java
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/InlongShiroImpl.java
index 5954a367b..178323cc8 100644
---
a/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/InlongShiroImpl.java
+++
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/InlongShiroImpl.java
@@ -94,9 +94,13 @@ public class InlongShiroImpl implements InlongShiro {
shiroFilterFactoryBean.setSecurityManager(securityManager);
// anon: can be accessed by anyone, authc: only authentication is
successful can be accessed
Map filters = new LinkedHashMap<>();
+
+// request filter
filters.put(FILTER_NAME_WEB, new AuthenticationFilter());
+
shiroFilterFactoryBean.setFilters(filters);
Map pathDefinitions = new LinkedHashMap<>();
+
// login, register request
pathDefinitions.put("/api/anno/**/*", "anon");
diff --git
a/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/web/AuthenticationFilter.java
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/web/AuthenticationFilter.java
index d40acb7c1..60f11b772 100644
---
a/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/web/AuthenticationFilter.java
+++
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/auth/web/AuthenticationFilter.java
@@ -59,7 +59,6 @@ public class AuthenticationFilter implements Filter {
public void doFilter(ServletRequest servletRequest, ServletResponse
servletResponse, FilterChain filterChain)
throws IOException, ServletException {
HttpServletRequest httpServletRequest = (HttpServletRequest)
servletRequest;
-
Subject subject = SecurityUtils.getSubject();
if (subject.isAuthenticated()) {
UserInfo loginUserInfo = (UserInfo) subject.getPrincipal();
diff --git
a/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/filter/HttpServletRequestFilter.java
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/filter/HttpServletRequestFilter.java
new file mode 100644
index 0..387095324
--- /dev/null
+++
b/inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/filter/HttpServletRequestFilter.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.web.filter;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.inlong.manager.web.utils.InlongRequestWrapper;
+import org.springframework.core.annotation.Order;
+import org.springframework.stereotype.Component;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.annotation.WebFilter;
+import javax.servlet.http.HttpServletRequest;
+import java.io.IOException;
+
+/**
+ * HttpServletRequestFilter
+ * Make All
+ */
+@Slf4j
+@Component
+@WebFilter
+@Order(0)
+public cl
[GitHub] [inlong] vernedeng commented on a diff in pull request #8081: [INLONG-8080][Manager] Fix the total parameter found on the page is different from the actual value
vernedeng commented on code in PR #8081: URL: https://github.com/apache/inlong/pull/8081#discussion_r1205192011 ## inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java: ## @@ -322,9 +323,11 @@ public PageResult listByCondition(SinkPageRequest request, PageHelper.startPage(request.getPageNum(), request.getPageSize()); OrderFieldEnum.checkOrderField(request); OrderTypeEnum.checkOrderType(request); -List entityPage = sinkMapper.selectByCondition(request); +Page entityPage = (Page) sinkMapper.selectByCondition(request); Review Comment: Please use pageHelper to do paging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8081: [INLONG-8080][Manager] Fix the total parameter found on the page is different from the actual value
vernedeng commented on code in PR #8081: URL: https://github.com/apache/inlong/pull/8081#discussion_r1205204892 ## inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/StreamSinkServiceImpl.java: ## @@ -322,9 +323,11 @@ public PageResult listByCondition(SinkPageRequest request, PageHelper.startPage(request.getPageNum(), request.getPageSize()); OrderFieldEnum.checkOrderField(request); OrderTypeEnum.checkOrderType(request); -List entityPage = sinkMapper.selectByCondition(request); +Page entityPage = (Page) sinkMapper.selectByCondition(request); Review Comment: Please use PageHelper to get the total number of entities. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
vernedeng commented on PR #8090: URL: https://github.com/apache/inlong/pull/8090#issuecomment-1562573366 > please update the `changes-1.8.0.sql` at the same time. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] hejiay opened a new pull request, #8095: [INLONG-7959][SORT]Dynamic schema evolution support delete,update column when multiple sink to iceberg table
hejiay opened a new pull request, #8095: URL: https://github.com/apache/inlong/pull/8095 ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes [#7959](https://github.com/apache/inlong/issues/7959) ### Motivation *support delete,update column when multiple sink to iceberg table* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] liaorui opened a new pull request, #8096: [INLONG-8092][Sort]Hive connector supports all database and multiple tables data transmi…
liaorui opened a new pull request, #8096: URL: https://github.com/apache/inlong/pull/8096 …ssion ### Prepare a Pull Request *(Change the title refer to the following example)* - Title Example: [INLONG-XYZ][Component] Title of the pull request *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)* - Fixes #8092 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* 1. Hive connector can only transmit data from source cdc to one hive table now. It may support all database and multiple tables data transmission with `sink.multiple.enable` and other `sink.multiple.*` options. You can find more description in InLong sort document. 2. Hive connector only supports hive 3.x version. Now it can support hive 2.x by `mvn clean install -pl org.apache.inlong:sort-connector-hive -am -DskipTests -Phive2`. Maven profile `hive2` in pom.xml imports `hive-exe:2.2.0` denpendency. ### Modifications *Describe the modifications you've done.* 1. Flink official `flink-connector-hive` connector can only write one hive table at once, as it reads one batch data and write into one HDFS path. In all db migration scene, we want it to read data from multiple source tables and write them into multiple hive tables. Parameter `org.apache.flink.core.fs.Path flinkPath`, in `openNewInProgressFile` method of `HadoopPathBasedBucketWriter` class, represents HDFS path, we should compose `flinkPath` with sink database and table dynamically. So we rewrite these classes `HadoopPathBasedPartFileWriter`, `HadoopPathBasedBulkFormatBuilder`, `DefaultHadoopFileCommitterFactory`, `DefaultHadoopFileCommitterFactory` and `PartitionCommitter`. 2. InLong hive connector can automatically create tables by inferring table schema from source CDC data. We cache `HiveWriterFactory` object for each hive table. `HiveWriterFactory` has `HiveShim` object which can be used to create or alter hive table. `HiveWriterFactory` also has columns information of hive table. `HiveTableUtil` is a tool class to create or alter hive table with columns information. `sqlType` is from source CDC data with debezium format. `cdc-base` module has tools to change `sqlType` into Flink `RowField`. We should change Flink `RowField` into hive dialect types. Hive connector can create hive table with time partition. Now the default partition name is `pt`, you can change this name by setting `sink.partition.name`. Hive connector supports three partition policies, which are `PROC_TIME`, `ASSIGN_FIELD` and `NONE`. With `PROC_TIME` partition policy, we format current system time millis with `-MM-dd` time format and set it to `pt` partition. You can modify time format with `partition.time-extractor.timestamp-pattern` option. With `ASSIGN_FIELD` partition policy, it will read field value from source CDC data with `source.partition.field.name` field name, and format it with `partition.time-extractor.timestamp-pattern` time format. For example, mysql has a `create_time` field with `2023-05-25 18:00:00`, hive connector can read this field value and format it to `20230525` and write to HDFS in `hdfs:///database/table/pt=20230525` path. With `NONE` partition policy, hive connector create table without partitions. Refer to `iceberg-connector`, hive connector only supports to add new fields by inferring `sqlType` from source CDC data. It cannot delete fields because hive does not support it. Hive connector also does not support other DDL schema auto changing. 3. Before this PR, InLong hive connector only supports hive 3.x version, as it depends on hive-exe 3.1.1. Now we can package different jars using `mvn clean install -pl org.apache.inlong:sort-connector-hive -am -DskipTests -Phive2`. `-Phive2` represents a maven profile which imports hive-exec 2.2.0 dependency. If you want to package jar matching hive 3.x, just remove `-Phive2` in maven command. ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? -
[GitHub] [inlong] healchow commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
healchow commented on code in PR #8090:
URL: https://github.com/apache/inlong/pull/8090#discussion_r1205470771
##
inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongTenantEntityMapper.java:
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.dao.mapper;
+
+import org.apache.inlong.manager.dao.entity.InlongTenantEntity;
+
+public interface InlongTenantEntityMapper {
Review Comment:
Please add `@Repository` annotation for the Mapper interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
healchow commented on code in PR #8090:
URL: https://github.com/apache/inlong/pull/8090#discussion_r1205472115
##
inlong-manager/manager-test/src/main/resources/h2/apache_inlong_manager.sql:
##
@@ -791,5 +791,29 @@ VALUES ('audit_sdk_collect', 'SDK', 0, '1'),
('audit_sort_kudu_output', 'KUDU', 1, '26');
--
+-- Table structure for inlong_tenant
+--
+CREATE TABLE IF NOT EXISTS `inlong_tenant`
Review Comment:
Please add this table into the `manager-test` module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #8089: [INLONG-8088][Manager] Support configure heartbeat timeout interval
healchow commented on code in PR #8089: URL: https://github.com/apache/inlong/pull/8089#discussion_r1205474316 ## inlong-manager/manager-web/src/main/resources/application-dev.properties: ## @@ -86,6 +86,9 @@ data.cleansing.batchSize=100 # Whether to use ZooKeeper to manage the Sort task config, default is false, which means not using ZooKeeper sort.enable.zookeeper=false +# cluster node timeout interval of heartbeat unit: 5s +cluster.heartbeat.interval=30 + # If turned on, synchronizing change the source status when the agent heartbeat times out Review Comment: ```suggestion # cluster node timeout interval of heartbeat, unit: second cluster.heartbeat.interval=30 # If turned on, synchronizing change the source status when the agent heartbeat timeout ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #8089: [INLONG-8088][Manager] Support configure heartbeat timeout interval
fuweng11 commented on code in PR #8089: URL: https://github.com/apache/inlong/pull/8089#discussion_r1205483068 ## inlong-manager/manager-web/src/main/resources/application-dev.properties: ## @@ -86,6 +86,9 @@ data.cleansing.batchSize=100 # Whether to use ZooKeeper to manage the Sort task config, default is false, which means not using ZooKeeper sort.enable.zookeeper=false +# cluster node timeout interval of heartbeat unit: 5s +cluster.heartbeat.interval=30 + # If turned on, synchronizing change the source status when the agent heartbeat times out Review Comment: The interval multiplied by 5s indicates the true timeout interval, because 5s is the interval shared by the agent and dataproxy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
vernedeng commented on code in PR #8090:
URL: https://github.com/apache/inlong/pull/8090#discussion_r1205747104
##
inlong-manager/manager-test/src/main/resources/h2/apache_inlong_manager.sql:
##
@@ -791,5 +791,29 @@ VALUES ('audit_sdk_collect', 'SDK', 0, '1'),
('audit_sort_kudu_output', 'KUDU', 1, '26');
--
+-- Table structure for inlong_tenant
+--
+CREATE TABLE IF NOT EXISTS `inlong_tenant`
Review Comment:
I have added this table into the sql in h2 dictionary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
vernedeng commented on code in PR #8090:
URL: https://github.com/apache/inlong/pull/8090#discussion_r1205751988
##
inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/InlongTenantEntityMapper.java:
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.dao.mapper;
+
+import org.apache.inlong.manager.dao.entity.InlongTenantEntity;
+
+public interface InlongTenantEntityMapper {
Review Comment:
fixed, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] vernedeng opened a new pull request, #8097: [INLONG-8093][Manager] Add inlong tenant related APIs
vernedeng opened a new pull request, #8097: URL: https://github.com/apache/inlong/pull/8097 ### Prepare a Pull Request - Fixes #8093 ### Motivation *Explain here the context, and why you're making that change. What is the problem you're trying to solve?* ### Modifications *Describe the modifications you've done.* ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #8097: [INLONG-8093][Manager] Add inlong tenant related APIs
fuweng11 commented on code in PR #8097:
URL: https://github.com/apache/inlong/pull/8097#discussion_r1206138379
##
inlong-manager/manager-dao/src/main/resources/mappers/InlongTenantEntityMapper.xml:
##
@@ -0,0 +1,151 @@
+
+
+
+http://mybatis.org/dtd/mybatis-3-mapper.dtd";>
+
+
+
+
+
+
+
+
+
+
+
+
+
+id, name, description, is_deleted, creator, modifier, create_time,
modify_time, version
+
+
+
+insert into inlong_tenant (id, name, description, creator, modifier)
+values (#{id,jdbcType=INTEGER}, #{name,jdbcType=VARCHAR},
#{description,jdbcType=VARCHAR},
+#{creator,jdbcType=VARCHAR}, #{modifier,jdbcType=VARCHAR})
+
+
+insert into inlong_tenant
+
+
+id,
+
+
+name,
+
+
+description,
+
+
+is_deleted,
+
+
+creator,
+
+
+modifier,
+
+
+version,
+
+
+
+
+#{id,jdbcType=INTEGER},
+
+
+name,
Review Comment:
#{name,jdbcType=VARCHAR},
##
inlong-manager/manager-dao/src/main/resources/mappers/InlongTenantEntityMapper.xml:
##
@@ -0,0 +1,151 @@
+
+
+
+http://mybatis.org/dtd/mybatis-3-mapper.dtd";>
+
+
+
+
+
+
+
+
+
+
+
+
+
+id, name, description, is_deleted, creator, modifier, create_time,
modify_time, version
+
+
+
+insert into inlong_tenant (id, name, description, creator, modifier)
+values (#{id,jdbcType=INTEGER}, #{name,jdbcType=VARCHAR},
#{description,jdbcType=VARCHAR},
+#{creator,jdbcType=VARCHAR}, #{modifier,jdbcType=VARCHAR})
+
+
+insert into inlong_tenant
+
+
+id,
+
+
+name,
+
+
+description,
+
+
+is_deleted,
+
+
+creator,
+
+
+modifier,
+
+
+version,
+
+
+
+
+#{id,jdbcType=INTEGER},
+
+
+name,
+
+
+description,
+
+
+#{isDeleted,jdbcType=INTEGER},
+
+
+creator,
Review Comment:
#{creator,jdbcType=VARCHAR},
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/tenant/InlongTenantServiceImpl.java:
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.tenant;
+
+import com.github.pagehelper.Page;
+import com.github.pagehelper.PageHelper;
+import com.github.pagehelper.PageInfo;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.inlong.manager.common.consts.InlongConstants;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.CommonBeanUtils;
+import org.apache.inlong.manager.dao.entity.InlongTenantEntity;
+import org.apache.inlong.manager.dao.mapper.InlongTenantEntityMapper;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantInfo;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantPageRequest;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantRequest;
+import org.apache.inlong.manager.service.user.LoginUserUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+@Service
+@Slf4j
+public class InlongTenantServiceImpl implements InlongTenantService {
+
+@Autowired
+private InlongTenantEntityMapper inlongTenantEntityMapper;
+
+@Override
+public InlongTenantInfo get(String name) {
+InlongTenantEntity entity =
inlongTenantEntityMapper.selectByName(name);
+return CommonBeanUtils.copyProperties(entity, InlongTenantInfo::new);
+}
+
+@Override
+public Integer save(InlongTenantRequest request) {
+String name = request.getName();
+InlongTenantEntity existEntity =
inlongTenantEntityMapper.selectByName(name);
+if (existEntity != null) {
+String errMsg = String.format("tenant already exist for name=%s)",
name);
+log.error(errMsg);
+throw
[GitHub] [inlong] dockerzhang merged pull request #8081: [INLONG-8080][Manager] Fix the total parameter found on the page is different from the actual value
dockerzhang merged PR #8081: URL: https://github.com/apache/inlong/pull/8081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (2da89aa15 -> 954361733)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 2da89aa15 [INLONG-8068][Manager] Support repeatable read for http request (#8075) add 954361733 [INLONG-8080][Manager] Fix the total parameter found on the page is different from the actual value (#8081) No new revisions were added by this update. Summary of changes: .../apache/inlong/manager/service/sink/StreamSinkServiceImpl.java | 5 +++-- .../inlong/manager/service/source/StreamSourceServiceImpl.java| 8 2 files changed, 7 insertions(+), 6 deletions(-)
[GitHub] [inlong] vernedeng commented on a diff in pull request #8097: [INLONG-8093][Manager] Add inlong tenant related APIs
vernedeng commented on code in PR #8097:
URL: https://github.com/apache/inlong/pull/8097#discussion_r1206156863
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/tenant/InlongTenantServiceImpl.java:
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.tenant;
+
+import com.github.pagehelper.Page;
+import com.github.pagehelper.PageHelper;
+import com.github.pagehelper.PageInfo;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.inlong.manager.common.consts.InlongConstants;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.CommonBeanUtils;
+import org.apache.inlong.manager.dao.entity.InlongTenantEntity;
+import org.apache.inlong.manager.dao.mapper.InlongTenantEntityMapper;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantInfo;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantPageRequest;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantRequest;
+import org.apache.inlong.manager.service.user.LoginUserUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+@Service
+@Slf4j
+public class InlongTenantServiceImpl implements InlongTenantService {
+
+@Autowired
+private InlongTenantEntityMapper inlongTenantEntityMapper;
+
+@Override
+public InlongTenantInfo get(String name) {
+InlongTenantEntity entity =
inlongTenantEntityMapper.selectByName(name);
+return CommonBeanUtils.copyProperties(entity, InlongTenantInfo::new);
+}
+
+@Override
+public Integer save(InlongTenantRequest request) {
+String name = request.getName();
+InlongTenantEntity existEntity =
inlongTenantEntityMapper.selectByName(name);
+if (existEntity != null) {
+String errMsg = String.format("tenant already exist for name=%s)",
name);
+log.error(errMsg);
+throw new BusinessException(errMsg);
+}
+InlongTenantEntity entity = CommonBeanUtils.copyProperties(request,
InlongTenantEntity::new);
+String operator = LoginUserUtils.getLoginUser().getName();
+entity.setCreator(operator);
+entity.setModifier(operator);
+inlongTenantEntityMapper.insert(entity);
+return entity.getId();
+}
+
+@Override
+public PageInfo listByCondition(InlongTenantPageRequest
request) {
+PageHelper.startPage(request.getPageNum(), request.getPageSize());
+Page entityPage =
inlongTenantEntityMapper.selectByCondition(request);
+return entityPage
+.toPageInfo(entity -> CommonBeanUtils.copyProperties(entity,
InlongTenantInfo::new));
+}
+
+@Override
+public Boolean update(InlongTenantRequest request) {
+InlongTenantEntity exist =
inlongTenantEntityMapper.selectByName(request.getName());
Review Comment:
If somebody change the tenant name, all the related resources needs to be
changed also. Hence I don't think it's a good idea to allow tenant owner to
change tenant name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8097: [INLONG-8093][Manager] Add inlong tenant related APIs
vernedeng commented on code in PR #8097:
URL: https://github.com/apache/inlong/pull/8097#discussion_r1206156863
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/tenant/InlongTenantServiceImpl.java:
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.tenant;
+
+import com.github.pagehelper.Page;
+import com.github.pagehelper.PageHelper;
+import com.github.pagehelper.PageInfo;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.inlong.manager.common.consts.InlongConstants;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.CommonBeanUtils;
+import org.apache.inlong.manager.dao.entity.InlongTenantEntity;
+import org.apache.inlong.manager.dao.mapper.InlongTenantEntityMapper;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantInfo;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantPageRequest;
+import org.apache.inlong.manager.pojo.tenant.InlongTenantRequest;
+import org.apache.inlong.manager.service.user.LoginUserUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+@Service
+@Slf4j
+public class InlongTenantServiceImpl implements InlongTenantService {
+
+@Autowired
+private InlongTenantEntityMapper inlongTenantEntityMapper;
+
+@Override
+public InlongTenantInfo get(String name) {
+InlongTenantEntity entity =
inlongTenantEntityMapper.selectByName(name);
+return CommonBeanUtils.copyProperties(entity, InlongTenantInfo::new);
+}
+
+@Override
+public Integer save(InlongTenantRequest request) {
+String name = request.getName();
+InlongTenantEntity existEntity =
inlongTenantEntityMapper.selectByName(name);
+if (existEntity != null) {
+String errMsg = String.format("tenant already exist for name=%s)",
name);
+log.error(errMsg);
+throw new BusinessException(errMsg);
+}
+InlongTenantEntity entity = CommonBeanUtils.copyProperties(request,
InlongTenantEntity::new);
+String operator = LoginUserUtils.getLoginUser().getName();
+entity.setCreator(operator);
+entity.setModifier(operator);
+inlongTenantEntityMapper.insert(entity);
+return entity.getId();
+}
+
+@Override
+public PageInfo listByCondition(InlongTenantPageRequest
request) {
+PageHelper.startPage(request.getPageNum(), request.getPageSize());
+Page entityPage =
inlongTenantEntityMapper.selectByCondition(request);
+return entityPage
+.toPageInfo(entity -> CommonBeanUtils.copyProperties(entity,
InlongTenantInfo::new));
+}
+
+@Override
+public Boolean update(InlongTenantRequest request) {
+InlongTenantEntity exist =
inlongTenantEntityMapper.selectByName(request.getName());
Review Comment:
If somebody changes the tenant name, all the related resources need to be
changed also. Hence I don't think it's a good idea to allow tenant owner to
change tenant name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8089: [INLONG-8088][Manager] Support configure heartbeat timeout interval
vernedeng commented on code in PR #8089:
URL: https://github.com/apache/inlong/pull/8089#discussion_r1206158061
##
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/heartbeat/HeartbeatManager.java:
##
@@ -106,7 +110,7 @@ public void init() {
// When the manager restarts, set the heartbeat timeout state of all
nodes
// and wait for the heartbeat report of the corresponding node
clusterNodeMapper.updateStatus(null,
NodeStatus.HEARTBEAT_TIMEOUT.getStatus(), NodeStatus.NORMAL.getStatus());
-long expireTime = heartbeatInterval() * 2L;
+long expireTime = heartbeatInterval() * heartbeatInterval;
Review Comment:
heartbeatInterval -> heartbeatIntervalFactor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
dockerzhang commented on code in PR #8090: URL: https://github.com/apache/inlong/pull/8090#discussion_r1206160454 ## inlong-manager/manager-web/sql/changes-1.8.0.sql: ## @@ -25,3 +25,22 @@ USE `apache_inlong_manager`; ALTER TABLE inlong_group CHANGE lightweight inlong_group_mode tinyint(1) DEFAULT 0 NULL COMMENT 'Inlong group mode, Standard mode: 0, DataSync mode: 1'; + Review Comment: It's better to add more annotations to show the purpose of this change. Like `For support multi-tenant management in InLong, see https://github.com/apache/inlong/pull/8090`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
vernedeng commented on code in PR #8090: URL: https://github.com/apache/inlong/pull/8090#discussion_r1206162470 ## inlong-manager/manager-web/sql/changes-1.8.0.sql: ## @@ -25,3 +25,22 @@ USE `apache_inlong_manager`; ALTER TABLE inlong_group CHANGE lightweight inlong_group_mode tinyint(1) DEFAULT 0 NULL COMMENT 'Inlong group mode, Standard mode: 0, DataSync mode: 1'; + Review Comment: fixed, thx. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #8095: [INLONG-7959][Sort] Dynamic schema evolution support delete and update columns when sink to Iceberg
dockerzhang commented on PR #8095: URL: https://github.com/apache/inlong/pull/8095#issuecomment-1563727579 @hejiay thanks for your contribution, there is a file missing the license header, you could execute `mvn spotless:apply` to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] hejiay commented on pull request #8095: [INLONG-7959][Sort] Dynamic schema evolution support delete and update columns when sink to Iceberg
hejiay commented on PR #8095: URL: https://github.com/apache/inlong/pull/8095#issuecomment-1563737733 @dockerzhang please check again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8089: [INLONG-8088][Manager] Support configure heartbeat timeout interval
dockerzhang merged PR #8089: URL: https://github.com/apache/inlong/pull/8089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (954361733 -> 9d66ab2ac)
This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/inlong.git from 954361733 [INLONG-8080][Manager] Fix the total parameter found on the page is different from the actual value (#8081) add 9d66ab2ac [INLONG-8088][Manager] Support configure heartbeat timeout interval (#8089) No new revisions were added by this update. Summary of changes: .../apache/inlong/manager/service/heartbeat/HeartbeatManager.java | 6 +- .../manager-web/src/main/resources/application-dev.properties | 3 +++ .../manager-web/src/main/resources/application-prod.properties | 3 +++ .../manager-web/src/main/resources/application-test.properties | 3 +++ 4 files changed, 14 insertions(+), 1 deletion(-)
[GitHub] [inlong] github-code-scanning[bot] commented on a diff in pull request #8096: [INLONG-8092][Sort] Support all database and multiple tables transmission for Hive
github-code-scanning[bot] commented on code in PR #8096:
URL: https://github.com/apache/inlong/pull/8096#discussion_r1206189684
##
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/hive/src/main/java/org/apache/inlong/sort/hive/filesystem/HadoopPathBasedPartFileWriter.java:
##
@@ -0,0 +1,788 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.hive.filesystem;
+
+import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_DATABASE_PATTERN;
+import static org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_ENABLE;
+import static org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_FORMAT;
+import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_TABLE_PATTERN;
+import static
org.apache.inlong.sort.hive.HiveOptions.HIVE_SCHEMA_SCAN_INTERVAL;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.ImmutableTriple;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.connectors.hive.FlinkHiveException;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.formats.hadoop.bulk.HadoopFileCommitter;
+import org.apache.flink.formats.hadoop.bulk.HadoopFileCommitterFactory;
+import org.apache.flink.formats.hadoop.bulk.HadoopPathBasedBulkWriter;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.AbstractPartFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.WriterProperties;
+
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.inlong.sort.base.dirty.DirtyOptions;
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.format.DynamicSchemaFormatFactory;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.PartitionPolicy;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.hive.HiveBulkWriterFactory;
+import org.apache.inlong.sort.hive.util.CacheHolder;
+import org.apache.inlong.sort.hive.util.HiveTableUtil;
+import org.apache.inlong.sort.hive.HiveWriterFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The part-file writer that writes to the specified hadoop path.
+ */
+public class HadoopPathBasedPartFileWriter extends
AbstractPartFileWriter {
+
+private static final Logger LOG =
LoggerFactory.getLogger(HadoopPathBasedPartFileWriter.class);
+
+private final InLongHadoopPathBasedBulkWriter writer;
+
+private final HadoopFileCommitter fileCommitter;
+
+private BucketID bucketID;
+private Path targetPath;
+private Path inProgressPath;
+private final HiveShim hiveShim;
+pri
[GitHub] [inlong] dockerzhang merged pull request #8090: [INLONG-8087][Manager] Add definition of Inlong tenant table
dockerzhang merged PR #8090: URL: https://github.com/apache/inlong/pull/8090 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (9d66ab2ac -> e4acb035f)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
from 9d66ab2ac [INLONG-8088][Manager] Support configure heartbeat timeout
interval (#8089)
add e4acb035f [INLONG-8087][Manager] Add definition of Inlong tenant table
(#8090)
No new revisions were added by this update.
Summary of changes:
.../{RoleEntity.java => InlongTenantEntity.java} | 13 +-
...tyMapper.java => InlongTenantEntityMapper.java} | 15 ++-
.../src/main/resources/generatorConfig.xml | 2 +-
.../resources/mappers/InlongTenantEntityMapper.xml | 134 +
.../TenantInfo.java} | 31 ++---
.../TenantPageRequest.java}| 27 ++---
.../TenantRequest.java}| 42 +++
.../main/resources/h2/apache_inlong_manager.sql| 24
.../manager-web/sql/apache_inlong_manager.sql | 25
inlong-manager/manager-web/sql/changes-1.8.0.sql | 20 +++
10 files changed, 254 insertions(+), 79 deletions(-)
copy
inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/{RoleEntity.java
=> InlongTenantEntity.java} (85%)
copy
inlong-manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/mapper/{RoleEntityMapper.java
=> InlongTenantEntityMapper.java} (74%)
create mode 100644
inlong-manager/manager-dao/src/main/resources/mappers/InlongTenantEntityMapper.xml
copy
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/{cluster/ClusterTagResponse.java
=> tenant/TenantInfo.java} (73%)
copy
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/{cluster/ClusterTagPageRequest.java
=> tenant/TenantPageRequest.java} (72%)
copy
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/{group/InlongGroupExtInfo.java
=> tenant/TenantRequest.java} (56%)
[GitHub] [inlong] vernedeng closed pull request #5873: [INLONG-5839][Sort] Add SortStandalone option in start script
vernedeng closed pull request #5873: [INLONG-5839][Sort] Add SortStandalone option in start script URL: https://github.com/apache/inlong/pull/5873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] liaorui commented on a diff in pull request #8096: [INLONG-8092][Sort] Support all database and multiple tables transmission for Hive
liaorui commented on code in PR #8096:
URL: https://github.com/apache/inlong/pull/8096#discussion_r1206201379
##
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/hive/src/main/java/org/apache/inlong/sort/hive/filesystem/HadoopPathBasedPartFileWriter.java:
##
@@ -0,0 +1,788 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.hive.filesystem;
+
+import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_DATABASE_PATTERN;
+import static org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_ENABLE;
+import static org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_FORMAT;
+import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_TABLE_PATTERN;
+import static
org.apache.inlong.sort.hive.HiveOptions.HIVE_SCHEMA_SCAN_INTERVAL;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.ImmutableTriple;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.connectors.hive.FlinkHiveException;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.formats.hadoop.bulk.HadoopFileCommitter;
+import org.apache.flink.formats.hadoop.bulk.HadoopFileCommitterFactory;
+import org.apache.flink.formats.hadoop.bulk.HadoopPathBasedBulkWriter;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.AbstractPartFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.WriterProperties;
+
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.catalog.hive.client.HiveShim;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.inlong.sort.base.dirty.DirtyOptions;
+import org.apache.inlong.sort.base.dirty.DirtySinkHelper;
+import org.apache.inlong.sort.base.dirty.DirtyType;
+import org.apache.inlong.sort.base.dirty.sink.DirtySink;
+import org.apache.inlong.sort.base.format.DynamicSchemaFormatFactory;
+import org.apache.inlong.sort.base.format.JsonDynamicSchemaFormat;
+import org.apache.inlong.sort.base.metric.sub.SinkTableMetricData;
+import org.apache.inlong.sort.base.sink.PartitionPolicy;
+import org.apache.inlong.sort.base.sink.SchemaUpdateExceptionPolicy;
+import org.apache.inlong.sort.hive.HiveBulkWriterFactory;
+import org.apache.inlong.sort.hive.util.CacheHolder;
+import org.apache.inlong.sort.hive.util.HiveTableUtil;
+import org.apache.inlong.sort.hive.HiveWriterFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The part-file writer that writes to the specified hadoop path.
+ */
+public class HadoopPathBasedPartFileWriter extends
AbstractPartFileWriter {
+
+private static final Logger LOG =
LoggerFactory.getLogger(HadoopPathBasedPartFileWriter.class);
+
+private final InLongHadoopPathBasedBulkWriter writer;
+
+private final HadoopFileCommitter fileCommitter;
+
+private BucketID bucketID;
+private Path targetPath;
+private Path inProgressPath;
+private final HiveShim hiveShim;
+private final String
[GitHub] [inlong-website] hnrainll opened a new issue, #762: [Bug][Doc]
hnrainll opened a new issue, #762: URL: https://github.com/apache/inlong-website/issues/762 ### What happened In the link https://inlong.apache.org/zh-CN/docs/modules/manager/quick_start/ the **Connector Binary file** is located at https://inlong.apache.org/download/main, but the page cannot be found.   ### What you expected to happen Hopefully it will jump to the correct address ### How to reproduce Hopefully it will jump to the correct address ### Environment _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] hnrainll opened a new pull request, #763: [INLONG-762][Doc] Connector Binary file link address error
hnrainll opened a new pull request, #763: URL: https://github.com/apache/inlong-website/pull/763 ### Prepare a Pull Request - Fixes #762 ### Motivation Hopefully it will jump to the correct address ### Modifications Change https://inlong.apache.org/download/main to https://inlong.apache.org/download -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8097: [INLONG-8093][Manager] Add inlong tenant related APIs
dockerzhang merged PR #8097: URL: https://github.com/apache/inlong/pull/8097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated (e4acb035f -> d2c900805)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
from e4acb035f [INLONG-8087][Manager] Add definition of Inlong tenant table
(#8090)
add d2c900805 [INLONG-8093][Manager] Add inlong tenant related APIs (#8097)
No new revisions were added by this update.
Summary of changes:
.../dao/mapper/InlongTenantEntityMapper.java | 6 ++
.../resources/mappers/InlongTenantEntityMapper.xml | 52
.../{TenantInfo.java => InlongTenantInfo.java} | 7 +-
...geRequest.java => InlongTenantPageRequest.java} | 11 ++-
...TenantRequest.java => InlongTenantRequest.java} | 4 +-
.../service/tenant/InlongTenantService.java| 58 +
.../service/tenant/InlongTenantServiceImpl.java| 99 ++
.../service/tenant/InlongTenantServiceTest.java| 98 +
.../web/controller/InlongTenantController.java | 75
9 files changed, 352 insertions(+), 58 deletions(-)
rename
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/tenant/{TenantInfo.java
=> InlongTenantInfo.java} (89%)
rename
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/tenant/{TenantPageRequest.java
=> InlongTenantPageRequest.java} (85%)
rename
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/tenant/{TenantRequest.java
=> InlongTenantRequest.java} (95%)
copy
inlong-sort-standalone/sort-standalone-common/src/main/java/org/apache/inlong/sort/standalone/config/pojo/type/CacheType.java
=>
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/tenant/InlongTenantService.java
(53%)
create mode 100644
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/tenant/InlongTenantServiceImpl.java
create mode 100644
inlong-manager/manager-service/src/test/java/org/apache/inlong/manager/service/tenant/InlongTenantServiceTest.java
create mode 100644
inlong-manager/manager-web/src/main/java/org/apache/inlong/manager/web/controller/InlongTenantController.java
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #763: [INLONG-762][Doc] Connector Binary file link address error
dockerzhang commented on code in PR #763: URL: https://github.com/apache/inlong-website/pull/763#discussion_r1206274958 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md: ## @@ -15,7 +15,7 @@ sidebar_position: 2 ``` ## 添加依赖 -- 下载 Sort [Connector Binary file](https://inlong.apache.org/download/main), 并解压放入`inlong-sort/connectors`目录。 +- 下载 Sort [Connector Binary file](https://inlong.apache.org/download), 并解压放入`inlong-sort/connectors`目录。 Review Comment: Here should be https://inlong.apache.org/zh-CN/download/ for Chinese doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] hnrainll commented on a diff in pull request #763: [INLONG-762][Doc] Connector Binary file link address error
hnrainll commented on code in PR #763: URL: https://github.com/apache/inlong-website/pull/763#discussion_r1206288737 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md: ## @@ -15,7 +15,7 @@ sidebar_position: 2 ``` ## 添加依赖 -- 下载 Sort [Connector Binary file](https://inlong.apache.org/download/main), 并解压放入`inlong-sort/connectors`目录。 +- 下载 Sort [Connector Binary file](https://inlong.apache.org/download), 并解压放入`inlong-sort/connectors`目录。 Review Comment: Ok, I have fixed this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong-website] hnrainll commented on a diff in pull request #763: [INLONG-762][Doc] Connector Binary file link address error
hnrainll commented on code in PR #763: URL: https://github.com/apache/inlong-website/pull/763#discussion_r1206289675 ## i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md: ## @@ -15,7 +15,7 @@ sidebar_position: 2 ``` ## 添加依赖 -- 下载 Sort [Connector Binary file](https://inlong.apache.org/download/main), 并解压放入`inlong-sort/connectors`目录。 +- 下载 Sort [Connector Binary file](https://inlong.apache.org/download), 并解压放入`inlong-sort/connectors`目录。 Review Comment: I have also modified the links to other Chinese versions as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
