[PR] [INLONG-9318][Manager] ManagerClient supports pulling clusters based on tenant roles [inlong]
fuweng11 opened a new pull request, #9319: URL: https://github.com/apache/inlong/pull/9319 ### Prepare a Pull Request - Fixes #9318 ### Motivation ManagerClient supports pulling clusters based on tenant roles. ### Modifications ManagerClient supports pulling clusters based on tenant roles.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-9318][Manager] ManagerClient supports pulling clusters based on tenant roles [inlong]
EMsnap merged PR #9319: URL: https://github.com/apache/inlong/pull/9319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9318][Manager] ManagerClient supports pulling clusters based on tenant roles (#9319)
This is an automated email from the ASF dual-hosted git repository.
zirui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 64f84becd8 [INLONG-9318][Manager] ManagerClient supports pulling
clusters based on tenant roles (#9319)
64f84becd8 is described below
commit 64f84becd8ed04d754d6b26ee25708ec03b2dd45
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Wed Nov 22 17:40:04 2023 +0800
[INLONG-9318][Manager] ManagerClient supports pulling clusters based on
tenant roles (#9319)
---
.../client/api/inner/client/InlongClusterClient.java| 13 +
.../inlong/manager/client/api/service/InlongClusterApi.java | 3 +++
2 files changed, 16 insertions(+)
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongClusterClient.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongClusterClient.java
index ccae50d6cc..a4558b43ad 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongClusterClient.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/inner/client/InlongClusterClient.java
@@ -154,6 +154,19 @@ public class InlongClusterClient {
return response.getData();
}
+/**
+ * Paging query clusters according to tenant role.
+ *
+ * @param request query conditions
+ * @return cluster list
+ */
+public PageResult listByTenantRole(ClusterPageRequest
request) {
+Response> response =
+
ClientUtils.executeHttpCall(inlongClusterApi.listByTenantRole(request));
+ClientUtils.assertRespSuccess(response);
+return response.getData();
+}
+
/**
* Update cluster information.
*
diff --git
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/service/InlongClusterApi.java
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/service/InlongClusterApi.java
index a83a6a0f2a..5be602cc3b 100644
---
a/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/service/InlongClusterApi.java
+++
b/inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/service/InlongClusterApi.java
@@ -66,6 +66,9 @@ public interface InlongClusterApi {
@POST("cluster/list")
Call>> list(@Body ClusterPageRequest
request);
+@POST("cluster/listByTenantRole")
+Call>> listByTenantRole(@Body
ClusterPageRequest request);
+
@POST("cluster/update")
Call> update(@Body ClusterRequest request);
[PR] [INLONG-9320][SDK] Support local disaster recovery manager configuration [inlong]
doleyzi opened a new pull request, #9321: URL: https://github.com/apache/inlong/pull/9321 ### Prepare a Pull Request - Fixes #9320 ### Motivation *Support local disaster recovery manager configuration* ### Modifications *Support local disaster recovery manager configuration.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] fix mysql-cdc and iceberg node doc [inlong-website]
EMsnap opened a new issue, #893: URL: https://github.com/apache/inlong-website/issues/893 ### What happened [Bug] fix mysql-cdc and iceberg node doc ### What you expected to happen [Bug] fix mysql-cdc and iceberg node doc ### How to reproduce [Bug] fix mysql-cdc and iceberg node doc ### Environment _No response_ ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-893][Doc] Fix mysql-cdc and iceberg node doc [inlong-website]
EMsnap opened a new pull request, #894: URL: https://github.com/apache/inlong-website/pull/894 ### Prepare a Pull Request - Fixes #893 ### Motivation Fix mysql-cdc and iceberg node doc ### Modifications Fix mysql-cdc and iceberg node doc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-9320][SDK] Support local disaster recovery manager configuration [inlong]
dockerzhang merged PR #9321: URL: https://github.com/apache/inlong/pull/9321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-9320][SDK] Support local disaster recovery manager configuration (#9321)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 08a974625c [INLONG-9320][SDK] Support local disaster recovery manager
configuration (#9321)
08a974625c is described below
commit 08a974625cd025559002a1e27b6c4d9ce5f54e45
Author: doleyzi <43397300+dole...@users.noreply.github.com>
AuthorDate: Thu Nov 23 10:31:12 2023 +0800
[INLONG-9320][SDK] Support local disaster recovery manager configuration
(#9321)
---
.../dataproxy-sdk-cpp/release/inc/sdk_conf.h | 1 +
.../dataproxy-sdk-cpp/src/config/sdk_conf.cc | 9 ++
.../dataproxy-sdk-cpp/src/core/api_imp.cc | 1 +
.../dataproxy-sdk-cpp/src/manager/proxy_manager.cc | 98 ++
.../dataproxy-sdk-cpp/src/manager/proxy_manager.h | 5 ++
.../dataproxy-sdk-cpp/src/utils/capi_constant.h| 4 +
6 files changed, 118 insertions(+)
diff --git
a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/inc/sdk_conf.h
b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/inc/sdk_conf.h
index 9df35d85a8..aa451caee9 100644
--- a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/inc/sdk_conf.h
+++ b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/inc/sdk_conf.h
@@ -90,6 +90,7 @@ private:
uint64_t tcp_idle_time_; // The time when tcpclient did not send
data
uint32_t tcp_detection_interval_; // tcp-client detection interval
bool enable_balance_;
+ bool enable_local_cache_;
// auth settings
bool need_auth_;
diff --git
a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/config/sdk_conf.cc
b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/config/sdk_conf.cc
index c2bf211b69..70cd2362e2 100644
--- a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/config/sdk_conf.cc
+++ b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/config/sdk_conf.cc
@@ -115,6 +115,7 @@ void SdkConfig::defaultInit() {
max_proxy_num_ = constants::kMaxProxyNum;
enable_isolation_ = constants::kEnableIsolation;
reserve_proxy_num_ = constants::kReserveProxyNum;
+ enable_local_cache_ = constants::kEnableLocalCache;
local_ip_ = constants::kSerIP;
local_port_ = constants::kSerPort;
@@ -359,6 +360,14 @@ void SdkConfig::InitManagerParam(const rapidjson::Value
&doc) {
} else {
enable_isolation_ = constants::kEnableIsolation;
}
+
+ // enable local cache
+ if (doc.HasMember("enable_local_cache") &&
doc["enable_local_cache"].IsBool()) {
+const rapidjson::Value &obj = doc["enable_local_cache"];
+enable_local_cache_ = obj.GetBool();
+ } else {
+enable_local_cache_ = constants::kEnableLocalCache;
+ }
}
void SdkConfig::InitTcpParam(const rapidjson::Value &doc) {
diff --git
a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/core/api_imp.cc
b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/core/api_imp.cc
index f058142841..79706a511e 100644
--- a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/core/api_imp.cc
+++ b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/core/api_imp.cc
@@ -104,6 +104,7 @@ int32_t ApiImp::DoInit() {
LOG_INFO("inlong dataproxy cpp sdk Init complete!");
ProxyManager::GetInstance()->Init();
+ ProxyManager::GetInstance()->ReadLocalCache();
for (int i = 0; i < SdkConfig::getInstance()->inlong_group_ids_.size(); i++)
{
LOG_INFO("DoInit CheckConf inlong_group_id:"
diff --git
a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/manager/proxy_manager.cc
b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/manager/proxy_manager.cc
index 084905e2c9..1c5f440119 100644
---
a/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/manager/proxy_manager.cc
+++
b/inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/manager/proxy_manager.cc
@@ -18,6 +18,7 @@
*/
#include "proxy_manager.h"
+#include "../config/ini_help.h"
#include "../utils/capi_constant.h"
#include "../utils/logger.h"
#include "../utils/utils.h"
@@ -107,6 +108,23 @@ void ProxyManager::DoUpdate() {
break;
} // request success
}
+
+ if (ret != SdkCode::kSuccess) {
+if (groupid_2_proxy_map_.find(groupid2cluster.first) !=
groupid_2_proxy_map_.end()) {
+ LOG_WARN("failed to request from manager, use previous " <<
groupid2cluster.first);
+ continue;
+}
+if (!SdkConfig::getInstance()->enable_local_cache_) {
+ LOG_WARN("failed to request from manager, forbid local cache!");
+ continue;
+}
+meta_data = RecoverFromLocalCache(groupid2cluster.first);
+if (meta_data.empty()) {
+ LOG_WARN("local cache is empty!");
+ continue;
+}
+ }
+
ProxyInfoVec proxyInfoVec;
ret = ParseAndGet(groupid2cluster.first, meta_data, proxyInfoVec);
if (ret != SdkCode::kSuccess) {
@@ -117,6 +135,7 @@ vo
[PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
EMsnap opened a new pull request, #9323: URL: https://github.com/apache/inlong/pull/9323 ### Prepare a Pull Request - Fixes #9205 ### Motivation Fix Hudi connector primary key doesn't exist error ### Modifications Fix Hudi connector primary key doesn't exist error ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-9324][SDK] Supports automatic creation of custom log directories [inlong]
EMsnap commented on code in PR #9325:
URL: https://github.com/apache/inlong/pull/9325#discussion_r1403001210
##
inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/utils/logger.h:
##
@@ -22,21 +22,42 @@
#define INLONG_SDK_LOGGER_H
#include "sdk_conf.h"
+#include
#include
#include
#include
#include
#include
#include
+#include
using namespace log4cplus::helpers;
namespace inlong {
+static const char kDefaultPath[] = "./";
+static const char kLogName[] = "/inlong-sdk.log";
+static bool CheckPath(const std::string &path) {
+ struct stat st_stat = {0};
+ int ret = stat(path.c_str(), &st_stat);
+ if (ret && errno != ENOENT) {
+std::cout << "Check directory error:" << strerror(errno) << std::endl;
+return false;
+ }
+
+ if ((ret && errno == ENOENT) || (!ret && !S_ISDIR(st_stat.st_mode))) {
+if (mkdir(path.c_str(), S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH)) {
+ std::cout << "Crate directory error:" << strerror(errno) << std::endl;
+ return false;
+}
+ }
+ return true;
+}
+
static void initLog4cplus() {
std::string log_path = SdkConfig::getInstance()->log_path_;
- if (log_path.back() != '/') {
-log_path = "./inlong-sdk-logs/";
+ if (!CheckPath(log_path)) {
Review Comment:
method naming style is not consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9324][SDK] Supports automatic creation of custom log directories [inlong]
EMsnap commented on code in PR #9325:
URL: https://github.com/apache/inlong/pull/9325#discussion_r1403001210
##
inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/src/utils/logger.h:
##
@@ -22,21 +22,42 @@
#define INLONG_SDK_LOGGER_H
#include "sdk_conf.h"
+#include
#include
#include
#include
#include
#include
#include
+#include
using namespace log4cplus::helpers;
namespace inlong {
+static const char kDefaultPath[] = "./";
+static const char kLogName[] = "/inlong-sdk.log";
+static bool CheckPath(const std::string &path) {
+ struct stat st_stat = {0};
+ int ret = stat(path.c_str(), &st_stat);
+ if (ret && errno != ENOENT) {
+std::cout << "Check directory error:" << strerror(errno) << std::endl;
+return false;
+ }
+
+ if ((ret && errno == ENOENT) || (!ret && !S_ISDIR(st_stat.st_mode))) {
+if (mkdir(path.c_str(), S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH)) {
+ std::cout << "Crate directory error:" << strerror(errno) << std::endl;
+ return false;
+}
+ }
+ return true;
+}
+
static void initLog4cplus() {
std::string log_path = SdkConfig::getInstance()->log_path_;
- if (log_path.back() != '/') {
-log_path = "./inlong-sdk-logs/";
+ if (!CheckPath(log_path)) {
Review Comment:
method naming style is not consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
e-mhui commented on code in PR #9323:
URL: https://github.com/apache/inlong/pull/9323#discussion_r1403009486
##
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HudiLoadNode.java:
##
@@ -160,7 +160,9 @@ public Map tableOptions() {
options.put(HUDI_OPTION_DATABASE_NAME, dbName);
options.put(HUDI_OPTION_TABLE_NAME, tableName);
-options.put(HUDI_OPTION_RECORD_KEY_FIELD_NAME, primaryKey);
+if (StringUtils.isNotEmpty(primaryKey)) {
Review Comment:
Using `isNotEmpty` without considering the situation of '', using
`isNotBlank` may be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
e-mhui commented on code in PR #9323:
URL: https://github.com/apache/inlong/pull/9323#discussion_r1403009486
##
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HudiLoadNode.java:
##
@@ -160,7 +160,9 @@ public Map tableOptions() {
options.put(HUDI_OPTION_DATABASE_NAME, dbName);
options.put(HUDI_OPTION_TABLE_NAME, tableName);
-options.put(HUDI_OPTION_RECORD_KEY_FIELD_NAME, primaryKey);
+if (StringUtils.isNotEmpty(primaryKey)) {
Review Comment:
Using `isNotEmpty` without considering the situation of ' ', using
`isNotBlank` may be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
e-mhui commented on code in PR #9323:
URL: https://github.com/apache/inlong/pull/9323#discussion_r1403009486
##
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HudiLoadNode.java:
##
@@ -160,7 +160,9 @@ public Map tableOptions() {
options.put(HUDI_OPTION_DATABASE_NAME, dbName);
options.put(HUDI_OPTION_TABLE_NAME, tableName);
-options.put(HUDI_OPTION_RECORD_KEY_FIELD_NAME, primaryKey);
+if (StringUtils.isNotEmpty(primaryKey)) {
Review Comment:
Using `isNotEmpty` without considering the situation of ' ', using
`isNotBlank` may be better.
![Uploading image.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
e-mhui commented on code in PR #9323:
URL: https://github.com/apache/inlong/pull/9323#discussion_r1403009486
##
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HudiLoadNode.java:
##
@@ -160,7 +160,9 @@ public Map tableOptions() {
options.put(HUDI_OPTION_DATABASE_NAME, dbName);
options.put(HUDI_OPTION_TABLE_NAME, tableName);
-options.put(HUDI_OPTION_RECORD_KEY_FIELD_NAME, primaryKey);
+if (StringUtils.isNotEmpty(primaryKey)) {
Review Comment:
Using `isNotEmpty` without considering the situation of ' ', using
`isNotBlank` may be better.
```java
System.out.println(StringUtils.isNotBlank(" ")); // output: false
System.out.println(StringUtils.isNotEmpty(" ")) // output: true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-9205][Sort] Fix Hudi connector primary key doesn't exist error [inlong]
e-mhui commented on code in PR #9323:
URL: https://github.com/apache/inlong/pull/9323#discussion_r1403009486
##
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/HudiLoadNode.java:
##
@@ -160,7 +160,9 @@ public Map tableOptions() {
options.put(HUDI_OPTION_DATABASE_NAME, dbName);
options.put(HUDI_OPTION_TABLE_NAME, tableName);
-options.put(HUDI_OPTION_RECORD_KEY_FIELD_NAME, primaryKey);
+if (StringUtils.isNotEmpty(primaryKey)) {
Review Comment:
Using `isNotEmpty` without considering the situation of " ", using
`isNotBlank` may be better.
```java
System.out.println(StringUtils.isNotBlank(" ")); // output: false
System.out.println(StringUtils.isNotEmpty(" ")) // output: true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
