[PR] [INLONG-10979][SDK] Transform support PI() function [inlong]
emptyOVO opened a new pull request, #10983: URL: https://github.com/apache/inlong/pull/10983 Fixes #10979 ### Motivation add PiFunction and provide unit test ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10982][SDK] Improve the test code structure [inlong]
ying-hua opened a new pull request, #10985: URL: https://github.com/apache/inlong/pull/10985 Fixes #10982 ### Motivation To improve usability, I have modified the getFieldList function in TestTransformProcess.java. The function can get some parameters to return diffirent FieldList now. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10984][SDK] Transform support RADIANS(x) function [inlong]
emptyOVO opened a new pull request, #10987: URL: https://github.com/apache/inlong/pull/10987 Fixes #10984 ### Motivation add Radiansfunction class and provide unit test ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10988][Manager] Data preview filters data in tubes based on streamId [inlong]
fuweng11 opened a new pull request, #10989: URL: https://github.com/apache/inlong/pull/10989 Fixes #10988 ### Motivation Data preview filters data in tubes based on streamId. ### Modifications Data preview filters data in tubes based on streamId. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10956][SDK] Add XML formatted data source for Transform [inlong]
Zkplo opened a new pull request, #10990: URL: https://github.com/apache/inlong/pull/10990 Fixes #10956 ### Motivation Add several classes: XmlSourceInfo, XmlNode, XmlSourceData, XmlSourceDecoder. Modified SourceDecoderFactory class. Also, add the corresponding unit test codes. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10988][Manager] Data preview filters data in tubes based on streamId [inlong]
aloyszhang merged PR #10989: URL: https://github.com/apache/inlong/pull/10989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10988][Manager] Data preview filters data in tubes based on streamId (#10989)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c4bf54bbb1 [INLONG-10988][Manager] Data preview filters data in tubes
based on streamId (#10989)
c4bf54bbb1 is described below
commit c4bf54bbb1bb0d83429c1aa9b93ef721bf376e90
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Mon Sep 2 20:24:25 2024 +0800
[INLONG-10988][Manager] Data preview filters data in tubes based on
streamId (#10989)
---
.../inlong/manager/service/resource/queue/tubemq/TubeMQOperator.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/tubemq/TubeMQOperator.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/tubemq/TubeMQOperator.java
index 050de07805..66d1fcf59a 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/tubemq/TubeMQOperator.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/queue/tubemq/TubeMQOperator.java
@@ -71,6 +71,7 @@ public class TubeMQOperator {
private static final String CREATE_USER = "&createUser=";
private static final String CONF_MOD_AUTH_TOKEN = "&confModAuthToken=";
private static final String MSG_COUNT = "&msgCount=";
+private static final String FILTER_CONDS = "&filterConds=";
private static final String QUERY_TOPIC_PATH =
"/webapi.htm?method=admin_query_cluster_topic_view";
private static final String QUERY_BROKER_PATH =
"/webapi.htm?method=admin_query_broker_run_status";
@@ -288,7 +289,7 @@ public class TubeMQOperator {
}
String url = "http://"; + brokerUrl + QUERY_MESSAGE_PATH +
TOPIC_NAME + topicName + MSG_COUNT
-+ request.getMessageCount();
++ request.getMessageCount() + FILTER_CONDS +
streamInfo.getInlongStreamId();
TubeMessageResponse response = HttpUtils.request(restTemplate,
url, HttpMethod.GET,
null, new HttpHeaders(), TubeMessageResponse.class);
if (response.getErrCode() != SUCCESS_CODE && response.getErrCode()
!= 200) {
Re: [PR] [INLONG-10940][SDK] Transform SQL support arithmetic functions(Including cot, tanh, cosh, asin, atan and atan2) [inlong]
MOONSakura0614 commented on code in PR #10950:
URL: https://github.com/apache/inlong/pull/10950#discussion_r1740871446
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Atan2Function.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+import java.math.BigDecimal;
+
+/**
+ * Atan2Function
+ * description: atan2(numeric)--returns the arc tangent of a coordinate
(numeric1, numeric2).
+ */
+@TransformFunction(names = {"atan2"})
+public class Atan2Function implements ValueParser {
+
+private ValueParser xParser;
+private ValueParser yParser;
+
+public Atan2Function(Function expr) {
+xParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+yParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object xObj = xParser.parse(sourceData, rowIndex, context);
+Object yObj = yParser.parse(sourceData, rowIndex, context);
+
+BigDecimal xValue = OperatorTools.parseBigDecimal(xObj);
Review Comment:
Thank you for your mindful review! Here is my little confuse:
What action should happen if null pointer been checked? Throw exceptions
like IllegalArgumentException or ArithmeticException, Or just return null as
result?
And why only these 2 methods need null pointer exception check？I know that
Java Math Functions need params ( not null, otherwise would cause compile
error) in raw data type which can transfer to DOUBLE, then dose other
trigonometric function also need check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10940][SDK] Transform SQL support arithmetic functions(Including cot, tanh, cosh, asin, atan and atan2) [inlong]
MOONSakura0614 commented on code in PR #10950:
URL: https://github.com/apache/inlong/pull/10950#discussion_r1740871446
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Atan2Function.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+import java.math.BigDecimal;
+
+/**
+ * Atan2Function
+ * description: atan2(numeric)--returns the arc tangent of a coordinate
(numeric1, numeric2).
+ */
+@TransformFunction(names = {"atan2"})
+public class Atan2Function implements ValueParser {
+
+private ValueParser xParser;
+private ValueParser yParser;
+
+public Atan2Function(Function expr) {
+xParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+yParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object xObj = xParser.parse(sourceData, rowIndex, context);
+Object yObj = yParser.parse(sourceData, rowIndex, context);
+
+BigDecimal xValue = OperatorTools.parseBigDecimal(xObj);
Review Comment:
Thank you for your mindful review! Here is my little confusion:
What action should happen if null pointer been checked? Throw exceptions
like IllegalArgumentException or ArithmeticException, Or just return null as
result?
And why only these 2 methods need null pointer exception check？I know that
Java Math Functions need params ( not null, otherwise would cause compile
error) in raw data type which can transfer to DOUBLE, then dose other
trigonometric function also need check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10287][Agent] Update the Redis Source [inlong]
justinwwhuang commented on code in PR #10801:
URL: https://github.com/apache/inlong/pull/10801#discussion_r1740891986
##
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/RedisSource.java:
##
@@ -84,6 +182,96 @@ public boolean sourceFinish() {
@Override
public boolean sourceExist() {
-return false;
+return true;
+}
+
+private String getRedisUri() {
+StringBuffer sb = new StringBuffer("redis://");
+sb.append(hostName).append(":").append(port);
+if (!StringUtils.isEmpty(dbNumber)) {
+sb.append("/").append(dbNumber);
+}
+sb.append("?");
+if (!StringUtils.isEmpty(authPassword)) {
+sb.append("authPassword=").append(authPassword).append("&");
+}
+if (!StringUtils.isEmpty(authUser)) {
+sb.append("authUser=").append(authUser).append("&");
+}
+if (!StringUtils.isEmpty(readTimeout)) {
+sb.append("readTimeout=").append(readTimeout).append("&");
+}
+if (ssl) {
+sb.append("ssl=").append("yes").append("&");
+}
+if (!StringUtils.isEmpty(snapShot)) {
+sb.append("replOffset=").append(snapShot).append("&");
+}
+if (!StringUtils.isEmpty(replId)) {
+sb.append("replId=").append(replId).append("&");
+}
+if (sb.charAt(sb.length() - 1) == '?' || sb.charAt(sb.length() - 1) ==
'&') {
+sb.deleteCharAt(sb.length() - 1);
+}
+return sb.toString();
+}
+
+private void initReplicator() {
+redisReplicator.addEventListener((replicator, event) -> {
Review Comment:
> > > i fix it as obtaining data by Jedis through commands first, then use
RedisReplicator to subscribe event and filtering data by command, is that
correct ?
> >
> >
> >
> > 1. We don't necessarily mean to collect all the data from the entire
Redis cluster. Your implementation here has always wanted to collect all the
data? From the earliest subscription to scanning all keys now.
> > 2. The reids command is not used as a filter, it is a different
collection method from subscription. Remember, there should be no filtering
logic.
>
> then I feel like we don't need jedis at all, we can just get the data we
need from subscribed events in redis replicator, Debezium does this for data
capture and synchronization of databases too
No, I hope to support data retrieval without subscription, only through
Redis commands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10287][Agent] Update the Redis Source [inlong]
emptyOVO closed pull request #10801: [INLONG-10287][Agent] Update the Redis Source URL: https://github.com/apache/inlong/pull/10801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10881][SDK] Transform SQL support Concat_ws function [inlong]
MOONSakura0614 opened a new pull request, #10992: URL: https://github.com/apache/inlong/pull/10992 Fixes #10881 ### Motivation ### Modifications Add a function class: ConcatWsFunction and its unit test. ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10986][SDK] Transform support REGEXP_MATCHES(str, pattern) function [inlong]
emptyOVO opened a new pull request, #10993:
URL: https://github.com/apache/inlong/pull/10993
…nction
Fixes #10986
### Motivation
add RegexpMatchesFunction class to support postgresql function
REGEXP_MATCHES(str, pattern), provide unit tests
### Modifications
### Verifying this change
Matching a single pattern:
Let's say we have a string from which we want to extract substrings that
match a particular pattern
* REGEXP_MATCHES('The quick brown fox', 'quick')--returns {quick}
Extract multiple subpatterns
Multiple substrings can be extracted using capture groups in regular
expressions
* REGEXP_MATCHES('User: Alice, ID: 12345', 'User: (\w+), ID:
(\d+)')--returns {Alice,12345}
Globally match multiple patterns
The g flag can be used when you want to match all the substrings that occur
* REGEXP_MATCHES('foo 123 bar 456', '\d+', 'g')--returns {"123","456"}
Case matching is ignored
Use the i flag to ignore case for matching
* REGEXP_MATCHES('Hello World', 'hello', 'i')--returns {Hello}
Multi-line pattern matching
Using the m flag allows regular expressions to match across multiple lines
* REGEXP_MATCHES('First line\nSecond line', '^Second', 'm')--returns {Second}
Some Announcements：
REGEXP_MATCHES() returns an array, so be careful about using arrays when
processing the results.
If the global flag g is not used, only the first matching substring is
returned.
If there are multiple flags, the effects need to be performed simultaneously
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [x] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up
issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [INLONG-10882][SDK] Transform SQL support ASCII function [inlong]
MOONSakura0614 opened a new pull request, #10994: URL: https://github.com/apache/inlong/pull/10994 Fixes #10882 ### Motivation ### Modifications Add a function class: AsciiFunction and its unit test. ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-10956][SDK] Transform SQL supports SHA encryption algorithm [inlong]
Zkplo opened a new pull request, #10996: URL: https://github.com/apache/inlong/pull/10996 Fixes #10956 ### Motivation Add two function classes: ShaFunction, Sha2Function. Also, add the corresponding unit test codes. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10986][SDK] Transform support REGEXP_MATCHES(str, pattern) function [inlong]
aloyszhang commented on code in PR #10993:
URL: https://github.com/apache/inlong/pull/10993#discussion_r1741352469
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/RegexpMatchesFunction.java:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+@TransformFunction(names = {"regexp_matches"})
+public class RegexpMatchesFunction implements ValueParser {
Review Comment:
Add descriptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10881][SDK] Transform SQL support concat_ws function [inlong]
aloyszhang commented on code in PR #10992:
URL: https://github.com/apache/inlong/pull/10992#discussion_r1741353536
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/ConcatWsFunction.java:
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+import net.sf.jsqlparser.expression.NullValue;
+
+import java.util.ArrayList;
+import java.util.List;
+
+@TransformFunction(names = {"concat_ws"})
Review Comment:
add descriptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10984][SDK] Transform support RADIANS(x) function [inlong]
luchunliang merged PR #10987: URL: https://github.com/apache/inlong/pull/10987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10984][SDK] Transform support RADIANS(x) function (#10987)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new c08549eff5 [INLONG-10984][SDK] Transform support RADIANS(x) function
(#10987)
c08549eff5 is described below
commit c08549eff5113d23b03f4436ca02a77949f4a49d
Author: emptyOVO <118812562+empty...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:23:03 2024 +0800
[INLONG-10984][SDK] Transform support RADIANS(x) function (#10987)
* [INLONG-10984][SDK] Transform support RADIANS(x) function
* fix: add description
---
.../process/function/RadiansFunction.java | 49 ++
.../TestTransformArithmeticFunctionsProcessor.java | 17
2 files changed, 66 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/RadiansFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/RadiansFunction.java
new file mode 100644
index 00..635d18b9b8
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/RadiansFunction.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+/**
+ * RadiansFunction
+ * description:
+ * - RADIANS(x)--returns radians of x, Convert degrees to radians
+ */
+@TransformFunction(names = {"radians"})
+public class RadiansFunction implements ValueParser {
+
+private ValueParser degreeParser;
+
+public RadiansFunction(Function expr) {
+degreeParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object degreeObj = degreeParser.parse(sourceData, rowIndex, context);
+if (degreeObj == null) {
+return null;
+}
+return
Math.toRadians(OperatorTools.parseBigDecimal(degreeObj).doubleValue());
+}
+}
diff --git
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
index 1e0f14025f..c36835e579 100644
---
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
+++
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
@@ -422,6 +422,23 @@ public class TestTransformArithmeticFunctionsProcessor {
Assert.assertEquals(output2.get(0), "result=2.302585092994046");
}
+@Test
+public void testRadiansFunction() throws Exception {
+String transformSql = "select radians(numeric1) from source";
+TransformConfig config = new TransformConfig(transformSql);
+TransformProcessor processor = TransformProcessor
+.create(config,
SourceDecoderFactory.createCsvDecoder(csvSource),
+SinkEncoderFactory.createKvEncoder(kvSink));
+// case1: radians(10)
+List output1 = processor.transform("10|4|6|8", new
HashMap<>());
+Assert.assertEquals(1, output1.size());
+Assert.assertEquals(output1.get(0), "result=0.17453292519943295");
+// case2: radians(18.97)
+List output2 = processor.transform("18.97|4|6|8", new
HashMap<>());
+Assert.assertEquals(1, output2.size());
+Assert.assertEquals(output2.get(0), "result=0.33108895910332425");
+}
+
@Test
public void testLog10Function() throws Exception {
String transformSql = "select log10(numeric1) f
Re: [PR] [INLONG-10982][SDK] Improve the test code structure [inlong]
luchunliang merged PR #10985: URL: https://github.com/apache/inlong/pull/10985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10982][SDK] Improve the test code structure (#10985)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new bc6e9610bb [INLONG-10982][SDK] Improve the test code structure (#10985)
bc6e9610bb is described below
commit bc6e9610bb84efe3e794cda5d79a7dda784b71e8
Author: Xincheng Huang <60057611+ying-...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:23:19 2024 +0800
[INLONG-10982][SDK] Improve the test code structure (#10985)
---
.../transform/process/TestTransformProcessor.java | 67 ++
1 file changed, 16 insertions(+), 51 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformProcessor.java
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformProcessor.java
index 20af097de6..3413f1aca3 100644
---
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformProcessor.java
+++
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformProcessor.java
@@ -44,13 +44,7 @@ public class TestTransformProcessor {
@Test
public void testCsv2Kv() throws Exception {
-List fields = new ArrayList<>();
-FieldInfo ftime = new FieldInfo();
-ftime.setName("ftime");
-fields.add(ftime);
-FieldInfo extinfo = new FieldInfo();
-extinfo.setName("extinfo");
-fields.add(extinfo);
+List fields = this.getTestFieldList("ftime", "extinfo");
CsvSourceInfo csvSource = new CsvSourceInfo("UTF-8", '|', '\\',
fields);
KvSinkInfo kvSink = new KvSinkInfo("UTF-8", fields);
String transformSql = "select ftime,extinfo from source where
extinfo='ok'";
@@ -97,13 +91,7 @@ public class TestTransformProcessor {
@Test
public void testKv2Csv() throws Exception {
-List fields = new ArrayList<>();
-FieldInfo ftime = new FieldInfo();
-ftime.setName("ftime");
-fields.add(ftime);
-FieldInfo extinfo = new FieldInfo();
-extinfo.setName("extinfo");
-fields.add(extinfo);
+List fields = this.getTestFieldList("ftime", "extinfo");
KvSourceInfo kvSource = new KvSourceInfo("UTF-8", fields);
CsvSinkInfo csvSink = new CsvSinkInfo("UTF-8", '|', '\\', fields);
String transformSql = "select ftime,extinfo from source where
extinfo='ok'";
@@ -148,7 +136,7 @@ public class TestTransformProcessor {
@Test
public void testJson2Csv() throws Exception {
-List fields1 = this.getTestFieldList();
+List fields1 = this.getTestFieldList("sid", "packageID",
"msgTime", "msg");
JsonSourceInfo jsonSource1 = new JsonSourceInfo("UTF-8", "msgs");
CsvSinkInfo csvSink1 = new CsvSinkInfo("UTF-8", '|', '\\', fields1);
String transformSql1 = "select
$root.sid,$root.packageID,$child.msgTime,$child.msg from source";
@@ -170,7 +158,7 @@ public class TestTransformProcessor {
Assert.assertEquals(output1.get(0),
"value1|value2|1713243918000|value4");
Assert.assertEquals(output1.get(1), "value1|value2|1713243918000|v4");
// case2
-List fields2 = this.getTestFieldList2();
+List fields2 = this.getTestFieldList("id", "itemId",
"subItemId", "msg");
JsonSourceInfo jsonSource2 = new JsonSourceInfo("UTF-8", "items");
CsvSinkInfo csvSink2 = new CsvSinkInfo("UTF-8", '|', '\\', fields2);
String transformSql2 =
@@ -205,7 +193,7 @@ public class TestTransformProcessor {
@Test
public void testJson2CsvForOne() throws Exception {
-List fields = this.getTestFieldList();
+List fields = this.getTestFieldList("sid", "packageID",
"msgTime", "msg");
JsonSourceInfo jsonSource = new JsonSourceInfo("UTF-8", "");
CsvSinkInfo csvSink = new CsvSinkInfo("UTF-8", '|', '\\', fields);
String transformSql = "select
$root.sid,$root.packageID,$root.msgs(1).msgTime,$root.msgs(0).msg from source";
@@ -229,7 +217,7 @@ public class TestTransformProcessor {
@Test
public void testPb2Csv() throws Exception {
-List fields = this.getTestFieldList();
+List fields = this.getTestFieldList("sid", "packageID",
"msgTime", "msg");
String transformBase64 = this.getPbTestDescription();
PbSourceInfo pbSource = new PbSourceInfo("UTF-8", transformBase64,
"SdkDataRequest", "msgs");
CsvSinkInfo csvSink = new CsvSinkInfo("UTF-8", '|', '\\', fields);
@@ -246,36 +234,13 @@ public class TestTransformProcessor {
Assert.assertEquals(output.get(1), "sid|1|1713243918002|msgValue42");
}
-private List getTestFieldList() {
-List fields = new ArrayList<>();
-FieldInfo sid = new FieldInfo();
-sid.setName("sid");
-fields.add(sid);
-
Re: [PR] [INLONG-10809][SDK] Improvements to TypeConverter field types and CompareValue in OperatorTools [inlong]
luchunliang merged PR #10817: URL: https://github.com/apache/inlong/pull/10817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10809][SDK] Improvements to TypeConverter field types and CompareValue in OperatorTools (#10817)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new dfd42dc88d [INLONG-10809][SDK] Improvements to TypeConverter field
types and CompareValue in OperatorTools (#10817)
dfd42dc88d is described below
commit dfd42dc88d7f5bb95d66165ae5cfcf6c92921bd7
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:24:08 2024 +0800
[INLONG-10809][SDK] Improvements to TypeConverter field types and
CompareValue in OperatorTools (#10817)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../inlong/sdk/transform/decode/CsvSourceData.java | 10 +-
.../sdk/transform/decode/CsvSourceDecoder.java | 8 +-
.../inlong/sdk/transform/decode/SourceData.java| 2 +-
.../inlong/sdk/transform/pojo/FieldInfo.java | 2 +-
.../converter/DoubleConverter.java}| 16 +-
.../converter/LongConverter.java} | 16 +-
.../sdk/transform/process/function/IfFunction.java | 55
.../transform/process/operator/OperatorTools.java | 21 +-
.../sdk/transform/process/parser/DoubleParser.java | 1 +
.../TestTransformExpressionOperatorsProcessor.java | 354 +
10 files changed, 445 insertions(+), 40 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceData.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceData.java
index d4492b4b85..e0bd9f794c 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceData.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceData.java
@@ -28,14 +28,14 @@ import java.util.Map;
*/
public class CsvSourceData implements SourceData {
-private List> rows = new ArrayList<>();
+private List> rows = new ArrayList<>();
-private Map currentRow;
+private Map currentRow;
public CsvSourceData() {
}
-public void putField(String fieldName, String fieldValue) {
+public void putField(String fieldName, Object fieldValue) {
this.currentRow.put(fieldName, fieldValue);
}
@@ -50,11 +50,11 @@ public class CsvSourceData implements SourceData {
}
@Override
-public String getField(int rowNum, String fieldName) {
+public Object getField(int rowNum, String fieldName) {
if (rowNum >= this.rows.size()) {
return null;
}
-Map targetRow = this.rows.get(rowNum);
+Map targetRow = this.rows.get(rowNum);
return targetRow.get(fieldName);
}
}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceDecoder.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceDecoder.java
index fb95dadc43..7b3dedb637 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceDecoder.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/CsvSourceDecoder.java
@@ -76,9 +76,13 @@ public class CsvSourceDecoder implements
SourceDecoder {
int fieldIndex = 0;
for (FieldInfo field : fields) {
String fieldName = field.getName();
-String fieldValue = null;
+Object fieldValue = null;
if (fieldIndex < fieldValues.length) {
-fieldValue = fieldValues[fieldIndex];
+try {
+fieldValue =
field.getConverter().convert(fieldValues[fieldIndex]);
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
}
sourceData.putField(fieldName, fieldValue);
fieldIndex++;
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/SourceData.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/SourceData.java
index 2c39948f2d..cf5f9c0fbe 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/SourceData.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/SourceData.java
@@ -26,5 +26,5 @@ public interface SourceData {
int getRowCount();
-String getField(int rowNum, String fieldName);
+Object getField(int rowNum, String fieldName);
}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/pojo/FieldInfo.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/pojo/FieldInfo.java
index 1027dad944..2a7834112a 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/pojo/FieldInfo.java
Re: [PR] [INLONG-10979][SDK] Transform support PI() function [inlong]
luchunliang merged PR #10983: URL: https://github.com/apache/inlong/pull/10983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10979][SDK] Transform support PI() function (#10983)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 0a394f49ea [INLONG-10979][SDK] Transform support PI() function (#10983)
0a394f49ea is described below
commit 0a394f49ea026b2536ef0b2a90e3a9183e4a51de
Author: emptyOVO <118812562+empty...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:25:35 2024 +0800
[INLONG-10979][SDK] Transform support PI() function (#10983)
---
.../sdk/transform/process/function/PiFunction.java | 40 ++
.../TestTransformArithmeticFunctionsProcessor.java | 13 +++
2 files changed, 53 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/PiFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/PiFunction.java
new file mode 100644
index 00..722955ebf5
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/PiFunction.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+/**
+ * PiFunction
+ * returns the mathematical constant PI
+ */
+@TransformFunction(names = {"pi"})
+public class PiFunction implements ValueParser {
+
+public PiFunction(Function expr) {
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+return String.valueOf(Math.PI);
+}
+
+}
diff --git
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
index c36835e579..4007a9a876 100644
---
a/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
+++
b/inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/TestTransformArithmeticFunctionsProcessor.java
@@ -728,4 +728,17 @@ public class TestTransformArithmeticFunctionsProcessor {
Assert.assertEquals(output5.get(0), "result=616263");
}
+@Test
+public void testPiFunction() throws Exception {
+String transformSql1 = "select pi() from source";
+TransformConfig config1 = new TransformConfig(transformSql1);
+TransformProcessor processor1 = TransformProcessor
+.create(config1,
SourceDecoderFactory.createCsvDecoder(csvSource),
+SinkEncoderFactory.createKvEncoder(kvSink));
+// case: pi()
+List output1 = processor1.transform("1007|4|6|8", new
HashMap<>());
+Assert.assertEquals(1, output1.size());
+Assert.assertEquals(output1.get(0), "result=3.141592653589793");
+}
+
}
Re: [PR] [INLONG-10971][SDK] Transform support INSERT function [inlong]
luchunliang merged PR #10972: URL: https://github.com/apache/inlong/pull/10972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10971][SDK] Transform support INSERT function (#10972)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 3e7f81d41c [INLONG-10971][SDK] Transform support INSERT function
(#10972)
3e7f81d41c is described below
commit 3e7f81d41c0e1ab8294d93b374e54515d0b7936b
Author: Xincheng Huang <60057611+ying-...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:25:52 2024 +0800
[INLONG-10971][SDK] Transform support INSERT function (#10972)
---
.../transform/process/function/InsertFunction.java | 101 +
.../TestTransformStringFunctionsProcessor.java | 34 +++
2 files changed, 135 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/InsertFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/InsertFunction.java
new file mode 100644
index 00..3473bc9d79
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/InsertFunction.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.List;
+/**
+ * InsertFunction
+ *
+ * Description:
+ * Returns a string where a specified substring is replaced by another string,
starting at a given position and for a specified length.
+ * If the position is out of the string's bounds, the original string is
returned.
+ * If the length exceeds the remaining length of the string from the given
position, the replacement continues to the end of the string.
+ * If any argument is null, the function returns null.
+ *
+ * Arguments:
+ * - str: The original string.
+ * - pos: The position to start the replacement (1-based index).
+ * - len: The number of characters to replace.
+ * - newstr: The string to insert.
+ *
+ * Examples:
+ * - INSERT('12345678', 3, 4, 'word') = '12word78'
+ * - INSERT('12345678', -1, 4, 'word') = '12345678'
+ * - INSERT('12345678', 3, 100, 'word') = '12word'
+ */
+@TransformFunction(names = {"insert"})
+public class InsertFunction implements ValueParser {
+
+private ValueParser strParser;
+private ValueParser posParser;
+private ValueParser lenParser;
+private ValueParser newStrParser;
+
+public InsertFunction(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+strParser = OperatorTools.buildParser(expressions.get(0));
+posParser = OperatorTools.buildParser(expressions.get(1));
+lenParser = OperatorTools.buildParser(expressions.get(2));
+newStrParser = OperatorTools.buildParser(expressions.get(3));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object strObject = strParser.parse(sourceData, rowIndex, context);
+Object posObject = posParser.parse(sourceData, rowIndex, context);
+Object lenObject = lenParser.parse(sourceData, rowIndex, context);
+Object newStrObject = newStrParser.parse(sourceData, rowIndex,
context);
+
+if (strObject == null || posObject == null || lenObject == null ||
newStrObject == null) {
+return null;
+}
+
+String str = OperatorTools.parseString(strObject);
+int pos = OperatorTools.parseBigDecimal(posObject).intValue();
+int len = OperatorTools.parseBigDecimal(lenObject).intValue();
+String newStr = OperatorTools.parseString(newStrObject);
+
+if (str == null || newStr == null) {
+return null;
+}
+
+if (pos < 1 || pos > str.length()) {
+return str;
+}
+
+int startIndex = pos - 1;
+int endIndex = Math.
Re: [PR] [INLONG-10965][SDK] Transform support Bitwise operation [inlong]
luchunliang merged PR #10970: URL: https://github.com/apache/inlong/pull/10970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10965][SDK] Transform support Bitwise operation (#10970)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new cb4b61b5a8 [INLONG-10965][SDK] Transform support Bitwise operation
(#10970)
cb4b61b5a8 is described below
commit cb4b61b5a889cbb035b51ad1dd48f6a02790573f
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:26:19 2024 +0800
[INLONG-10965][SDK] Transform support Bitwise operation (#10970)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../transform/process/parser/BitwiseAndParser.java | 61 +++
.../process/parser/BitwiseLeftShiftParser.java | 67
.../transform/process/parser/BitwiseOrParser.java | 62
.../process/parser/BitwiseRightShiftParser.java| 66
.../transform/process/parser/BitwiseXorParser.java | 62
.../sdk/transform/process/parser/SignParser.java | 26 ++-
.../TestTransformArithmeticFunctionsProcessor.java | 175 +
7 files changed, 513 insertions(+), 6 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseAndParser.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseAndParser.java
new file mode 100644
index 00..b8920434ef
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseAndParser.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.parser;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.operators.arithmetic.BitwiseAnd;
+
+import java.math.BigInteger;
+
+/**
+ * BitwiseAndParser
+ */
+@Slf4j
+@TransformParser(values = BitwiseAnd.class)
+public class BitwiseAndParser implements ValueParser {
+
+private final ValueParser left;
+
+private final ValueParser right;
+
+public BitwiseAndParser(BitwiseAnd expr) {
+this.left = OperatorTools.buildParser(expr.getLeftExpression());
+this.right = OperatorTools.buildParser(expr.getRightExpression());
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+try {
+Object leftObj = this.left.parse(sourceData, rowIndex, context);
+Object rightObj = this.right.parse(sourceData, rowIndex, context);
+if (leftObj == null || rightObj == null) {
+return null;
+}
+BigInteger leftValue =
OperatorTools.parseBigDecimal(leftObj).toBigInteger();
+BigInteger rightValue =
OperatorTools.parseBigDecimal(rightObj).toBigInteger();
+return
Long.toUnsignedString(leftValue.and(rightValue).longValue());
+} catch (Exception e) {
+log.error("Value parsing failed", e);
+return null;
+}
+}
+}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseLeftShiftParser.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseLeftShiftParser.java
new file mode 100644
index 00..c68bc89e54
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/BitwiseLeftShiftParser.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in wr
Re: [PR] [INLONG-10873][SDK] Transform support factorial function [inlong]
aloyszhang commented on PR #10874: URL: https://github.com/apache/inlong/pull/10874#issuecomment-2325485969 please fix the conflicting file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10968][SDK] Support Inlong Transform operator annotation [inlong]
aloyszhang merged PR #10969: URL: https://github.com/apache/inlong/pull/10969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10968][SDK] Support Inlong Transform operator annotation (#10969)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new f1285ed4b1 [INLONG-10968][SDK] Support Inlong Transform operator
annotation (#10969)
f1285ed4b1 is described below
commit f1285ed4b1c9ec1ff16942c6324f6fad46c6898c
Author: emptyOVO <118812562+empty...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:33:10 2024 +0800
[INLONG-10968][SDK] Support Inlong Transform operator annotation (#10969)
---
.../transform/process/operator/AndOperator.java| 1 +
.../process/operator/EqualsToOperator.java | 1 +
.../operator/GreaterThanEqualsOperator.java| 1 +
.../process/operator/GreaterThanOperator.java | 1 +
.../process/operator/MinorThanEqualsOperator.java | 1 +
.../process/operator/MinorThanOperator.java| 1 +
.../process/operator/NotEqualsToOperator.java | 1 +
.../transform/process/operator/NotOperator.java| 1 +
.../transform/process/operator/OperatorTools.java | 89 ++
.../sdk/transform/process/operator/OrOperator.java | 1 +
.../process/operator/ParenthesisOperator.java | 1 +
.../{NotOperator.java => TransformOperator.java} | 33 ++--
12 files changed, 77 insertions(+), 55 deletions(-)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/AndOperator.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/AndOperator.java
index a9dcd42606..f438d4295c 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/AndOperator.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/AndOperator.java
@@ -26,6 +26,7 @@ import
net.sf.jsqlparser.expression.operators.conditional.AndExpression;
* AndOperator
*
*/
+@TransformOperator(values = AndExpression.class)
public class AndOperator implements ExpressionOperator {
private final ExpressionOperator left;
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/EqualsToOperator.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/EqualsToOperator.java
index 709537e8a0..13010c854b 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/EqualsToOperator.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/EqualsToOperator.java
@@ -27,6 +27,7 @@ import
net.sf.jsqlparser.expression.operators.relational.EqualsTo;
* EqualsToOperator
*
*/
+@TransformOperator(values = EqualsTo.class)
public class EqualsToOperator implements ExpressionOperator {
private final ValueParser left;
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanEqualsOperator.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanEqualsOperator.java
index 3a53968e10..e703afdbda 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanEqualsOperator.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanEqualsOperator.java
@@ -27,6 +27,7 @@ import
net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
* GreaterThanEqualsOperator
*
*/
+@TransformOperator(values = GreaterThanEquals.class)
public class GreaterThanEqualsOperator implements ExpressionOperator {
private final ValueParser left;
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanOperator.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanOperator.java
index a1cd8c2ea2..ba73ef4c4d 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanOperator.java
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/GreaterThanOperator.java
@@ -27,6 +27,7 @@ import
net.sf.jsqlparser.expression.operators.relational.GreaterThan;
* GreaterThanOperator
*
*/
+@TransformOperator(values = GreaterThan.class)
public class GreaterThanOperator implements ExpressionOperator {
private final ValueParser left;
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/MinorThanEqualsOperator.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/MinorThanEqualsOperator.java
index 4248cf1d36..e8104e2cc3 100644
---
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/operator/MinorThanEqualsOperator.java
+++
b/inlong-sdk
Re: [PR] [INLONG-10940][SDK] Transform SQL support arithmetic functions(Including cot, tanh, cosh, asin, atan and atan2) [inlong]
aloyszhang commented on PR #10950: URL: https://github.com/apache/inlong/pull/10950#issuecomment-2325490511 please fix the conflicting file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10940][SDK] Transform SQL support arithmetic functions(Including cot, tanh, cosh, asin, atan and atan2) [inlong]
luchunliang commented on code in PR #10950:
URL: https://github.com/apache/inlong/pull/10950#discussion_r1741360749
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Atan2Function.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+import java.math.BigDecimal;
+
+/**
+ * Atan2Function
+ * description: atan2(numeric)--returns the arc tangent of a coordinate
(numeric1, numeric2).
+ */
+@TransformFunction(names = {"atan2"})
+public class Atan2Function implements ValueParser {
+
+private ValueParser xParser;
+private ValueParser yParser;
+
+public Atan2Function(Function expr) {
+xParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+yParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object xObj = xParser.parse(sourceData, rowIndex, context);
+Object yObj = yParser.parse(sourceData, rowIndex, context);
+
+BigDecimal xValue = OperatorTools.parseBigDecimal(xObj);
Review Comment:
If the parameter can not be parsed, then return null.
For general business, it is acceptable to miss field, but miss data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10940][SDK] Transform SQL support arithmetic functions(Including cot, tanh, cosh, asin, atan and atan2) [inlong]
luchunliang commented on code in PR #10950:
URL: https://github.com/apache/inlong/pull/10950#discussion_r1741360874
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Atan2Function.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Function;
+
+import java.math.BigDecimal;
+
+/**
+ * Atan2Function
+ * description: atan2(numeric)--returns the arc tangent of a coordinate
(numeric1, numeric2).
+ */
+@TransformFunction(names = {"atan2"})
+public class Atan2Function implements ValueParser {
+
+private ValueParser xParser;
+private ValueParser yParser;
+
+public Atan2Function(Function expr) {
+xParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
+yParser =
OperatorTools.buildParser(expr.getParameters().getExpressions().get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object xObj = xParser.parse(sourceData, rowIndex, context);
+Object yObj = yParser.parse(sourceData, rowIndex, context);
+
+BigDecimal xValue = OperatorTools.parseBigDecimal(xObj);
Review Comment:
For some strict validation scenarios, adopting a dirty data shunting scheme
after throwing an exception is a better choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10956][SDK] Transform SQL supports SHA encryption algorithm [inlong]
luchunliang merged PR #10996: URL: https://github.com/apache/inlong/pull/10996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10956][SDK] Transform SQL supports SHA encryption algorithm (#10996)
This is an automated email from the ASF dual-hosted git repository.
luchunliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new e394d74076 [INLONG-10956][SDK] Transform SQL supports SHA encryption
algorithm (#10996)
e394d74076 is described below
commit e394d740769730f6960e7022e0bc8f4dee1e2e9b
Author: Zkplo <87751516+zk...@users.noreply.github.com>
AuthorDate: Tue Sep 3 10:39:27 2024 +0800
[INLONG-10956][SDK] Transform SQL supports SHA encryption algorithm (#10996)
Co-authored-by: ZKpLo <14148880+zk...@user.noreply.gitee.com>
---
.../transform/process/function/Sha2Function.java | 75 +
.../transform/process/function/ShaFunction.java| 54 +++
.../TestTransformArithmeticFunctionsProcessor.java | 78 ++
3 files changed, 207 insertions(+)
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Sha2Function.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Sha2Function.java
new file mode 100644
index 00..ad120712bf
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/Sha2Function.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+import org.apache.commons.codec.digest.DigestUtils;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+import static org.apache.commons.codec.digest.MessageDigestAlgorithms.SHA_224;
+
+/**
+ * Sha2Function
+ * description: SHA2(str, hash_length): Calculates the SHA-2 family of hash
functions (SHA-224, SHA-256, SHA-384, and SHA-512)
+ * return NULL If either argument is NULL or the hash length(224 256 384 512)
is not one of the permitted values
+ * return a hash value containing the desired number of bits.
+ */
+@TransformFunction(names = {"sha2"})
+public class Sha2Function implements ValueParser {
+
+private final ValueParser msgParser;
+private final ValueParser lenParser;
+
+public Sha2Function(Function expr) {
+List expressions = expr.getParameters().getExpressions();
+msgParser = OperatorTools.buildParser(expressions.get(0));
+lenParser = OperatorTools.buildParser(expressions.get(1));
+}
+
+@Override
+public Object parse(SourceData sourceData, int rowIndex, Context context) {
+Object msgObj = msgParser.parse(sourceData, rowIndex, context);
+Object lenObj = lenParser.parse(sourceData, rowIndex, context);
+if (msgObj == null || lenObj == null) {
+return null;
+}
+String msg = msgObj.toString();
+int len = Integer.parseInt(lenObj.toString());
+switch (len) {
+case 0:
+case 256:
+return
DigestUtils.sha256Hex(msg.getBytes(StandardCharsets.UTF_8));
+case 224:
+return new
DigestUtils(SHA_224).digestAsHex(msg.getBytes(StandardCharsets.UTF_8));
+case 384:
+return
DigestUtils.sha384Hex(msg.getBytes(StandardCharsets.UTF_8));
+case 512:
+return
DigestUtils.sha512Hex(msg.getBytes(StandardCharsets.UTF_8));
+default:
+return null;
+}
+}
+}
diff --git
a/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/ShaFunction.java
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/ShaFunction.java
new file mode 100644
index 00..7da5e6c3a9
--- /dev/null
+++
b/inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/ShaFunction.java
@@ -0,0 +1,54 @@
+/*
+ * Lic
Re: [PR] [INLONG-10882][SDK] Transform SQL support ASCII function [inlong]
aloyszhang commented on PR #10994: URL: https://github.com/apache/inlong/pull/10994#issuecomment-2325494715 @MOONSakura0614 Pleas fix the conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10956][SDK] Add XML formatted data source for Transform [inlong]
Zkplo closed pull request #10990: [INLONG-10956][SDK] Add XML formatted data source for Transform URL: https://github.com/apache/inlong/pull/10990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [INLONG-11000][SDK] Add XML formatted data source for Transform [inlong]
Zkplo opened a new pull request, #11001: URL: https://github.com/apache/inlong/pull/11001 Fixes #11000 ### Motivation Add several classes: XmlSourceInfo, XmlNode, XmlSourceData, XmlSourceDecoder. Modified SourceDecoderFactory class. Also, add the corresponding unit test codes. ### Modifications ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [x] This change is already covered by existing tests, such as: *(please describe tests)* - [ ] This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10986][SDK] Transform support REGEXP_MATCHES(str, pattern) function [inlong]
emptyOVO commented on code in PR #10993:
URL: https://github.com/apache/inlong/pull/10993#discussion_r1741439222
##
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/RegexpMatchesFunction.java:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.expression.Function;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+@TransformFunction(names = {"regexp_matches"})
+public class RegexpMatchesFunction implements ValueParser {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [INLONG-10882][SDK] Transform SQL support ASCII function [inlong]
MOONSakura0614 commented on PR #10994: URL: https://github.com/apache/inlong/pull/10994#issuecomment-2325662128 > @MOONSakura0614 Pleas fix the conflict done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(inlong) branch master updated: [INLONG-10997][Manager] Incorrect setting of transformSQL in dataflowconfig (#10998)
This is an automated email from the ASF dual-hosted git repository.
aloyszhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new e188410668 [INLONG-10997][Manager] Incorrect setting of transformSQL
in dataflowconfig (#10998)
e188410668 is described below
commit e1884106681a4062555d413f6f2cdc24ef741229
Author: fuweng11 <76141879+fuwen...@users.noreply.github.com>
AuthorDate: Tue Sep 3 14:07:56 2024 +0800
[INLONG-10997][Manager] Incorrect setting of transformSQL in dataflowconfig
(#10998)
---
.../inlong/manager/service/resource/sort/DefaultSortConfigOperator.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sort/DefaultSortConfigOperator.java
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sort/DefaultSortConfigOperator.java
index 116a8f0ea2..17ca4b9bad 100644
---
a/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sort/DefaultSortConfigOperator.java
+++
b/inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sort/DefaultSortConfigOperator.java
@@ -175,6 +175,7 @@ public class DefaultSortConfigOperator implements
SortConfigOperator {
.dataflowId(String.valueOf(sink.getId()))
.sourceConfig(getSourceConfig(groupInfo, streamInfo, sink))
.auditTag(String.valueOf(sink.getId()))
+.transformSql(sink.getTransformSql())
.sinkConfig(getSinkConfig(groupInfo, streamInfo, sink))
.inlongGroupId(groupInfo.getInlongGroupId())
.inlongStreamId(streamInfo.getInlongStreamId())
Re: [PR] [INLONG-10997][Manager] Fix the problem of incorrect setting of transformSQL in dataflowconfig [inlong]
aloyszhang merged PR #10998: URL: https://github.com/apache/inlong/pull/10998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10883][SDK] Transform SQL support InitCap function [inlong]
MOONSakura0614 commented on PR #10892: URL: https://github.com/apache/inlong/pull/10892#issuecomment-2325700764 > @MOONSakura0614 please fix the conflicts and failed workflows. already resolve the conflicts and force push. Your careful review is much appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [INLONG-10883][SDK] Transform SQL support InitCap function [inlong]
MOONSakura0614 commented on PR #10892: URL: https://github.com/apache/inlong/pull/10892#issuecomment-2325701250 > plz rebase the master to resolve the conflict done and thank you for checking again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
