This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 8e04720bd [INLONG-7030][Manager][Sort] Build tool for local debugging
environment (#7031)
8e04720bd is described below
commit 8e04720bd93bf55dcd3e954ef5ff40212102c7c4
Author: feat <featzh...@outlook.com>
AuthorDate: Wed Feb 8 11:33:47 2023 +0800
[INLONG-7030][Manager][Sort] Build tool for local debugging environment
(#7031)
---
.gitignore | 10 +++
inlong-tools/dev/README.md | 138 +++++++++++++++++++++++++++++++
inlong-tools/dev/inlong-dev-toolkit.sh | 143 +++++++++++++++++++++++++++++++++
3 files changed, 291 insertions(+)
diff --git a/.gitignore b/.gitignore
index 4c0d2df5c..655dd273d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,3 +71,13 @@ inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-cpp/release/bin/
# Docker mysql storage
docker/docker-compose/mysql
+
+# Directories for local debugging
+## The connectors for loading to Flink job
+/inlong-sort/connectors
+## The plugins for submitting job to Flink cluster
+/plugins
+## The jar for parsing job info
+/inlong-sort/sort-dist.jar
+## The directory for load plugins by manager
+/inlong-manager/plugins
\ No newline at end of file
diff --git a/inlong-tools/dev/README.md b/inlong-tools/dev/README.md
new file mode 100644
index 000000000..95ba5a0ec
--- /dev/null
+++ b/inlong-tools/dev/README.md
@@ -0,0 +1,138 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+## Apache InLong dev toolkit
+
+### Overview
+
+The Apache InLong project includes modules such as "Dashboard", "DataProxy",
"Manager" and "TubeMQ".
+When debugging in a development environment such as `Intellij IDEA`,
+The environment is difficult to build.
+
+Take the `inlong-manager` as an example, it depends on too many configurations
and packages.
+When debugging locally in the IDE, multiple default directories need to be
created, which is more complicated.
+So a script is urgently needed to quickly support local debugging.
+
+Also, similar tools are required for local debugging of other modules.
+Temporarily named `InLong dev toolkit`, looking forward to adding more
features.
+
+### Use case
+
+#### Help
+
+```shell
+❯ inlong-tools/dev/inlong-dev-toolkit.sh
+####################################################################################
+# Welcome to use the Apache InLong dev toolkit!
#
+# @2022-12-23 20:38:34
#
+####################################################################################
+
+
+inlong-tools/dev/inlong-dev-toolkit.sh help | h
+ :help
+
+inlong-tools/dev/inlong-dev-toolkit.sh manager | m
+ :build manager local debugging environment
+
+Have a nice day, bye!
+
+```
+
+#### Manager
+
+Rely of Manager:
+
+| Directory/File |
link target
| Rely modules |
+|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|---------------------------------|
+| `./plugins` |
`inlong-manager/manger-plugins/target/plugins`
| `inlong-sort/sort-plugin` |
+| `./inlong-sort/connectors` | -
| `inlong-sort/sort-connectors/*` |
|
+| `./inlong-sort/connectors/sort-connector-${name}-${project.version}.jar` |
`inlong-sort/sort-connectors/${connector.name}/target/sort-connector-${connector.name}-${project.version}.jar`
| `inlong-sort/sort-connectors/*` |
+| `./sort-dist.jar` |
`sort-dist/target/sort-dist-${project.version}.jar`
| `inlong-sort/sort-dist` |
+
+```shell
+❯ inlong-tools/dev/inlong-dev-toolkit.sh m
+Execute action: manager
+# start build manager local debugging environment ...
+current_version: 1.5.0-SNAPSHOT
+associate plugins directory: inlong-manager/manager-plugins/target/plugins
+associate sort dist: inlong-sort/sort-dist/target/sort-dist
+recreate connector dir: inlong-sort/connectors
+All connector names:
+hive mysql-cdc kafka jdbc pulsar iceberg postgres-cdc mongodb-cdc
sqlserver-cdc oracle-cdc elasticsearch-6 elasticsearch-7 redis tubemq
filesystem doris starrocks hudi
+associate connector: hive
+associate connector: mysql-cdc
+associate connector: kafka
+associate connector: jdbc
+associate connector: pulsar
+associate connector: iceberg
+associate connector: postgres-cdc
+associate connector: mongodb-cdc
+associate connector: sqlserver-cdc
+associate connector: oracle-cdc
+associate connector: elasticsearch-6
+associate connector: elasticsearch-7
+associate connector: redis
+associate connector: tubemq
+associate connector: filesystem
+associate connector: doris
+associate connector: starrocks
+associate connector: hudi
+build dev env of manager finished.
+Have a nice day, bye!
+```
+
+### Development: add more function
+
+#### Public variables
+
+| inner variable | implication |
+|--------------------|-----------------------------------|
+| `script_dir` | the directory of this script file |
+| `script_file_name` | the script file name |
+| `base_dir` | the root directory of project |
+
+#### Step 1. implement function
+
+#### Step 2. register function
+
+Create a global array variable:
+
+- It is recommended to end with `_action`, such as `manager_action`.
+- It must have 4 elements.
+- The 1st element is the long opt command.
+- The 2nd element is the sort opt command.
+- The 3rd element is the details of the function.
+- The 4th element is the function name, which implements in Step 1.
+
+Add the variable to the `actions` array, like:
+
+```shell
+actions=(
+ help_action
+ manger_action
+)
+```
+
+#### Step 3. ignore the temporary file
+
+Add temporary files to `.gitignore`
+
+> Notice: must base on project base directory
diff --git a/inlong-tools/dev/inlong-dev-toolkit.sh
b/inlong-tools/dev/inlong-dev-toolkit.sh
new file mode 100755
index 000000000..89b253953
--- /dev/null
+++ b/inlong-tools/dev/inlong-dev-toolkit.sh
@@ -0,0 +1,143 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Initialize the configuration files of inlong components
+
+script_dir=$(dirname "$0")
+script_file_name=$(basename "$0")
+
+# the absolute dir of project
+base_dir=$(
+ cd $script_dir
+ cd ../..
+ pwd
+)
+
+help_action=(
+ 'help'
+ 'h'
+ 'help: show all actions'
+ 'welcome'
+)
+
+manager_action=(
+ 'manager'
+ 'm'
+ 'build manager local debugging environment'
+ 'manager'
+)
+
+actions=(
+ help_action
+ manager_action
+)
+
+function welcome() {
+ local_date_time=$(date +"%Y-%m-%d %H:%M:%S")
+ echo
'####################################################################################'
+ echo '# Welcome to use Apache InLong dev toolkit !
#'
+ echo '# @'$local_date_time'
#'
+ echo
'####################################################################################'
+ echo ''
+
+ # shellcheck disable=SC2068
+ for action in ${actions[@]}; do
+ # shellcheck disable=SC1087
+ TMP=$action[@]
+ TempB=("${!TMP}")
+ name=${TempB[0]}
+ simple_name=${TempB[1]}
+ desc=${TempB[2]}
+
+ echo $script_dir'/'$script_file_name' '$name' | '$simple_name
+ echo ' :'$desc
+ done
+ echo ''
+}
+
+function manager() {
+ echo '# start build manager local debugging environment ...'
+
+ project_version=$(mvn -q \
+ -Dexec.executable=echo \
+ -Dexec.args='${project.version}' \
+ --non-recursive \
+ exec:exec)
+
+ echo 'current_version: '"$project_version"
+ #
+ echo 'associate plugins directory:
inlong-manager/manager-plugins/target/plugins'
+ # plugins -> manager-plugins/target/plugins
+ cd "$base_dir"
+ rm -rf plugins
+ ln -s inlong-manager/manager-plugins/target/plugins plugins
+ #
+ echo 'associate sort dist: inlong-sort/sort-dist/target/sort-dist'
+ cd "$base_dir"/inlong-sort
+ rm -rf sort-dist.jar
+ ln -s sort-dist/target/sort-dist-"$project_version".jar sort-dist.jar
+ # inlong-manager: plugins -> manager-plugins/target/plugins
+ cd "$base_dir"/inlong-manager
+ rm -rf plugins
+ ln -s manager-plugins/target/plugins plugins
+ # mkdir inlong-sort/connectors
+ sort_connector_dir=$base_dir/inlong-sort/connectors
+ echo 'recreate connector dir: '"$sort_connector_dir"
+ # shellcheck disable=SC2086
+ rm -rf $sort_connector_dir
+ mkdir "$sort_connector_dir"
+ cd "$sort_connector_dir"
+ connector_names=$(grep '<module>'
"$base_dir"/inlong-sort/sort-connectors/pom.xml | sed 's/<module>//g' | sed
's/<\/module>//g' | grep -v base)
+
+ echo 'All connector names: '
+ echo $connector_names | tr -d '\n'
+
+ for connector_name in $connector_names; do
+ echo 'associate connector: '"$connector_name"
+ connector_suffix_name=$(echo "$connector_name" | sed
's/elasticsearch-6/elasticsearch6/g' | sed 's/elasticsearch-7/elasticsearch7/g')
+ ln -s
../sort-connectors/"${connector_name}"/target/sort-connector-"${connector_suffix_name}"-"$project_version".jar
sort-connector-"${connector_name}".jar
+ done
+
+ echo 'build dev env of manager finished .'
+}
+
+function main() {
+ action=$1
+
+ if [ ! -n "$action" ]; then
+ welcome
+ fi
+
+ # shellcheck disable=SC2068
+ for one_action in ${actions[@]}; do
+ # shellcheck disable=SC1087
+ TMP=$one_action[@]
+ TempB=("${!TMP}")
+ name=${TempB[0]}
+ simple_name=${TempB[1]}
+ function_name=${TempB[3]}
+ desc=${TempB[4]}
+
+ if [[ X"$name" == X"$action" ]] || [[ X"$simple_name" == X"$action" ]];
then
+ echo 'Execute action: '"$function_name"
+ $function_name
+ fi
+ done
+
+ echo 'Have a nice day, bye!'
+}
+
+main $1
