This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino-playground.git
The following commit(s) were added to refs/heads/main by this push:
new 7a25815 [#142] fix: Spark Using Apache Iceberg REST service example
(#148)
7a25815 is described below
commit 7a25815a16f7559e19d7062140f6d8aa552db89a
Author: Leonard Siu <[email protected]>
AuthorDate: Mon Sep 29 14:39:01 2025 +0800
[#142] fix: Spark Using Apache Iceberg REST service example (#148)
### What changes were proposed in this pull request?
Also source `.env` file so environment variables are available to
`xxx-dependency.sh` scripts called by `playground.sh`.
### Why are the changes needed?
The examples in [Using Apache Iceberg REST
service](https://github.com/apache/gravitino-playground?tab=readme-ov-file#using-apache-iceberg-rest-service)
does not work because the dependencies were not properly download.
This is because during the `playground.sh start`, `spark-dependency.sh`
get called and the environment variables in `.env` are not properly
sourced. The problem shows up as an error message when running
`playground.sh start`:
```
Downloading iceberg-spark-runtime-3.4_2.12-1.5.2.jar to
/home/yk/src/gravitino-playground/init/spark/packages
Downloading gravitino-spark-connector-runtime-3.4_2.12 to
/home/yk/src/gravitino-playground/init/spark/packages
Use gravitino-spark-connector-runtime-3.4_2.12.md5 to MD5 checksum
gravitino-spark-connector-runtime-3.4_2.12 verification failed, Please delete
it.
```
I believe the change for
[Issue#142](https://github.com/apache/gravitino-playground/issues/142) /
[PR#143](https://github.com/apache/gravitino-playground/pull/143) expect
environment variables defined in [.env file for docker
compose](https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/#env-file)
is also available to `xxx-dependency.sh` scripts, but this is not the
case.
With out fixing this.
Fix: #142
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested.
* testing e2e with `playground.sh` should also pass downloading
gravitino-spark-connector-runtime-3.4_2.12.jar without error.
* just running `init/spark/spark-dependency.sh` also reproduce this
error and show this fix does address the issue.
---
init/common/common.sh | 10 ++++++++++
init/spark/spark-dependency.sh | 2 +-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/init/common/common.sh b/init/common/common.sh
index 49ed6f3..b2e9428 100644
--- a/init/common/common.sh
+++ b/init/common/common.sh
@@ -18,6 +18,16 @@
# under the License.
#
+echo "common $(dirname "${BASH_SOURCE-$0}")"
+common_dir="$(dirname "${BASH_SOURCE-$0}")"
+common_dir="$(
+ cd "${common_dir}" >/dev/null || exit 1
+ pwd
+)"
+
+# Load environment variables
+. "${common_dir}/../../.env"
+
download_and_verify() {
local jar_url=$1
local md5_url=$2
diff --git a/init/spark/spark-dependency.sh b/init/spark/spark-dependency.sh
index 0cfb844..1485762 100755
--- a/init/spark/spark-dependency.sh
+++ b/init/spark/spark-dependency.sh
@@ -33,7 +33,7 @@
ICEBERG_SPARK_RUNTIME_JAR="https://repo1.maven.org/maven2/org/apache/iceberg/ice
ICEBERG_SPARK_RUNTIME_MD5="${ICEBERG_SPARK_RUNTIME_JAR}.md5"
download_and_verify "${ICEBERG_SPARK_RUNTIME_JAR}"
"${ICEBERG_SPARK_RUNTIME_MD5}" "${spark_dir}"
-GRAVITINO_SPARK_CONNECTOR_RUNTIME_JAR="https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-spark-connector-runtime-3.4_2.12/${GRAVITINO_VERSION}/${GRAVITINO_SPARK_CONNECTOR_RUNTIME_JAR}";
+GRAVITINO_SPARK_CONNECTOR_RUNTIME_JAR="https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-spark-connector-runtime-3.4_2.12/${GRAVITINO_VERSION}/${SPARK_CONNECTOR_JAR}";
GRAVITINO_SPARK_CONNECTOR_RUNTIME_MD5="${GRAVITINO_SPARK_CONNECTOR_RUNTIME_JAR}.md5"
download_and_verify "${GRAVITINO_SPARK_CONNECTOR_RUNTIME_JAR}"
"${GRAVITINO_SPARK_CONNECTOR_RUNTIME_MD5}" "${spark_dir}"
