[
https://issues.apache.org/jira/browse/FLINK-32315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893685#comment-17893685
]
melin edited comment on FLINK-32315 at 10/29/24 3:34 AM:
---------------------------------------------------------
Can't find minio AK/SK, how to set parameters? [~ferenc-csaky] [~Paul Lin]
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image.ref=fregistry.cn-hangzhou.aliyuncs.com/melin1204/flink:1.20.0
\
-Dkubernetes.artifacts.local-upload-enabled=true \
-Dkubernetes.artifacts.local-upload-target=s3://flink/flink-1.20 \
-Dkubernetes.artifacts.local-upload-overwrite=true \
-Dkubernetes.rest-service.exposed.type=NodePort \
-Dkubernetes.namespace=superior \
-Dkubernetes.service-account=superior \
-Ds3.path.style.access=true \
-Ds3.access-key=McvVnpOziVsWv7Qlyut7 \
-Ds3.secret-key=PbICbD6H7iyq0PuefHa383YoqJn3JCjedQHSYmbp \
-Dflink.hadoop.fs.s3a.endpoint=[http://172.18.6.181:9330|http://172.18.6.181:9330/]
\
-Dflink.hadoop.fs.s3a.access.key=McvVnpOziVsWv7Qlyut7 \
-Dflink.hadoop.fs.s3a.secret.key=PbICbD6H7iyq0PuefHa383YoqJn3JCjedQHSYmbp \
-Dfs.allowed-fallback-filesystems=s3 \
-Dkubernetes.container.image.pull-secrets=docker-reg-demos \
-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.20.0.jar
\
-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.20.0.jar
\
local:///root/flink-1.20.0/examples/streaming/SocketWindowWordCount.jar
--hostname 172.18.5.44 --port 9000
!image-2024-10-29-11-32-25-979.png!
was (Author: melin):
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image.ref=fregistry.cn-hangzhou.aliyuncs.com/melin1204/flink:1.20.0
\
-Dkubernetes.artifacts.local-upload-enabled=true \
-Dkubernetes.artifacts.local-upload-target=s3://flink/flink-1.20 \
-Dkubernetes.artifacts.local-upload-overwrite=true \
-Dkubernetes.rest-service.exposed.type=NodePort \
-Dkubernetes.namespace=superior \
-Dkubernetes.service-account=superior \
-Ds3.path.style.access=true \
-Ds3.access-key=McvVnpOziVsWv7Qlyut7 \
-Ds3.secret-key=PbICbD6H7iyq0PuefHa383YoqJn3JCjedQHSYmbp \
-Dflink.hadoop.fs.s3a.endpoint=http://172.18.6.181:9330 \
-Dflink.hadoop.fs.s3a.access.key=McvVnpOziVsWv7Qlyut7 \
-Dflink.hadoop.fs.s3a.secret.key=PbICbD6H7iyq0PuefHa383YoqJn3JCjedQHSYmbp \
-Dfs.allowed-fallback-filesystems=s3 \
-Dkubernetes.container.image.pull-secrets=docker-reg-demos \
-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.20.0.jar
\
-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.20.0.jar
\
local:///root/flink-1.20.0/examples/streaming/SocketWindowWordCount.jar
--hostname 172.18.5.44 --port 9000
Can't find minio AK/SK, how to set parameters?
!image-2024-10-29-11-32-25-979.png!
> Support local file upload in K8s mode
> -------------------------------------
>
> Key: FLINK-32315
> URL: https://issues.apache.org/jira/browse/FLINK-32315
> Project: Flink
> Issue Type: New Feature
> Components: Client / Job Submission, Deployment / Kubernetes
> Reporter: Paul Lin
> Assignee: Ferenc Csaky
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Attachments: image-2024-10-29-11-32-25-979.png
>
>
> Currently, Flink assumes all resources are locally accessible in the pods,
> which requires users to prepare the resources by mounting storages,
> downloading resources with init containers, or rebuilding images for each
> execution.
> We could make things much easier by introducing a built-in file distribution
> mechanism based on Flink-supported filesystems. It's implemented in two steps:
>
> 1. KubernetesClusterDescripter uploads all local resources to remote storage
> via Flink filesystem (skips if the resources are already remote).
> 2. KubernetesApplicationClusterEntrypoint and KubernetesTaskExecutorRunner
> download the resources and put them in the classpath during startup.
>
> The 2nd step is mostly done by
> [FLINK-28915|https://issues.apache.org/jira/browse/FLINK-28915], thus this
> issue is focused on the upload part.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
