Hi Robert, Tobias
I have tried many ways to build and validate the image.
1.put the s3 dependency to plugin subdirectory, the Dockerfile content is
below:
> FROM apache/flink:1.13.1-scala_2.11
> ADD ./flink-s3-fs-hadoop-1.13.1.jar
> /opt/flink/plugins/s3-hadoop/flink-s3-fs-hadoop-1.13.1.jar
> ADD ./flink-s3-fs-presto-1.13.1.jar
> /opt/flink/plugins/s3-presto/flink-s3-fs-presto-1.13.1.jar
>
This time the image can be run on k8s but would also hit a error like
"org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not
find a file system implementation for scheme 's3'.", it seems like flink
can not find the s3 filesystem supports dynamically. When I want to run the
image using 'docker run -it ', it would also report
'standard_init_linux.go:190: exec user process caused "exec format error"'.
2.put the s3 dependency to plugin directly, the Dockerfile content is below:
> FROM apache/flink:1.13.1-scala_2.11
> ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/plugins
> ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/plugins
>
The image can not run on the k8s and report error just the same as run
the image using 'docker run -it ', 'standard_init_linux.go:190: exec user
process caused "exec format error"'.
3.just run the community edition image flink:1.13.1-scala_2.11 locally as
docker run -it, it will also hit the same error
'standard_init_linux.go:190: exec user process caused "exec format error"',
but the flink:1.13.1-scala_2.11 can be run on the k8s without s3
requirement.
4.import the s3 dependency as a kubernetes parameter
I submit the session with '
-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.13.0.jar
\
-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=
flink-s3-fs-hadoop-1.13.0.jar', the session can be start, but report error
as below
> Caused by: java.lang.NullPointerException: null uri host.
> at java.util.Objects.requireNonNull(Objects.java:228)
> at
> org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:72)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:467)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234)
> at
> org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123)
>
but I have set the s3 staff in the flink-conf.yaml as below:
> high-availability:
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: s3:///flink-test/recovery
s3.endpoint: http://xxx.yyy.zzz.net
s3.path.style.access: true
s3.access-key: 111111111111111111111111111
s3.secret-key: 222222222222222222222222222
I think I supplied all the s3 information in the flink-conf.yaml, but it
did not work.
I will try other ways to complete the s3 ha on k8s. Thank your guys.
Yours sincerely
Joshua
Robert Metzger <rmetz...@apache.org> 于2021年8月4日周三 下午11:35写道：
> Hey Joshua,
>
> Can you first validate if the docker image you've built is valid by
> running it locally on your machine?
>
> I would recommend putting the s3 filesystem files into the plugins [1]
> directory to avoid classloading issues.
> Also, you don't need to build custom images if you want to use build-in
> plugins [2]
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins
>
> On Wed, Aug 4, 2021 at 3:06 PM Joshua Fan <joshuafat...@gmail.com> wrote:
>
>> Hi All
>> I want to build a custom flink image to run on k8s, below is my
>> Dockerfile content:
>>
>>> FROM apache/flink:1.13.1-scala_2.11
>>> ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/lib
>>> ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/lib
>>>
>> I just put the s3 fs dependency to the {flink home}/lib, and then I build
>> the image and push it to the repo.
>>
>> When I submit the flink session from the custom image, a error will be
>> reported like "exec /docker-entrypoint.sh failed: Exec format error".
>>
>> I googled a lot, but it seems no useful information.
>>
>> Thanks for your help.
>>
>> Yours sincerely
>> Joshua
>>
>
