[
https://issues.apache.org/jira/browse/FLINK-22081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Qin updated FLINK-22081:
-----------------------------
Attachment: image (13).png
> Entropy key not resolved if flink-s3-fs-hadoop is added as a plugin
> -------------------------------------------------------------------
>
> Key: FLINK-22081
> URL: https://issues.apache.org/jira/browse/FLINK-22081
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Reporter: Chen Qin
> Assignee: Prem Santosh
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.11.0, 1.11.1, 1.11.2,
> 1.11.3, 1.11.4, 1.12.0, 1.12.1, 1.12.2, 1.13.0, 1.12.3
>
> Attachments: image (13).png
>
>
> Using flink 1.11.2
> I added the flink-s3-fs-hadoop jar in plugins dir but I am seeing the
> checkpoints paths like
> {{s3://my_app/__ENTROPY__/app_name-staging/flink/checkpoints/e10f47968ae74934bd833108d2272419/chk-3071}}
> which means the entropy injection key is not being resolved. After some
> debugging I found that in the
> [EntropyInjector|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/EntropyInjector.java#L97]
> we check if the given fileSystem is of type {{ClassLoaderFixingFileSystem}}
> and if so we check if the filesysystem is of type
> {{SafetyNetWrapperFileSystem as well as it's delegate }}but don't check for
> {{[ClassLoaderFixingFileSystem|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/PluginFileSystemFactory.java#L65]}}
> directly in getEntorpyFs method which would be the type if S3 file system
> dependencies are added as a plugin.
>
> Repro steps:
> Flink 1.11.2 with flink-s3-fs-hadoop as plugin and turn on entropy injection
> key _entropy_
> observe checkpoint dir with entropy marker not removed.
> s3a://xxx/dev/checkpoints/_entropy_/xenon/event-stream-splitter/jobid/chk-5/
> compare to removed when running Flink 1.9.1
> s3a://xxx/dev/checkpoints/xenon/event-stream-splitter/jobid/chk-5/
>
> Add some logging to getEntropyFs, observe it return null because passed in
> parameter is not {{SafetyNetWrapperFileSystem}} but
> {{ClassLoaderFixingFileSystem}}
>
> Apply patch, build release and run same job
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
