[
https://issues.apache.org/jira/browse/FLINK-40627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121966#comment-18121966
]
Robert Metzger commented on FLINK-40627:
----------------------------------------
{quote}[~rmetzger] If I get the artifact fetchers to fetch a sub-directory
derived from the URI (including query parameters) and metadata from the
FileStatus (e.g. last-modified, file length) this would write to
{{baseDir/some-hash-for-first-uri-and-file/test.jar}} the first time and
{{baseDir/some-hash-for-second-uri-and-file/test.jar}}
The only downside I see is that unless we also add something to clean up old
jars, we potentially end up stacking up multiple old copies of jars every time
we switch to a new jar.
{quote}
I think that downside is acceptable – let's go with your proposal, and keep it
simple, no smarter cache.
> Partially fetched user artifacts are reused on subsequent starts
> ----------------------------------------------------------------
>
> Key: FLINK-40627
> URL: https://issues.apache.org/jira/browse/FLINK-40627
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
> Reporter: Dale Lane
> Assignee: Dale Lane
> Priority: Minor
>
> {{ArtifactFetchManager.fetchArtifact}} returns any file already present at
> the target path without re-fetching ("Already fetched user artifacts are
> kept"). {{HttpArtifactFetcher}} and {{FsArtifactFetcher}} write directly to
> that final path via {{{}FileUtils.copyToFile{}}}, so if a transfer ends part
> way through, a truncated file is left at the final path. Nothing cleans up
> files from the artifacts directory.
> A transfer can end part way through in several ways, all of which leave a
> truncated file:
> * fetch fails with an exception (e.g. connection reset)
> * Job Manager process is killed mid-transfer (e.g. OOMKilled, liveness-probe
> kill in Kubernetes)
> * an HTTP server without a {{Content-Length}} ends the response early (the
> fetch then reports success, and the job fails on its first start as well as
> every later one)
> Any later start that sees the same directory reuses the truncated file and
> never retries the download. The job fails on every start with no fetch
> traffic:
> {{org.apache.flink.client.program.ProgramInvocationException: Error while
> opening jar file '.../job.jar'}}
> {{Caused by: java.io.IOException: Error while opening jar file '.../job.jar'}}
> {{Caused by: java.util.zip.ZipException: zip END header not found}}
> This affects:
> * Native Kubernetes Application Mode: the artifacts dir
> ({{{}<user.artifacts.base-dir>/<namespace>/<cluster-id>{}}}) is an
> {{{}emptyDir{}}}, which survives Job Manager container restarts within a pod,
> so the pod is stuck until it is deleted. With a persistent volume or
> {{hostPath}} mounted at {{{}base-dir{}}}, it is stuck permanently.
> * Standalone Application Mode with a persistent
> {{{}user.artifacts.base-dir{}}}.
> Clearing it today needs manual deletion of the file (or pod deletion for
> {{{}emptyDir{}}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
