[
https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16644.
-------------------------------------
Resolution: Won't Fix
Update: put result doesn't include timestamp. unless we do a HEAD every time,
this is in trouble.
I'm going to close the patch as is. If we were to revisit this it would be
needed for every file and I'd make an option for some special cases (Yarn job
submission) where timestamp mismatch is a blocker
> Retrive modtime of PUT file from store, via response or HEAD
> ------------------------------------------------------------
>
> Key: HADOOP-16644
> URL: https://issues.apache.org/jira/browse/HADOOP-16644
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Environment: -Dparallel-tests -DtestsThreadCount=8
> -Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale
> h2. Hypothesis:
> the timestamp of the source file is being picked up from S3Guard, but when
> the NM does a getFileStatus call, a HEAD check is made -and this (due to the
> overloaded test system) is out of sync with the listing. S3Guard is updated,
> the corrected date returned and the localisation fails.
> Reporter: Steve Loughran
> Priority: Major
>
> Terasort of directory committer failing in resource localisaton -the
> partitions.lst file has a different TS from that expected
> Happens under loaded integration tests (threads = 8; not standalone);
> non-auth s3guard
> {code}
> 2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:processHeartbeat(1150)) - {
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst,
> 1570531828143, FILE, null } failed: Resource
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst
> changed on src filesystem (expected 1570531828143, was 1570531828000
> java.io.IOException: Resource
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst
> changed on src filesystem (expected 1570531828143, was 1570531828000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
