[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat
John S. Gruber added the comment: The patch worked--it eliminated the redundant copy caused by this issue. Thank you very much, Éric. On Fri, Jun 3, 2011 at 12:39 PM, Éric Araujo wrote: > > Éric Araujo added the comment: > > Can you test this patch? > ___ > Python tracker > <http://bugs.python.org/issue11933> > ___ > -- title: newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME] -> newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
New submission from John S. Gruber : In researching a bug I was surprised that a newly created file was being replaced when being processed a second time (it shouldn't have been processed a second time). I tracked the surprise to diff Lib/distutils/dep_util.py @ 57642:9211a5d7d0b4 where a comparison with a resolution of 1 second was replaced by a floating point comparison, though the new file was copied by file_util.py which tried to preserve the time using a method accurate to only 1 second (truncating the fraction). Since a new file with a truncated modification time looks older than an older file with a full precision stamp the problem resulted. (For convenience--stat.st_mtime is floating point, stat[ST_MTIME] is integer seconds unless os.stat_float_times has been called to change the floating point behavior. Using all floating point doesn't resolve the issue as OS timestamped files can have more resolution than python floating point may hold, again causing truncation and confusion. For reference see Python issue 10148. In my humble opinion time setting and comparison should all be done consistently, and, if sub-second comparisons are desired, some fuzz should be used such that the comparison makes sense for the platform with the least amount of precision available with its floating point implementation. I briefly looked at branches other than 2.7 and they don't seem to have the above change. Probably of minor importance in most cases. Thanks all for the good work, all. I've been learning python and I love it! -- assignee: tarek components: Distutils messages: 134526 nosy: eric.araujo, jsjgruber, tarek priority: normal severity: normal status: open title: newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME] type: behavior versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
John S. Gruber added the comment: Thanks for considering my report so quickly. Attached is a simple test to reproduce the bug, as you suggested. Please note that I am not suggesting the code base use stat.st_mtime. Running the attached with ext4, which keeps sub-second file timestamps: gruber@gruber-Satellite-L355D:~$ ~/setuptest running build running build_py creating build creating build/lib.linux-i686-2.7 copying charley.py -> build/lib.linux-i686-2.7 running build running build_py copying charley.py -> build/lib.linux-i686-2.7 Under an ext3 filesystem which only accounts to the nearest second: gruber@gruber-Satellite-L355D:~$ pushd /media/EXT3/ /media/EXT3 ~ gruber@gruber-Satellite-L355D:/media/EXT3$ ~/setuptest running build running build_py copying charley.py -> build/lib.linux-i686-2.7 running build running build_py You can see under ext3 the newer() function of dep_util.py works correctly and the redundant copy of charley.py to the build directory, when setup.py is run a second time, is skipped. Above, with ext4, the mtime comparison was confused. As I said in the report, not a big problem--but it obviously isn't what was intended so I hoped a problem report would be worthwhile to the distutils developers. Thanks. -- Added file: http://bugs.python.org/file21799/setuptest ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
John S. Gruber added the comment: The original bug report is at: https://bugs.launchpad.net/ubuntu/+source/python-distutils-extra/+bug/770566 As you can see it had to do with a symbolic link created by distutils-extra before distutils was called upon to copy anything. Since this was done behind distutils back, LP: 770566 really was a distutils-extra problem and Martin's fix means this issue no longer arises there. To recreate the problem in that environment distutils-extra had to be modified to coerce processing to happen in a particular order so setup.py would complete, and then results compared between an ext4 run and the build farm's presumably ext3 run. In the ext4 case the symbolic link was predictably replaced by the file itself. The test case I posted above should be easier to follow and doesn't involve any smoke and mirrors being done behind distutils back! Please let me know if I provide any further information. -- ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
John S. Gruber added the comment: I can understand what you are saying about side effects. I was trying to suggest that the move to stat.st_mtime in dep_util.py in the hg commit I mentioned be reverted back to use stat[ST_MTIME], thereby matching the other python releases and the old behavior (and matching file_util.py). It seems to me that would be the safest course when it comes to side-effects. If it were desired to determine which file was newer using sub-second values, perhaps that would make a reasonable change in distutils2, but files created with a few microseconds would have to be considered equivalent due to the reduced precision available in python floats (53 bits on my platform, if I understand correctly) as compared to the 64 bit precision used by some operating systems and file systems. However, I think I'd prefer to turn the decision and further process over to you, if you don't mind. I thank you and your colleagues for your excellent work with python, and with making it platform independent--a very difficult part of the work indeed. -- ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
John S. Gruber added the comment: As I thought about it, I suppose I should demonstrate the problem with stat.st_mtime. Here's an example and its output on an ext4 file system: gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933305.5525582, 1303933305.5525582) (1303933305.552558, 1303933305.552558) (1303933305.552557, 1303933305.552557) (1303933305.552556, 1303933305.552556) (1303933305.5525581837 1303933305.5525581837) (1303933305.5525579453 1303933305.5525579453) (1303933305.5525569916 1303933305.5525569916) (1303933305.5525560379 1303933305.5525560379) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:45.552558258 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:45.552558000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:45.552557000 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:45.552556000 -0400 target3 Traceback (most recent call last): File "./mtime.py", line 28, in assert timetargetthree == timetargettwo AssertionError gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933306.3885624, 1303933306.3885624) (1303933306.388562, 1303933306.388562) (1303933306.388561, 1303933306.388561) (1303933306.388561, 1303933306.388561) (1303933306.3885624409 1303933306.3885624409) (1303933306.3885619640 1303933306.3885619640) (1303933306.3885610104 1303933306.3885610104) (1303933306.3885610104 1303933306.3885610104) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:46.388562342 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:46.388562000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:46.388561000 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:46.388561000 -0400 target3 Traceback (most recent call last): File "./mtime.py", line 29, in assert timetargettwo == timetargetone AssertionError gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933307.0525656, 1303933307.0525656) (1303933307.052565, 1303933307.052565) (1303933307.052565, 1303933307.052565) (1303933307.052565, 1303933307.052565) (1303933307.0525655746 1303933307.0525655746) (1303933307.0525650978 1303933307.0525650978) (1303933307.0525650978 1303933307.0525650978) (1303933307.0525650978 1303933307.0525650978) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:47.052565592 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:47.052565000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:47.052565000 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:47.052565000 -0400 target3 Traceback (most recent call last): File "./mtime.py", line 30, in assert timetargetone == timeone AssertionError gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933308.2285714, 1303933308.2285714) (1303933308.228571, 1303933308.228571) (1303933308.22857, 1303933308.22857) (1303933308.228569, 1303933308.228569) (1303933308.2285714149 1303933308.2285714149) (1303933308.2285709381 1303933308.2285709381) (1303933308.2285699844 1303933308.2285699844) (1303933308.2285690308 1303933308.2285690308) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:48.228571344 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:48.228571000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:48.22857 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:48.228569000 -0400 target3 Traceback (most recent call last): File "./mtime.py", line 28, in assert timetargetthree == timetargettwo AssertionError gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933309.0685754, 1303933309.0685754) (1303933309.068575, 1303933309.068575) (1303933309.068574, 1303933309.068574) (1303933309.068573, 1303933309.068573) (1303933309.0685753822 1303933309.0685753822) (1303933309.0685749054 1303933309.0685749054) (1303933309.0685739517 1303933309.0685739517) (1303933309.0685729980 1303933309.0685729980) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.068575446 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.068575000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.068574000 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.068573000 -0400 target3 Traceback (most recent call last): File "./mtime.py", line 28, in assert timetargetthree == timetargettwo AssertionError gruber@gruber-Satellite-L355D:~$ ./mtime.py (1303933309.6765785, 1303933309.6765785) (1303933309.676578, 1303933309.676578) (1303933309.676578, 1303933309.676578) (1303933309.676578, 1303933309.676578) (1303933309.6765785217 1303933309.6765785217) (1303933309.6765780449 1303933309.6765780449) (1303933309.6765780449 1303933309.6765780449) (1303933309.6765780449 1303933309.6765780449) total 0 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.676578416 -0400 one -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.676578000 -0400 target1 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.676578000 -0400 target2 -rw-r--r-- 1 gruber gruber 0 2011-04-27 15:41:49.676578000 -0400 target3 Traceback (m
[issue11993] Is it desired to distinguish new files from old with sub-second resolution?
New submission from John S. Gruber : This report is meant to prompt discussion, if desired, on the advisability of distinguishing new files from old using subsecond data. (It isn't clear to me that it is important to do this.) Some file systems keep sub-second modification times, but the number of seconds since the epoch grows to a very large number, and given the limited number of significant bits in floating point numbers, it's important to carry out this comparison carefully (or use some new integer data) so newly created files don't appear to be older than their source files due to rounding and other conversion anomalies. Current floats don't have the precision to hold both the number of seconds since the epoch and a nanosecond precision fractional second. -- assignee: tarek components: Distutils2 messages: 135090 nosy: alexis, eric.araujo, jsjgruber, tarek priority: normal severity: normal status: open title: Is it desired to distinguish new files from old with sub-second resolution? type: feature request versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue11993> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11993] Is it desired to distinguish new files from old with sub-second resolution?
John S. Gruber added the comment: Thanks for writing so quickly. This topic arose as part of issue 11933 (similar number). Éric Araujo asked that I bring this up in either mail to the fellowship of the packaging or as a separate bug report, so I chose to file this. Please see msg 134634 from that issue. I suppose Éric may believe sub-second time comparison is a possibility for distutils2. I think *if* it is done the integer/float conversion implication should be considered, given my limited experience with the topic in distutils.i Thanks again. -- ___ Python tracker <http://bugs.python.org/issue11993> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11933] newer() function in dep_util.py mixes up new vs. old files due stat.st_mtime vs stat[ST_MTIME]
John S. Gruber added the comment: As Éric suggested I opened issue 11993 a couple of days ago. -- ___ Python tracker <http://bugs.python.org/issue11933> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
