[issue19715] test_touch_common failure under Windows

A.M. Kuchling added the comment: It looks like Antoine filed the precision-loss error as #19738 and that ticket is now closed. I conclude there's nothing left to do on this ticket, so I'll close it. -- nosy: +akuchling status: open -> closed ___ Py

[issue19715] test_touch_common failure under Windows

Mark Lawrence added the comment: #19727 is fixed so do we need issues for "the precision loss in pytime.c" from msg204062 and the analysis in msg204148 ? -- nosy: +BreamoreBoy ___ Python tracker __

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: [MvL] > A. t1=t2=1385161652120375500 > B. pygettimeofday truncates this to 1385161652.120375 > C. time.time() converts this to float, yielding > 0x1.4a3f8ed07b439p+30 i.e. > (0.6450161580556887, 31) > 1385161652.120375 (really .1203749566283776) > D. _PyTime_Ob

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > Antoine, FYI, with the current code test_pathlib passed 500 times in a > row on my box. Success :-) This has been a tedious one :-) I'm gonna open a separate issue for the precision loss in pytime.c, then. -- resolution: -> fixed stage: needs patch

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: Antoine, FYI, with the current code test_pathlib passed 500 times in a row on my box. Success :-) -- ___ Python tracker ___ __

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset b5e9d61f6987 by Antoine Pitrou in branch 'default': Issue #19715: try the utime(..., None) approach again, now that it should be more precise under Windows http://hg.python.org/cpython/rev/b5e9d61f6987 -- __

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: Note that issue #19727 would possibly allow for an easy fix. -- ___ Python tracker ___ ___ Python-bu

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: > AssertionError: 1385161652120374900 not greater than or equal to > 1385161652120375500 > Anything coming from T() would have "000" at the end, not "900" or "500". But "900" *is* "000" :-) A. t1=t2=1385161652120375500 B. pygettimeofday truncates this to 138

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: On sam., 2013-11-23 at 09:53 +, Tim Peters wrote: > I suggest too that's a better way to fix the test than the current > > delta = 1e-6 if os.name == 'nt' else 0 > > dance. Except that it would test something else (i.e. it wouldn't test the st_

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: Martin, I don't see how: > What *can* happen is that ts1 > T(ts2) _in this test_. As shown in many failure examples *both* nanosecond timestamps had non-zero nanoseconds. Like: AssertionError: 1385161652120374900 not greater than or equal to 1385161652120375500

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: Tim, > First, if *both* timestamps went through this code, it's monotonically > non-decreasing. ts1 < ts2 [...] but T(ts1) > T(ts2) can't happen It's as Steve says (and I tried to say before): What *can* happen is that ts1 > T(ts2). ts1 is used by Windows

[issue19715] test_touch_common failure under Windows

Steve Dower added the comment: Or as Martin suggested earlier, time.time() could be returning different values to what the system uses when it creates the file (which I assume is GetSystemTimeAsFileTime/SetFileTime or the kernel-mode equivalent). I only looked briefly at the touch() implementa

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: I have a different theory about this. As explained all over the place, on FAT file creation times are good to 10 milliseconds but modification times only good to 2 seconds. But I can't find one credible word about what the various precisions are for NTFS. For e

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: Steve, I'm afraid sleeping 100ns wouldn't be enough. The more data I collect, the more bizarre this gets :-( Across 300 runs I recorded the difference, in nanoseconds, between the "old" and "new" timestamps. A negative difference is a test failure. I was very s

[issue19715] test_touch_common failure under Windows

Steve Dower added the comment: The "000" or "500" still smells of floating point rounding to me. Windows appears to round-trip the values provided perfectly: import ctypes def do_test(t): h = ctypes.windll.kernel32.CreateFileW("test.txt", 0xC000, 7, None, 2, 0, 0) assert h != -1

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: [MvL] > One "obvious" conversion error is this code from > > http://hg.python.org/cpython/file/4101bfaa76fe/Python/pytime.c#l35 > > microseconds = large.QuadPart / 10 - 116444736; > > This discard the 100ns part of the current time, rounding it > down to the

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: One "obvious" conversion error is this code from http://hg.python.org/cpython/file/4101bfaa76fe/Python/pytime.c#l35 microseconds = large.QuadPart / 10 - 116444736; This discard the 100ns part of the current time, rounding it down to the previous mic

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: I thought that using utime(..., None) would be better, but it's actually much worse: it calls GetSystemTime, which only has millisecond precision, while time.time() calls GetSystemTimeAsFileTime which has better precision. (this is probably a bug in utime())

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset a273f99159e0 by Antoine Pitrou in branch 'default': Revert utime(..., None) strategy (it has too poor resolution under Windows) and restore the previous test workaround http://hg.python.org/cpython/rev/a273f99159e0 -- _

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset d647a4a8505e by Antoine Pitrou in branch 'default': Trying other strategy for #19715: use utime(..., None) http://hg.python.org/cpython/rev/d647a4a8505e -- ___ Python tracker

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: OTOH, a time.time() in units of 100ns needs 53.6 bits, so it should "almost" fit. -- ___ Python tracker ___ __

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: Looking at one of the failure instances: >>> math.frexp(1385150397.383464)[0] * 2**53 5809741852347053.0 >>> math.frexp(1385150397.3834648)[0] * 2**53 5809741852347056.0 It doesn't seem that lack of FP precision would be sufficient to explain the problem, alth

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: Looking at the ReactOS sources, it appears that Windows doesn't do any rounding, so it's all Python's fault. Windows keeps its system time (in KI_USER_SHARED_DATA) in essentially a FILETIME represenation (i.e. units of 100ns since 1601), and never needs to co

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset 602062d2a008 by Antoine Pitrou in branch 'default': Try to fix issue #19715 (timestamp rounding inconsistencies under Windows?) http://hg.python.org/cpython/rev/602062d2a008 -- ___ Python tracker

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > No, I really meant step 4 (touch). The second touch call *really* > steps backwards. I don't find that very surprising, since it is > actually the first call that passes an actual time. Ok, now I see what you mean. Indeed, we are calling utime() on the file w

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: >> Then, the utime call in step 4 asked to set this to > You mean step 3, right? ("date back 10s") No, I really meant step 4 (touch). The second touch call *really* steps backwards. I don't find that very surprising, since it is actually the first call that

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset 716e41100553 by Victor Stinner in branch 'default': Issue #19715: Ensure that consecutive calls to monotonic() are monotonic http://hg.python.org/cpython/rev/716e41100553 -- ___ Python tracker

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: Here's a failure with the patch: self.assertGreaterEqual(st.st_mtime_ns, old_mtime_ns) AssertionError: 1385161652120374900 not greater than or equal to 1385161652120375500 And another: AssertionError: 1385161754170484000 not greater than or equal to 1385161754

[issue19715] test_touch_common failure under Windows

Steve Dower added the comment: > 7. verify original_mtime - 0.001 < p.stat().st_mtime < original_mtime + 0.001 Actually, don't check the upper-bound here... that's a bad idea :) -- ___ Python tracker _

[issue19715] test_touch_common failure under Windows

Steve Dower added the comment: I don't have any extra insight into this. The documented resolution for mtime on NTFS is 100ns (2s on FAT32), so without delaying by at least that long you're not going to see an official change. The noise is probably from floating-point conversions (in _PyTime_O

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > 3. Python uses an insane (non-monotonic) scheme to convert the time > to a float. Yes, I was at least hoping to clear that out. The code is there: http://hg.python.org/cpython/file/4101bfaa76fe/Modules/posixmodule.c#l1459 Other than that, I suppose I can hack

[issue19715] test_touch_common failure under Windows

Tim Peters added the comment: FYI, the test fails on my box (32-bit Windows Vista) about 1 time in 3. Here's the latest failure: AssertionError: 1385160333.612968 not greater than or equal to 1385160333.6129684 And here's another: AssertionError: 1385160530.348423 not greater than or equal

[issue19715] test_touch_common failure under Windows

Roundup Robot added the comment: New changeset 4101bfaa76fe by Antoine Pitrou in branch 'default': Try to debug issue #19715 http://hg.python.org/cpython/rev/4101bfaa76fe -- nosy: +python-dev ___ Python tracker ___

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: To recap a bit, here is a very simplified view of the test: A. path is opened for writing (and then closed) B. its st_mtime is recorded in old_mtime C. path is opened again for writing (and closed) D. assert `path's current mtime` >= old_mtime Whatever the deta

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > I think I found the problem. In one run, the current time (as reported > by time.time()) was > > 1385154213.291315 > > On the first touch call (in step 1), the file is not there, and gets > its current time internally from the system (not sure which part > ex

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: I think I found the problem. In one run, the current time (as reported by time.time()) was 1385154213.291315 On the first touch call (in step 1), the file is not there, and gets its current time internally from the system (not sure which part exactly assigns

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > IIUC, the sequence of events is this: > 1. touch > 2. read old_mtime > 3. date back 10s > 4. touch > 5. read mtime > > So the time stamp that is set in step 3 is never read, correct? So > there is no test that it is newer than the 10s-old-stamp, but only > new

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: IIUC, the sequence of events is this: 1. touch 2. read old_mtime 3. date back 10s 4. touch 5. read mtime So the time stamp that is set in step 3 is never read, correct? So there is no test that it is newer than the 10s-old-stamp, but only newer then the recen

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: > I don't quite understand the test: why is it dating back the file by > 10s, but then not using the dated-back time stamp for anything? Well, it does, unless I'm misunderstanding your question: self.assertGreaterEqual(p.stat().st_mtime, old_mtime) Thi

[issue19715] test_touch_common failure under Windows

Martin v. Löwis added the comment: I don't quite understand the test: why is it dating back the file by 10s, but then not using the dated-back time stamp for anything? -- nosy: +loewis ___ Python tracker _

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: And indeed, the test seems to pass on another Windows buildbot: http://buildbot.python.org/all/builders/x86%20Windows%20Server%202008%20%5BSB%5D%203.x (note: my VM is under Windows 7) -- ___ Python tracker

[issue19715] test_touch_common failure under Windows

Antoine Pitrou added the comment: Now that I think, I didn't encounter the failure when testing in my own Windows VM... uh. -- nosy: +brian.curtin, tim.golden ___ Python tracker ___

[issue19715] test_touch_common failure under Windows

New submission from Antoine Pitrou: Path.touch() doesn't seem to change the mtime under Windows, which leads to the following failure: == FAIL: test_touch_common (test.test_pathlib.PathTest)