[issue16936] Documentation for stat.S_IFMT inconsistent
New submission from Jens Lechtenboerger: The documentation for the stat module is inconsistent (Doc/library/stat.rst, at least for Python 2.7.2 and 3.3.0): It talks about a function stat.S_IFMT() and a bit mask stat.S_IFMT. Only the former does exist. Besides, it states: "For complete details about the stat(), fstat() and lstat() calls, consult the documentation for your system." I suggest to add some pointers on what systems one might consult what documentation: "(e.g., on GNU/Linux invoke 'man 2 stat')" I don't know about other systems, though. So, doing "man 2 stat", which refers to the POSIX standard and which seems to have served as blueprint for stat's interface, I expected to find the bit mask S_IFMT. However, that does not exist. In contrast, in stat.py that bit mask is hard-coded as 0o17 in the definition of S_IFMT(). As long-term, backwards-incompatible fix, I suggest to export S_IFMT = 0o17 and to rename the function S_IFMT(). That way, stat would in fact be aligned with the documentation for my system. As short-term fix, I suggest to correct stat.rst. Replace > Use of the functions above is more portable than use of the first > set of flags: > > stat.S_IFMT >Bit mask for the file type bit fields. with > Use of the functions above may be more portable than use of the > first set of flags. > > Warning: Note that the stat module does not export a bit mask > S_IFMT. (As stated incorrectly in previous versions of the > documentation.) Here, I replaced "is more portable" with "may be more portable" as the constant 0o17 is hard-coded into S_IFMT() in stat.py. Maybe somebody could add a hint in what sense portability is improved? Alternatively that sentence could be deleted. A diff for stat.rst (Python 3.3.0) is attached. -- assignee: docs@python components: Documentation files: stat.rst.diff keywords: patch messages: 179687 nosy: docs@python, lechten priority: normal severity: normal status: open title: Documentation for stat.S_IFMT inconsistent type: behavior versions: Python 3.3 Added file: http://bugs.python.org/file28697/stat.rst.diff ___ Python tracker <http://bugs.python.org/issue16936> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
New submission from Jens Lechtenboerger:
Maybe I'm misreading the documentation of multiprocessing.Value and
multiprocessing.sharedctypes.Value.
I thought that access to the value field of Value instances was protected by
locks to avoid lost updates.
Specifically, for multiprocessing.Value(typecode_or_type, *args[, lock]) and
multiprocessing.sharedctypes.Value(typecode_or_type, *args[, lock]) the
documentation states:
> By default the return value is actually a synchronized wrapper for the
> object. [...]
> If lock is True (the default) then a new lock object is created to
> synchronize access to the value. If lock is a Lock or RLock object then that
> will be used to synchronize access to the value. If lock is False then
> access to the returned object will not be automatically protected by a lock,
> so it will not necessarily be “process-safe”.
(By the way, I'm not sure why both, multiprocessing.Value and
multiprocessing.sharedctypes.Value are documented. They appear to be the same
thing.)
The following tests (also attached as file) show that lost updates may occur
if several instances of multiprocessing.Process increment the same Value that
is passed as args parameter.
def do_inc(integer):
"""Increment integer.value for multiprocessing.Value integer."""
integer.value += 1
def do_test(notasks):
"""Create notasks processes, each incrementing the same Value.
As the Value is initialized to 0, its final value is expected to be
notasks.
"""
tasks = list()
integer = multiprocessing.sharedctypes.Value("i", 0)
for run in range(notasks):
proc = multiprocessing.Process(target=do_inc, args=(integer,))
proc.start()
tasks.append(proc)
for proc in tasks:
proc.join()
if integer.value != notasks:
logging.error(
"Unexpected value: %d (expected: %d)", integer.value, notasks)
if __name__ == "__main__":
do_test(100)
Sample invocations and results:
Note that on a single CPU machine the error is not reported for every
execution but only for about every third run.
$ python --version
Python 2.6.5
$ uname -a
Linux ubuntu-desktop 2.6.32.11+drm33.2 #2 Fri Jun 18 20:30:49 CEST 2010 i686
GNU/Linux
$ python test_multiprocessing.py
ERROR:root:Unexpected value: 99 (expected: 100)
On a quadcore, the error occurs almost every time.
$ uname -a
Linux PC 2.6.35.13 #4 SMP Tue Dec 20 15:22:02 CET 2011 x86_64 GNU/Linux
$ ~/local/Python-2.7.3/python test_multiprocessing.py
ERROR:root:Unexpected value: 95 (expected: 100)
$ ~/local/Python-3.3.0/python test_multiprocessing.py
ERROR:root:Unexpected value: 86 (expected: 100)
--
components: Library (Lib)
files: test_multiprocessing.py
messages: 180251
nosy: lechten
priority: normal
severity: normal
status: open
title: Lost updates with multiprocessing.Value
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file28782/test_multiprocessing.py
___
Python tracker
<http://bugs.python.org/issue16998>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
Jens Lechtenboerger added the comment: > Loads and stores are both atomic. But "+=" is made up of two operations, a > load followed by a store, and the lock is dropped between the two. I see. Then this is a documentation bug. The examples in the documentation use such non-thread-safe assignments (combined with the statement "These shared objects will be process and thread safe."). -- ___ Python tracker <http://bugs.python.org/issue16998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
Jens Lechtenboerger added the comment: > It only says that accesses are synchronized. The problem is that you were > assuming that "+=" involves a single access -- but that is not how python > works. Yes, I understand that by now (actually since your first comment). > Where in the examples is there "non-process-safe" access? (Note that waiting > for the only process which modifies a value to terminate using join() will > prevent races.) In section "The multiprocessing.sharedctypes module" the assignments in the first example (function modify()) are unsafe. None of them is protected by a lock as suggested in your first comment. Strictly speaking, no lock is necessary in the example as there are no race conditions (the processes work in an alternating fashion without concurrency). I certainly did not see that the example (for a *shared* memory data structure) is based on the implicit assumption of a single writer. In contrast, I assumed that some "magic" should offer process-safe usage of "+=", which made me file this bug. That assumption has turned out to be wrong. To prevent others from being mislead in the same way I suggest to either protect those operations with locks (and comment on their effect) or to state the implicit assumption explicitly. Maybe add the following after "Below is an example where a number of ctypes objects are modified by a child process:" Note that assignments such n.value **= 2 are not executed atomically but involve two operations, a load followed by a store. Each of these operations is protected by the Value's lock, which is dropped in between. Thus, in scenarios with concurrent modifying processes you may want to explicitly acquire the Value's lock to ensure atomic execution (avoiding race conditions and lost updates), e.g.: with n.get_lock(): n.value **= 2 -- ___ Python tracker <http://bugs.python.org/issue16998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
