Light (general) Inter-Process Mutex/Wait/Notify Synchronization?
I am interested in the lightest mechanism to use in a Python application for mutex/wait/notify between processes, one of which may be a program which does not have a shared ancestral benefactor, or may not be a Python program at all. I would ideally like to have something which does not need to make a system call in an uncontended case for the mutex. In C on Linux I would use mmap to get shared memory, then pthread_cond_wait with a cond/mutex which has the PTHREAD_PROCESS_SHARED attribute set. For pthread_cond_wait, it would of course be fine to make a system call. I am familiar with the "multiprocessing" package for the case of a closely related set of Python programs. I am looking for something where the other side is not a Python application using "multiprocessing". A thin layer over pthreads would be nice, or which is using POSIX semaphores/queues would be ok. I can use mmap for shared memory; that is not problem. The synchronization is where I need help. -- http://mail.python.org/mailman/listinfo/python-list
Re: Light (general) Inter-Process Mutex/Wait/Notify Synchronization?
> Try this: http://nikitathespider.com/python/shm/ I took a look at that (especially the posix_ipc at http://semanchuk.com/philip/posix_ipc/). I am hoping not to plug something underneath the Python VM; I would rather use a socket, or use signals. If I were to use a C library, I imagine I would just go with a thin layer on top of pthread_*, but I think I will prefer something which uses system calls and is bundled with Python, slow as that approach may be. -- http://mail.python.org/mailman/listinfo/python-list
Re: Light (general) Inter-Process Mutex/Wait/Notify Synchronization?
> If you don't want to use a 3rd party module you could > use the multiprocessing module That is definitely good for when I have a tree of processes which are all Python applications. I use it for that. But I am looking for something where the Python application can interact conveniently with an arbitrary external application. Using a socket/pipe and shared memory is convenient, but not feasible when system call overhead matters. -- http://mail.python.org/mailman/listinfo/python-list
Re: Light (general) Inter-Process Mutex/Wait/Notify Synchronization?
> > > Try this: http://nikitathespider.com/python/shm/ > > > > I am hoping not to plug something underneath the Python > > VM; I would rather use a socket, or use signals. > > I'm not sure what you mean. It's just an extension module > that you'd import like any of the stdlib modules. I cannot import it unless a corresponding extension implementation is plugined underneath the Python VM. > Certainly anything involving sockets or signals is > going to be a lot slower. Indeed, but I can "import signal" or "import socket" wherever I go, without worrying about requiring (or worse yet, having to bundle) a CPython plugin (presuming I am even running on top of CPython). I am hoping for something which does not require a context switch (or worse) for every wait/notify. -- http://mail.python.org/mailman/listinfo/python-list
Re: Light (general) Inter-Process Mutex/Wait/Notify Synchronization?
> Linux doesn't do interprocess communication very well. > The options are [...] and shared memory (unsafe). I think the bar has to be set pretty high to say shared memory is too unsafe an approach for active entities to communicate. > If you're using CPython, don't worry about socket overhead. > CPython is so slow you'll never notice it. It is not so much the socket overhead as the context switch. If processes could send data to each other through sockets without making system calls, that would be great. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to walk up parent directories?
Be careful not to presume "/a/b/" is the same directory as "/a/b/c/../" I would consider the latter the parent of /a/b/c/, not the former. -- http://mail.python.org/mailman/listinfo/python-list
x.abc vs x['abc']
Presuming it is very common to have objects created
on the fly using some sort of external data
definitions, is there an obvious common standard
way to take a dict object and create an object
whose attribute names are the keys from the dict?
I realize I can do something like:
>>> d = {"hello": "world"}
>>> x = type("", (object,), d)()
>>> x.hello
world
but that seems like an arcane way to do something
which would ideally be transparent... if there is
a function in the standard library, that would be
good, even if I have to import it. I guess there is
collections.namedtuple... that would not look much
prettier... but the main thing to me is for it to
be the same way everybody else does it. I do not
prefer the new object be a dict, but it would be ok.
--
http://mail.python.org/mailman/listinfo/python-list
Block-Local Variables using "with"
Presuming there is a reason to want block-local variables,
does this seem like a good way to do something like it?
@contextlib.contextmanager
def blocklocal(**kwargs):
bl = type('', (object,), {})()
for (k, v) in kwargs.items():
bl.__setattr__(k, v)
yield bl
for k in bl.__dict__.copy():
bl.__delattr__(k)
with blocklocal(a=12, b="hello") as bl:
bl.c = "world"
print(bl.a, bl.b, bl.c)
The "bl" variable would still be there but empty.
Are there cleaner ways? (without nested "def"s)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Context manager, atexit processing, and PEP 3143 DaemonContext.close
> Anyone else? If there is a function which triggers a one-shot switch, I like to have a way to find out if it has already been triggered, I prefer to have the function tell me if it triggered the switch or not, but I would not want that to be by raising an exception. -- http://mail.python.org/mailman/listinfo/python-list
Re: Context manager, atexit processing, and PEP 3143 DaemonContext.close
> > If there is a function which triggers a one-shot switch, I like
> > to have a way to find out if it has already been triggered, I
> > prefer to have the function tell me if it triggered the switch
> > or not, but I would not want that to be by raising an exception.
>
> In this case, though, we're talking about a class intended to
> implement the context manager API (in addition to other API).
>
> So, following your argument above, we might expect a generic
> context manager object to be able to tell us whether we're
> currently inside that context or not. Does such a thing exist?
By a "one-shot switch" I meant as something which
changes once atomically in isolation, not as
something whose state comprises a transaction during
which the state of multiple externally visible
objects might change.
I imagine if there is an object used as a context
manager whose context state ("currently inside
that context or not") is not strictly under the
control of the single "with" statement where it is
fulfilling the "context manager" contract, would not
be a generic context manager.
In other words, to me it seems like the concept of
"currently inside a context or not" is not something
visible external to the object except in a case
where there is another thread of execution, which is
not a case I would expect to be part of the generic
context manager protocol, but can be added by an
object which implements the context manager protocol
and also keeps track of its state by setting flags
in the __enter__ and __exit__ functions.
Does this make sense?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Performance java vs. python
My experience has been that if the execution stays inside the VM, then for a "server side" application, the JVM is faster, and proportionally even faster when there are more threads ready to do something. When the VM has to do a lot of interaction with the OS, then I think it is difficult to make general statements about Java vs Python, because there is so much dependency on how a particular library interacts with the operating system or with a remote server. For example a database driver which is doing a lot of conversion between the data representation of the database and the native representation of the VM may chew up more time than anything else, and for CPython that driver may be doing all that stuff in a C library where the Java version of the driver may be doing it all in the JVM and taking an order of magnitude longer. It seems to me like the guts of the JVM and CPython VMs are pretty comparable in terms of performance, and both languages are amenable to taking advantage of the performance of the VM. Other than the library implementations for interfacing with the outside world, for me the major performance distinction is with concurrency. I tend to implement solutions which rely pretty heavily on shared objects and the java.util.concurrent facilities, and it seems to me like the CPython VM is not able to easily provide the level of simple clean shared object concurrency support the JVM does. And my experience with the Python "multiprocessing" module is that while it is better than nothing, it does not seem to facilitate nearly the level of efficient shared object support the JVM does. -- http://mail.python.org/mailman/listinfo/python-list
Re: What text editor is everyone using for Python
> This, ladies and gentlemen of the jury, points > up the essential difference between a modal and > a non-modal way of doing things. That is one reason I love emacs... I not only get a selection of several "major modes", I can also have multiple "minor modes" active at the same time! And I can extend it with a pure language like LISP rather than something like Vim which tends to prefer Python for extension. -- http://mail.python.org/mailman/listinfo/python-list
Re: preferring [] or () in list of error codes?
> [In this tuple] >dodge_city = (1781, 1870, 1823) >(population, feet_above_sea_level, establishment_year) = dodge_city > each index in the sequence implies something very > different about each value. The semantic meaning > of each index is *more* than just the position in > the sequence; it matters *for interpreting that > component*, and that component would not mean the > same thing in a different index position. A tuple > is the right choice, for that reason. I think I would have difficulty holding a position that this should not be a class (or equivalent via namedtuple()) or a dict. It seems to me like a case could be made that there are far more situations where it makes sense to use tuples as immutable sequences than as objects whose attributes are named implicitly by an index. This dodge_city definitely does not seem to me like a good candidate for a plain tuple. -- http://mail.python.org/mailman/listinfo/python-list
Re: preferring [] or () in list of error codes?
> Try, then, this tuple: > >event_timestamp = (2009, 06, 04, 05, 02, 03) >(year, month, day, hour, minute, second) = event_timestamp > > A list would be wrong for this value, because each position in the > sequence has a specific meaning beyond its mere sequential position. Yet > it also matters to the reader that these items are in a specific > sequence, since that's a fairly standard ordering for those items. > > In this case, a tuple is superior to a list because it correctly conveys > the semantic meaning of the overall value: the items must retain their > sequential order to have the intended meaning, and to alter any one of > them is conceptually to create a new timestamp value. I totally agree about anything to do with immutability, I think the relative ordering of the elements in this example may be orthogonal to the concept of a tuple as an object whose elements have a semantic meaning implicitly defined by location in the sequence... in other words knowing that element i+1 is in some sense ordinally smaller than element i does not give me much information about what element i+1 actually is. To me a timestamp could be (date, time), or (days, seconds, microseconds) (as in datetime.timedelta()), so it is not clear to me that using a tuple as something where the semantic meaning of the element at position i should readily apparent would be the best approach for timestamps, or enough to distinguish list and tuple (in other words I am not suggesting a dict or class). In the case of something like (x, y) or (real, imag), or (longitude, latitude), or any case where there is common agreement and understanding, such that using names is arguably superfluous... I think in those cases the concept makes sense of a tuple as a sequence of attributes whose elements have a semantic meaning implicitly defined by position in the sequence. My feeling is the number of cases where tuples are better than lists for that is small relative to the number of cases where tuple adds value as an immutable list. I do not mean to be suggesting that a tuple should only ever be used or thought of as a "frozenlist" though. -- http://mail.python.org/mailman/listinfo/python-list
Re: preferring [] or () in list of error codes?
> > >event_timestamp = (2009, 06, 04, 05, 02, 03) > > >(year, month, day, hour, minute, second) = event_timestamp > > > > [...] > > The point of each position having a different semantic meaning is that > tuple unpacking works as above. You need to know the meaning of each > position in order to unpack it to separate names, as above. > > So two tuples that differ only in the sequence of their items are > different in meaning. This is unlike a list, where the sequence of items > does *not* affect the semantic meaning of each item. I do not feel the above is significantly different enough from event_timestamp = [2009, 06, 04, 05, 02, 03] (year, month, day, hour, minute, second) = event_timestamp event_timestamp = (2009, 06, 04, 05, 02, 03) (year, month, day, hour, minute, second) = event_timestamp event_timestamp = [2009, 06, 04, 05, 02, 03] [year, month, day, hour, minute, second] = event_timestamp to suggest tuples are really adding significant value in this case, especially when I can do something like event_timestamp = (2009, 06, 04, 05, 02, 03) (year, month, day, hour, second, minute) = event_timestamp and not have any indication I have done the wrong thing. I guess to me, fundamentally, the interpretation of tuple as a sequence whose elements have semantic meaning implicitly defined by position is a relatively abstract intrepretation whose value is dubious relative to the value of immutability, since it seems like a shortcut which sacrifices explicitness for the sake of brevity. I would feel differently if seemed unusual to find good Python code which iterates through the elements of a tuple as a variable length homogenous ordered collection. But then I would be wishing for immutable lists... -- http://mail.python.org/mailman/listinfo/python-list
warnings.warn vs logging.warning
When I want to issue a warning, I am uncertain about the distinction between warnings.warn() and logging.warning(). My naive thought is to presume "warning" means the same thing in both cases, and so if I call logging.warning(), it should take care of making sure something equivalent to my calling warnings.warn() will happen... more than just having a message go to wherever a message to warnings.warn() would go. If I want to utilize the capabilities of both mechanisms (like having an exception raised), should I call both functions? -- http://mail.python.org/mailman/listinfo/python-list
