unittest and threading
Is it safe to use unittest with threads? In particular, if a unit test fails in some thread other than the one that launched the test, will that information be captured properly? A search of the net shows a suggestion that all failures must be reported in the main thread, but I couldn't find anything definitive. If it matters, I'm using CPython 2.7. Thanks. If you're using email, I'd appreciate a cc. Ross Boylan -- http://mail.python.org/mailman/listinfo/python-list
Re: unittest and threading
On Tue, 2012-01-24 at 13:54 -0800, Ross Boylan wrote: > Is it safe to use unittest with threads? > > In particular, if a unit test fails in some thread other than the one > that launched the test, will that information be captured properly? > > A search of the net shows a suggestion that all failures must be > reported in the main thread, but I couldn't find anything definitive. > > If it matters, I'm using CPython 2.7. > > Thanks. If you're using email, I'd appreciate a cc. > Ross Boylan > Steven D'Aprano wrote > I think you need to explain what you mean here in a little more detail. > > If you mean, "I have a library that uses threads internally, and I want > to test it with unittest", then the answer is almost certainly yes it is > safe. > > If you mean, "I want to write unit tests which use threads as part of the > test", then the answer again remains almost certainly yes it is safe. Thanks for your responses (only partially excerpted above). The code I want to test uses threads, but that is not entirely internal from the standpoint of the unit test framework. The unit test will be executing in one thread, but some of the assertions may occur in other threads. The question is whether that will work, in particular whether assertion failures will be properly captured and logged by the test framework. Concretely, a test may exercise some code that triggers a callback; the callback might come in a different thread, and the code that is triggered might make various assertions. There are two issues: whether assertions and their failures that happen in other threads will be correctly received by the test framework, and whether the framework is robust against several assertions being raised "simultaneously" in different threads. The latter seems a bit much to hope for. I assume that, at a minimum, the my test code will need to use locks or other coordination mechanisms so the test doesn't end before all code under test executes. Finally, I'll mention two senses of threads in tests that my question does not concern, although they are also interesting. I am not concerned with testing the performance of my code, in the sense of asserting that an operation must complete before x seconds or after y seconds. Some potential implementations of such tests might use threads even if the code under test was single-threaded. The question also does not concern running lots of unit tests in parallel. Ross -- http://mail.python.org/mailman/listinfo/python-list
Condition.wait() behavior with timeout
The Python 2.7 documents for the threading module says, in part, wait([timeout])ΒΆ Wait until notified or until a timeout occurs. If the calling thread has not acquired the lock when this method is called, a RuntimeError is raised. This method releases the underlying lock, and then blocks until it is awakened by a notify() or notifyAll() call for the same condition variable in another thread, or until the optional timeout occurs. Once awakened or timed out, it re-acquires the lock and returns. First, the documentation does not say what the return value is. I was hoping it was True or False depending on whether a timeout occurred, as Event.wait(). Second, the "Once awakened or timed out, it re-acquires the lock and returns" sounds very strange. If there was a timeout then an attempt to acquire the lock will block until it is released. Since there was no notify there almost certainly will be no release() immediately after the tiemout. Which would make the timeout pretty useless (since the thread that called wait() blocks even after the timeout expires), and might cause a race on the condition object. I've googled around, but haven't found anything quite on topic. http://bugs.python.org/issue1175933 from 2005 requested adding a timeout to Condition.wait(), a proposal rejected in 2009. Clearly it's there now. http://www.gossamer-threads.com/lists/python/dev/761847 complains there is no return value from wait and so no way to determine if a timeout occurred. One response was >GR> How am I supposed to know if it was notified or if it timed out? Normally you wouldn't have to know. The logic of your program should be such that you wait until a certain condition is satisfied. After each wait you usually check that condition anyway, like: http://bugs.python.org/issue1175933#msg48141 also refers to the need to check things after returning from wait(). But both of these cases seem to refer to a scenario in which there are many workers waiting on the condition, not one with notify() and a single thread waiting (which is what I'm thinking about). The thread does say there is no return value; it seems to me it would be useful to document that if it's still true (or True :). Can anyone help me understand what's going on? Thanks. Ross Boylan -- http://mail.python.org/mailman/listinfo/python-list
#line in python
The ast module shows that elements of the syntax tree have line and column numbers. Would it be sensible to attempt to revise them to achieve effects like the #line directive in C? Context: Using noweb, a literate programming tool, which from a source file foo.nw produces foo.py. The lines in the two files may be in completely different sequenes. For debugging, it is useful to receive error reports that refer to the original line number in foo.nw. I am not sure how such rewriting would interact with debugger commands that set a breakpoint at a file and line number. I'm also not sure it would change the reported line numbers of errors. The lack of a file name could be problematic if multiple sources contributed to the same .py file, but that is an unlikely scenario. As an extension or alternate, could there be a decorator like @source_line(lineno, filename) for classes and methods that could do the conversion on the fly? I don't know if there's a way to go from the function (or class) object the decorator receives to the AST. Comments? Ross Boylan -- http://mail.python.org/mailman/listinfo/python-list
Re: #line in python (dirty tricks)
Duncan Booth wrote
> Ross Boylan wrote:
>
> > As an extension or alternate, could there be a decorator like
> > @source_line(lineno, filename)
> > for classes and methods that could do the conversion on the fly? I
> > don't know if there's a way to go from the function (or class) object
> > the decorator receives to the AST.
> >
> No [easy] way to go from bytecodes back to AST, but I see no reason why you
> can't create a new code object with your filename and line numbers and then
> create a new function using your modified code object.
Could you elaborate? I don't understand what you are suggesting.
>
> If you don't have a 1:1 correspondence of lines then you'll need to pick
> out all the existing line numbers from the code object co_lnotab and modify
> them: see dis.py findlinestarts() for how to do this.
>
> Classes would be harder: the decorator doesn't run until after the class
> body has executed, so you can't change the line numbers that way until it's
> too late. The only thing I can think would be to put all of the generated
> code inside a function and fix up that function with a decorator that scans
> the bytecode to find all contained classes and fix them up.
>
> Or you could generate a .pyc file and then fix up line numbers in the whole
> file: see
> http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html for
> some code that shows you what's in a .pyc
>
My latest concept is to produce code that rewrites itself. Suppose the
naive file would be
--- mycode.py (naive)
class SomeClass:
"class comment"
def some_function(self, bar):
pass
- end ---
Protect that code by putting an "if 0:" in front of it and indenting
each line one space. Than prepend a bit of code to do the rewriting,
and add indicators of the original line numbers.
--- mycode.py (after wrapping) -
from detangle import detangle
detangle("mycode.py", "mycode.nw")
if 0:
# original code goes here
class SomeClass:
"class comment"
#and when line numbering changes
#line 35
def some_function(self, bar):
pass
- end ---
I would write detangle so that it scans through the file in which it
appears (named in the first argument), rewriting so that it appears to
come from the original file (mycode.nw) given in the second argument.
The scanning would look for the "if 0:" in the file. At that point it
would accumulate code by reading lines and stripping the leading space.
If it found a #line directive it would remember it and then remove it
from the string it was accumulating. Finally, detangle would would
pass the string of code to ast.compile, catching any syntax errors and
rewriting the file and line number (I might rewrite columns too with an
extension) and then rethrowing them.
If compilation succeeded detangle could rewrite the AST and then exec
it.
Ross
--
http://mail.python.org/mailman/listinfo/python-list
inserting \ in regular expressions
I want to replace every \ and " (the two characters for backslash and double quotes) with a \ and the same character, i.e., \ -> \\ " -> \" I have not been able to figure out how to do that. The documentation for re.sub says "repl can be a string or a function; if it is a string, any backslash escapes in it are processed.That is, \n is converted to a single newline character, \r is converted to a carriage return, and so forth. Unknown escapes such as \j are left alone." \\ is apparently unknown, and so is left as is. So I'm unable to get a single \. Here are some tries in Python 2.5.2. The document suggested the result of a function might not be subject to the same problem, but it seems to be. >>> def f(m): ...return "\\"+m.group(1) ... >>> re.sub(r"([\\\"])", f, 'Silly " quote') 'Silly \\" quote' >>> re.sub(r"([\\\"])", r"\\1", 'Silly " quote') 'Silly \\1 quote' >>> re.sub(r"([\\\"])", "1", 'Silly " quote') 'Silly \\1 quote' >>> re.sub(r"([\\\"])", "\1", 'Silly " quote') 'Silly \\\x01 quote' >>> re.sub(r"([\\\"])", "\\1", 'Silly " quote') 'Silly \\" quote' Or perhaps I'm confused about what the displayed results mean. If a string has a literal \, does it get shown as \\? I'd appreciate it if you cc me on the reply. Thanks. Ross Boylan -- http://mail.python.org/mailman/listinfo/python-list
Re: inserting \ in regular expressions [solved]
On Wed, 2011-10-26 at 12:48 -0700, Ross Boylan wrote: > I want to replace every \ and " (the two characters for backslash and > double quotes) with a \ and the same character, i.e., > \ -> \\ > " -> \" I'd like to thank Ian, Dave, MRAB, and John for their helpful responses. I hadn't realized the interpreter was giving me the repr, and that differed from the str. I've since found one other solution: email.utils.quote() does exactly the substitution I was looking for--not surprising, since I was doing it for email. Here's my little test program: #! /usr/bin/python import email, email.utils, re s0 = r'I am " a silly \quote' print s0 print re.sub(r'(\\|")', r'\\\1', s0) print email.utils.quote(s0) Output I am " a silly \quote I am \" a silly \\quote I am \" a silly \\quote Ross -- http://mail.python.org/mailman/listinfo/python-list
logging module and threading
I would like my different threads to log without stepping on each
other.
Past advice on this list (that I've found) mostly says to send the
messages to a Queue. That would work, but bypasses the logging
module's facilities.
The logging module itself is "thread-safe", but I think that just
means that individual output is protected. If I have, in temporarly
sequence:
thread 1: warning("A")
thread 2: info("something")
thread 1: warning("B")
then I think I'll get them output in this order. It's thread-safe in
that the log will not end up with an entry like
A
some
B
thing
(I think). But I want to get, for example,
A
B
something
What I would like is for each thread to emit a chunk of log messages
when it finishes a unit of work.
It looks as if I might be able to use a MemoryHandler to accumulate
the log locally and then flush it into the main log (I'd like to send
it to the main logger, but it looks as if I must send it to a specific
handler). Would something like the following work?
class MyThread (threading.Thread):
def __init__(self):
# do I need the next line?
threading.Thread.__init__(self)
self._log = logging.getLogger(self.getName())
# flush into main log
self._log = logging.MemoryHandler(999,
, # default flushlevel
logging.getLogger().handlers[1] )
def run(self):
j = getjob()
while j:
# do stuff
# log like this
self._log.info("some message")
# when done
self._log.flush()
j = getjob()
I'm also puzzled by how the logger hierarchy works. The docs say that
everything that is logged by the kids is also logged by the parent.
That would seem to defeat what I'm trying to do above, since the
parent would get each logged event right away. However,
logging.getLogger("a").error("test")
produces only a single log message indicating an associated object of "a".
The docs lead me to expect that I'd see one message from "a" and
another from root.
When I add handlers (e.g., FileHandlers) I do get the message recorded
by each.
Can anyone explain what's going on?
Thanks.
Ross Boylan
--
http://mail.python.org/mailman/listinfo/python-list
question about subprocess and shells
If one uses subprocess.Popen(args, ..., shell=True, ...) When args finishes execution, does the shell terminate? Either way seems problematic. If it does not terminate, then it seems as if calls like wait and communicate would never return. It also seems the subprocess would never die, and that most of the examples with the shell True would leave processes lying around. If it does terminate, then how can you stuff new commands down the pipe to the subprocesses stdin? Does this module contemplate receiving multiple commands for the shell to execute? I'm also unsure of the semantics of the pipes for the processes standard file handles. Do they need to be closed (judging from the examples, no)? When reads/writes to them return, and what state is the stream in at the time? Thanks for any wisdom you can offer. Ross Boylan -- http://mail.python.org/mailman/listinfo/python-list
