Re: Hardlink sub-directories and files

2011-08-03 Thread Nobody
On Tue, 02 Aug 2011 02:32:54 -0700, loial wrote:

> However I do not think it is possible to hard link directories ?

Modern Unices disallow hard links to directories, as it makes the
directory "tree" not a tree, so anything which performs a recursive walk
must explicitly check for cycles to avoid infnite recursion.

Older systems which allowed hard links to directories required root
privilege to do so.

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: python import error, what's wrong?

2011-08-03 Thread Ulrich Eckhardt
smith jack wrote:
> I am using pydev plugin in eclipse, all things works just as well
> but now i have confronted with a confusing problem, that is i can
> import a module write by myself successfully, but when i try to run
> this program,
> error just shows up, what's wrong?
> 
> the directory structure is as follows:
> 
> src
>   org.test
>   A.py
>   org.lab
>   B.py

Python uses a dot as separator between different parts of nested structures, 
like e.g. the directories during import. I guess your directory naming 
conflicts with that. Try this structure instead:

  src
org
  test
A.py
  lab
B.py

or maybe

  src
org_test
  A.py
org_lab
  B.py


> contents of A seems like:
> class A
> ...
> 
> contents of B seems like:  // I try to run B.py,
> python import error just appears, why?

"just appears" is not a sufficient discription of an error. I'd suggest 
reading Eric S. Raymond's excellent essay on "Asking Smart Questions", it 
will avoid such mistakes.

Uli

-- 
Domino Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: python import error, what's wrong?

2011-08-03 Thread Ben Finney
Ulrich Eckhardt  writes:

> > the directory structure is as follows:
> > 
> > src
> >   org.test
> >   A.py
> >   org.lab
> >   B.py
>
> Python uses a dot as separator between different parts of nested
> structures, like e.g. the directories during import.

More generally, the name of a module must be a valid Python identifier.
Hence, the module-name part of a directory or of a filename must also be
a valid Python identifier.

The ‘.’ (U+002E FULL STOP) character is not valid in an identifier.

-- 
 \“Pinky, are you pondering what I'm pondering?” “Wuh, I think |
  `\   so, Brain, but wouldn't anything lose its flavor on the bedpost |
_o__)   overnight?” —_Pinky and The Brain_ |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Syntactic sugar for assignment statements: one value to multiple targets?

2011-08-03 Thread Steven D'Aprano
gc wrote:

> Target lists using comma separation are great, but they don't work
> very well for this task. What I want is something like
> 
> a,b,c,d,e = *dict()


a, b, c, d, e = [dict() for i in range(5)]

Unfortunately there is no way of doing so without counting the assignment
targets. While slightly ugly, it doesn't seem ugly enough to justify the
extra complexity of special syntax for such a special case.


-- 
Steven

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Syntactic sugar for assignment statements: one value to multiple targets?

2011-08-03 Thread Gregory Ewing

gc wrote:


Alternatively, is there a version of iterable multiplication that
creates new objects rather than just copying the reference?


You can use a list comprehension:

  a, b, c, d, e = [dict() for i in xrange(5)]

or a generator expression:

  a, b, c, d, e = (dict() for i in xrange(5))

--
Greg
--
http://mail.python.org/mailman/listinfo/python-list


Re: Syntactic sugar for assignment statements: one value to multiple targets?

2011-08-03 Thread Katriel Cohn-Gordon
On Wed, Aug 3, 2011 at 9:25 AM, Steven D'Aprano <
steve+comp.lang.pyt...@pearwood.info> wrote:

> gc wrote:
>
> > Target lists using comma separation are great, but they don't work
> > very well for this task. What I want is something like
> >
> > a,b,c,d,e = *dict()
>
> a, b, c, d, e = [dict() for i in range(5)]
>

I think this is good code -- if you want five different dicts, then you
should call dict five times. Otherwise Python will magically call your
expression more than once, which isn't very nice. And what if your datatype
constructor has side-effects?

Katriel
-- 
http://mail.python.org/mailman/listinfo/python-list


Dynamically linking python into my vc project - help required

2011-08-03 Thread mrinalini

Hi,

I am trying to embed python into my MFC application. I have done this 
before by statically linking to the python lib. But I want to change 
this now.
The idea is to take the information from the registry for the installed 
version of python on the target machine. Then load python using 
loadlibrary call and use the functions from python using pointers to 
functions returned by GetProcAddress .


For example -

   hModPython = AfxLoadLibrary("Python23.dll");

pFnPyRun_SimpleString *pFunction = NULL;
pFnPy_Initialize *pPy_Initialize = NULL;

	pFunction = (pFnPyRun_SimpleString *)::GetProcAddress(hModPython, 
"PyRun_SimpleString");
	pPy_Initialize = (pFnPy_Initialize *)::GetProcAddress(hModPython, 
"Py_Initialize");


try
{
pPy_Initialize();

if ( pFunction )
{
(*pFunction)("import sys");   // call the code
}
else
{
			AfxMessageBox("unable to access function from python23.dll.", 
MB_ICONSTOP|MB_OK);

}
}
catch(...)
{

}

And then I want to execute a python script through my MFC application  
-


HANDLE hFile = CreateFile(file, GENERIC_READ, FILE_SHARE_READ, 
NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);


DWORD dwSize = GetFileSize(hFile, NULL);
DWORD dwRead;

char *s = new char[dwSize +1];

ReadFile(hFile, s, dwSize, &dwRead, NULL);

s[dwSize] = '\0';

CString wholefile(s);

wholefile.Remove('\r');
wholefile+="\n";
CloseHandle(hFile);

	pFnPy_CompileString *pFPy_CompileString = (pFnPy_CompileString 
*)::GetProcAddress(hModPython, "Py_CompileString");


CString fl(file);

	PyObject* pCodeObject = pFPy_CompileString(wholefile.GetBuffer(0), 
fl.GetBuffer(0), Py_file_input);


if (pCodeObject != NULL)
{
		pFnPyEval_EvalCode *pFPyEval_EvalCode = (pFnPyEval_EvalCode 
*)::GetProcAddress(hModPython, "PyEval_EvalCode");


		PyObject* pObject = pFPyEval_EvalCode((PyCodeObject*)pCodeObject, 
m_Dictionary, m_Dictionary);

}


I am facing two problems here , though I want to link to python 
dynamically I am required to include 	python.h for my code to compile 
the following declaration.


PyObject* pCodeObject



I tried copying some of the python definitions including  PyObject into 
a header in my mfc app. Then it complies fine. but Py_CompileString call 
fails. so finally I am unable to run script from my MFC application by 
linking to python dynamically.


How can this be done ? Please help. Is there a different approach to 
linking to python dynamically. Please could you write to me ?


Thanks in advance,
Mrinalini


--
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Chris Angelico
On Wed, Aug 3, 2011 at 5:46 AM, Dennis Lee Bieber  wrote:
>        Horrors... That looks like some MUF code I've seen (I never had a
> MUF flag on my old characters, so had no privilege to write in MUF --
> MPI was available to all, and even it had some links to MUF operations
> using similar magic numbers).
>

Hehe. I didn't mean that it should be thus in the source code, but
that some simple notation would tell the interpreter that it can make
that translation in memory - at the tokenization stage, or perhaps
when the function is invoked. The idea is that the interpreter can
skip one step - the step of taking the name "len", looking it up, and
finding the object  that it represents. It can
do that part of the work once, and patch in the function's address
somewhere as a special token - right into the tokenized source code.

Parenthesis: Reading the Wikipedia article on MUF shows that it has a
number of similarities to what I do today in coding my MUD - I have an
extremely rudimentary editor interface (smart editing is done on the
client), submit my code to the server, compile it, and invoke it - and
it's governed by permissions bits. However, I use a rather better
language than MUF :)

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Stefan Behnel

Chris Angelico, 03.08.2011 00:08:

So... Would this potentially produce wrong results? Would it be of any
use, or would its benefit be only valued in times when the whole
function needs to be redone in C?


Note that, in most cases, you do not need to "redo the whole function in 
C". You can just use Cython to compile it, thus leaving it in Python syntax 
and keeping up Python semantics, unless you decide to explicitly divert 
from them in favour of optimisations.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list


Text to image with same results on any platform

2011-08-03 Thread Gelonida N
Hi,


>From within a django application
I'd like create a small image file (e.g. .png)
which just contains some text.

I wondered what library would be appropriate and would yield the same
result independent of the OS (assuming the versions of the python
libraries are the same)
Images should be pixel identical independent on the platform on which
the image is created.

I made some attempts with PIL (Image / ImageFont / ImageDraw),
but have difficulties getting the same font under Linux and windows.





# code starts here
import Image
import ImageFont
import ImageDraw

# Here I don't know how to get the same font file
# for all platforms
# Perhaps there is a tiny trick that I overlooked.
font_file = get_font_file_name()
# Is truetype a good choice for portable fonts?
font = ImageFont.truetype(font_file, 20)

img = Image.new("RGB", (800, 600), "#FF")
draw = ImageDraw.Draw(image)

draw.text( (100, 100), font=font, fill="#00FF00")
#  end of code


I could also install any other library (matplotlib??)
if it were capable of creating consistent images
independent on the platform.

Thanks in advance for any suggestions.







-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Thomas Jollans
On 03/08/11 03:59, Dan Stromberg wrote:
> 
> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans  > wrote:
> 
> On 02/08/11 11:32, loial wrote:
> > I am trying to hardlink all files in a directory structure using
> > os.link.
> >
> > However I do not think it is possible to hard link directories ?
> 
> 
> That is pretty true.  I've heard of hardlinked directories on Solaris,
> but that's kind of an exception to the general rule.
>  
> 
> > So presumably I would need to do a mkdir for each sub-directory
> > encountered?
> > Or is there an easier way to hardlink everything in a directory
> > structure?.
> >
> > The requirement is for hard links, not symbolic links
> >
> 
> Yes, you have to mkdir everything. However, there is an easier way:
> 
> subprocess.Popen(['cp','-Rl','target','link'])
> 
> This is assuming that you're only supporting Unices with a working cp
> program, but as you're using hard links, that's quite a safe bet, I
> should think.
> 
> 
> A little more portable way:
> 
> $ cd from; find . -print | cpio -pdlv ../to
> cpio: ./b linked to ../to/./b
> ../to/./b
> cpio: ./a linked to ../to/./a
> ../to/./a
> cpio: ./c linked to ../to/./c
> ../to/./c
> ../to/./d
> cpio: ./d/1 linked to ../to/./d/1
> ../to/./d/1
> cpio: ./d/2 linked to ../to/./d/2
> ../to/./d/2
> cpio: ./d/3 linked to ../to/./d/3
> ../to/./d/3
> 0 blocks
> 
> However, you could do it without a shell command (IOW in pure python)
> using os.path.walk().

Is it more portable? I don't actually have cpio installed on this
system. Which implementations of cp don't implement -R and -l? Of
course, the best way is probably implementing this in Python, either
with os.path.walk, or with a monkey-patched shutil.copytree, as Peter
suggested.

Thomas

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: PyWart: os.path needs immediate attention!

2011-08-03 Thread Gelonida N
On 07/29/2011 11:43 PM, Chris Angelico wrote:
> On Sat, Jul 30, 2011 at 6:44 AM, Corey Richardson  wrote:
>> Excerpts from rantingrick's message of Fri Jul 29 13:22:04 -0400 2011:
>>>  * New path module will ONLY support one path sep!
>>
>> People who use windows are used to \ being their pathsep. If you show
>> them a path that looks like C:/whatever/the/path in an app, they are
>> going to think you are nuts. It isn't up to us to decide what anyone
>> uses as a path separator. They use \ natively, so should we. If at
>> any point Windows as an OS starts using /'s, and not support, actually
>> uses (in Windows Explorer as default (or whatever the filebrowser's
>> name is)), it would be great to switch over.
> 
> Just tested this: You can type c:/foldername into the box at the top
> of a folder in Explorer, and it works. It will translate it to
> c:\foldername though.
> 
> We are not here to talk solely to OSes. We are here primarily to talk
> to users. If you want to display a path to the user, it's best to do
> so in the way they're accustomed to - so on Windows, display
> backslashes.
> 
>>>  - normcase --> path = path.lower()
>>
>> Not quite, here are a few implementations of normcase (pulled from 2.7)
>>
>> # NT
>>return s.replace("/", "\\").lower()
>>
>> # Mac (Correct in this case)
>>return path.lower()
>>
>> # POSIX
>>return s
>>
>> But I can't think of where I would ever use that. Isn't case important on
>> Windows?
> 


well normcase might help to find out, whether two filenames are the same.


under unix 'myfile' and 'myFile' would be different
under windows they would refer to the same file.

normcase (perhaps to be combined with normpath) can tel you this.

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Thomas Jollans
On 03/08/11 00:08, Chris Angelico wrote:
> With the local-variable-snapshot technique ("len = len"), can anything
> be optimized, since the parser can guarantee that nothing ever
> reassigns to it? If not, perhaps this would be a place where something
> might be implemented:
> [...]

Yes, it can. The parser can know when it is reassigned to just by
looking at the one function, and even if it didn't, or if it were
reassigned, it wouldn't make much of a difference because the
interpreter would still only have to fetch one of a set of locals known
at compile time, which can be (is? I'm not sure) as simple as an array
lookup: two pointers being dereferences instead of one.

Chris Rebert raises a very important point though: Python dynamic
namespaces make optimising variable lookup comparatively difficult, but
not impossible: good JavaScript JITs are smart enough, so a good Python
JIT could be as well.

Thomas

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Steven D'Aprano
Chris Angelico wrote:

> On Tue, Aug 2, 2011 at 9:23 PM, Terry Reedy  wrote:
>> On 8/2/2011 12:55 PM, Chris Angelico wrote:
>>>
>>> As I understand it, Python exclusively late-binds names; when you
>>> define a function, nothing is ever pre-bound.
>>
>> By 'pre-bound' you presumably mean bound at definition time rather than
>> call time. Default arg objects *are* pre-computed and pre-bound to
>> internal slots at definition time.
> 
> Of course; that's a different issue altogether. No, I'm talking about
> the way a tight loop will involve repeated lookups for the same name.

It's not really a different issue. The "standard" approach for performing
early binding in Python is by binding a global name to a local name at
function definition time. In CPython at least, local lookups are faster
than globals: locals are stored in a fixed array, and the function knows
the numeric offset of each variable. Non-locals have to search multiple
namespaces (globals + built-ins), using a hash table. That's fast, but
locals are faster.

(Aside: this is why locals() is not generally writable: the dict is a copy
of that internal array. You can write to the dict, but the changes don't
propagate back to the actual local table of the function.)

So a good way to speed up name lookups is to turn a global into a local, and
one common way to do that is to do it when the function is defined. For
example, you will see this in the random module:

def randrange(self, start, stop=None, step=1, int=int, default=None,
  maxwidth=1< Unfortunately, there is no way - by definition - to guarantee that a
> binding won't change. Even in the example of getting the lengths of
> lines in a file, it's entirely possible for __len__ to rebind the
> global name "len" - so you can't rely on the repeated callings of
> len() to be calling the same function.
> 
> But who WOULD do that? It's somewhat ridiculous to consider, and
> there's a huge amount of code out there that does these repeated calls
> and does not do stupid rebindings in the middle. So Python permits
> crazy behaviour at the cost of the performance of normal behaviour.

This is a common criticism of Python, and the truth be told, it is right:
Python makes crazy monkey-patching that we're told not to do easy, at the
expense of allowing the compiler to optimise really common things.

But, are you *sure* that name lookups are *really* the bottleneck? Name
lookups are pretty fast. If you want them faster, turn them into a local
variable. It's not clear to me that syntax, or a pragma directive, or some
other form of magic, is better than an explicit assignment:

def func(x):
_len = len  # bind at function call time
for i in x:
_len(i)  # lookups of _len will be faster than len


The argument that Python would be faster if it did early binding is not a
given. For trivial code that doesn't do much, I dare say it might shave off
a significant percentage, but trivial code is already pretty fast. Who
cares if you speed up a trivial operation? In non-trivial code that does
real work, it's not obvious that early binding will result in meaningful
time savings. ("Yay, my code now runs in 17.1 milliseconds instead of
17.2!") The onus is on the person proposing this change to prove that the
benefit is worth the extra complexity.

Knowing what we know now, I'm not sure whether Guido would have kept the
late binding semantics for Python, or privileged builtins in some fashion.
I suspect that he wouldn't have changed a thing, but if he did, I suspect
he'd be more concerned about accidental shadowing of builtins than about
optimizations.

E.g. people say list = [1,2,3], and then later try to call list(something).

Or maybe that's just me :)

Guido is generally very, very suspicious about proposed optimisations. They
tend to increase the code's complexity by far more than they increase its
speed. (E.g. an order of magnitude more complex to get 10% speed increase.)
And they often subtly break the semantics of the code. E.g. if you do
floating point maths, optimizing compilers that assume that x + y - x
equals y are BROKEN. 


> With the local-variable-snapshot technique ("len = len"), can anything
> be optimized, since the parser can guarantee that nothing ever
> reassigns to it? 

Very likely. But the optimization happens at runtime, not at compile time.

Default values for parameters are stored as an attribute of the function at
definition time. So if you have this:

def f(x, len=len):
return len(x)

then f gets a publicly accessible attribute func_defaults:

>>> f.func_defaults
(,)

Even if you re-bind len, that doesn't change the binding here, and the
*local* variable len will be assigned that value when f is called. Here's a
practical example:


>>> def test(len=len):
... while True:
... yield len
...
>>>
>>> it = test()
>>> next(it)

>>> len = None  # rebind the global len
>>> next(it)

>>> test.func_defaults = (None, )
>>> next(it)


But note that n

Re: Early binding as an option

2011-08-03 Thread Chris Angelico
On Wed, Aug 3, 2011 at 11:16 AM, Steven D'Aprano
 wrote:
> Chris Angelico wrote:
>> Of course; that's a different issue altogether. No, I'm talking about
>> the way a tight loop will involve repeated lookups for the same name.
>
> It's not really a different issue. The "standard" approach for performing
> early binding in Python is by binding a global name to a local name at
> function definition time.

Thanks for the detailed response!

My concept was that this wouldn't affect the dynamism significantly,
and that if the function's caller wanted to change the definition of
'len', s/he could still do so; the "snapshotting" would occur as the
function begins executing. This would make things rather safer, as
there's a limited window during which this could affect things.

> In CPython at least, local lookups are faster
> than globals: locals are stored in a fixed array, and the function knows
> the numeric offset of each variable.

Ah! I was not aware of this, and thought that locals were a dictionary
too. Of course, it makes a lot of sense. In that case, the classic
"grab it as a local" isn't just loading down the locals dictionary
with more names and thus slower lookups.

>        Do not supply the 'int', 'default', and 'maxwidth' arguments.
>        """

I love the way Python doesn't stop you from doing stupid things. This
is an invitation to do something... hmm.
>>> random.randrange(5,10,1,int=lambda x: (print(x),int(x))[1])

> But, are you *sure* that name lookups are *really* the bottleneck? Name
> lookups are pretty fast. If you want them faster, turn them into a local
> variable. It's not clear to me that syntax, or a pragma directive, or some
> other form of magic, is better than an explicit assignment:

No, I'm not sure. Unfortunately I have no convenient way to compare;
all I can do is compare with a completely different language, which is
of course NEVER going to be fair. The only actual data I have on the
subject is the perfect-numbers search the other day; Pike managed the
same task in a fraction of the time that Python spent on it. Pike has
a single integer type that quietly switches from small to large at
2**32, with a noticeable performance change. This is the same as
Python 2, but could explain some of the Python 3 penalty. There's only
two obvious possibilities (there may be others, of course): firstly,
that the actual name lookup is expensive; and secondly, that Pike is
able to optimize integer arithmetic by knowing that the value in
question is an int, and it will never be anything else. Since I was
under the impression that local variables were stored in a hash table,
I concluded that the first option was reasonably likely, and tried to
get more data.

>> So... Would this potentially produce wrong results? Would it be of any
>> use, or would its benefit be only valued in times when the whole
>> function needs to be redone in C?
>
> You really would need to demonstrate that the bottleneck in useful code (as
> opposed to toy examples) was the name lookups.

And especially, prove that the same can't be done with local variables.

So which is the better idiom?

def func(x):
   _len = len  # bind at function call time
   for i in x:
   _len(i)  # lookups of _len will be faster than len

or:

def func(x):
   len = len  # localize len
   for i in x:
   len(i)  # use it exactly as you otherwise would

In other words, is it "Explicit is better than implicit" or "Simple is
better than complex"? Since this does absolutely exactly the same
thing as the original, I'd be inclined to go with the second form; a
single line at the top of the function, with an explanatory comment
perhaps, and the rest of the code is unaware of the optimization. But
this has the potential to be insanely confusing if anything goes
wrong, because down in the body of the function there's no clue that
anything was changed.

Unfortunately I can't offer any real-world use cases or examples. This
is purely a thought experiment at the moment.

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Syntactic sugar for assignment statements: one value to multiple targets?

2011-08-03 Thread Tim Chase

On 08/03/2011 03:25 AM, Steven D'Aprano wrote:

gc wrote:


Target lists using comma separation are great, but they don't work
very well for this task. What I want is something like

a,b,c,d,e = *dict()



a, b, c, d, e = [dict() for i in range(5)]

Unfortunately there is no way of doing so without counting the assignment
targets. While slightly ugly, it doesn't seem ugly enough to justify the
extra complexity of special syntax for such a special case.


I understand that in Py3k (and perhaps back-ported into later 2.x 
series?) one can do something like


a, b, c, d, e, *junk = (dict() for _ in range())

to prevent the need to count.  However, I was disappointed with 
all the generator-ification of things in Py3k, that this "and the 
rest" syntax slurps up the entire generator, rather than just 
assigning the iterator.  That would much more tidily be written as


a,b,c,d,e, *more_dict_generator = (dict() for _ in itertools.count())

(itertools.count() happening to be a infinite generator).  I can 
see the need to slurp if you have things afterward:


  a,b,c, *junk ,d,e = iterator(...)

but when the "and the rest" is the last one, it would make sense 
not to force a slurp.


-tkc





--
http://mail.python.org/mailman/listinfo/python-list


Re: Syntactic sugar for assignment statements: one value to multiple targets?

2011-08-03 Thread Tim Chase

On 08/03/2011 03:36 AM, Katriel Cohn-Gordon wrote:

On Wed, Aug 3, 2011 at 9:25 AM, Steven D'Aprano wrote:

a, b, c, d, e = [dict() for i in range(5)]


I think this is good code -- if you want five different dicts,
then you should call dict five times. Otherwise Python will
magically call your expression more than once, which isn't
very nice. And what if your datatype constructor has
side-effects?


If the side-effects are correct behavior (perhaps opening files, 
network connections, or even updating a class variable) then 
constructor side-effects are just doing what they're supposed to. 
 E.g. something I use somewhat regularly in my code[*]:


 a,b,c,d = (file('file%i.txt', 'w') for i in range(4))

If the side-effects aren't performing the correct behavior, fix 
the constructor. :)


-tkc


[*] okay, it's more like

(features,
 adjustments,
 internet,
 ) = (file(fname) for fname in (
   'features.txt',
   'adjustments.txt',
   'internet.txt'
   )

or even

(features,
 adjustments,
 internet,
 ) = (
   set(
 line.strip().upper()
 for line
 in file(fname)
 if line.strip()
 )
   for fname in (
   'features.txt',
   'adjustments.txt',
   'internet.txt'
   )

to load various set() data from text-files.
--
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Stefan Behnel

Thomas Jollans, 03.08.2011 12:00:

JavaScript JITs are smart enough, so a good Python
JIT could be as well.


Why "could"? There's PyPy if you need it.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list


timeout issue with modules dbus & gobject

2011-08-03 Thread Makiavelik
Hi,
Here is a sample code that reproduces the issue :
[code]
import logging
import unittest
import signal
import gobject
import dbus
from functools import wraps
from dbus.mainloop.glib import DBusGMainLoop

class TimeoutException(Exception):
pass

def timeout(timeout_time=1800):
"""
decorator function catching the argument
"""
def timeout_function(func):
"""
decorator function
"""
@wraps(func)
def _timeout_function(self):
"""
create a signal handler
set the timeout with the argument given while calling the
decorator @timeout
call the function
catch a timeout exception if necessary
"""
def timeout_handler(signum, frame):
print 'Timeout (%s sec) reached' % str(timeout_time)
raise TimeoutException()

old_handler = signal.signal(signal.SIGALRM,
timeout_handler)
signal.alarm(timeout_time) # triger alarm in timeout_time
seconds
try:
retval = func(self)
finally:
signal.signal(signal.SIGALRM, old_handler)
signal.alarm(0)
return retval
return _timeout_function
return timeout_function

class Test_loopRun_And_Timeout(unittest.TestCase):
def __init__(self,*args,**kwargs):
super(Test_loopRun_And_Timeout, self).__init__(*args,**kwargs)
dbus_loop = DBusGMainLoop(set_as_default=True)
self.bus = dbus.SessionBus(private=True,mainloop=dbus_loop)
self.loop = gobject.MainLoop()

logging.basicConfig()
self.__logger = logging.getLogger("Tests.%s" %
self.__class__.__name__)
self.__logger.setLevel(logging.DEBUG)

def setUp(self):
'''
in this part, mediarouter can not be created
Setup are all launch in //
So if 50 tests are run in this class, 50 mediarouters are
created
'''
pass


def tearDown(self):
'''
'''

@timeout(5)
def test_001(self):
'''
'''
self.__logger.info('[CHECKPOINT] test_001')
try:
self.__logger.info('entering a waiting loop')
self.loop.run()
self.__logger.info('dummy log, should not appear')
self.fail()

except KeyboardInterrupt:
self.__logger.exception('Catching a Ctrl+c event (user or
timeout)')
except :
self.__logger.exception('Unexpected error')
self.fail()


@timeout(5)
def test_002(self):
'''
'''
def loop_quit(loop):
loop.quit()
return False


self.__logger.info('[CHECKPOINT] test_002')
try:
self.__logger.info('entering a waiting loop')
gobject.timeout_add(1000, loop_quit, self.loop)
self.loop.run()
self.__logger.info('exiting the loop')

except KeyboardInterrupt:
self.__logger.exception('Catching a Ctrl+c event (user or
timeout)')
self.fail()
except :
self.__logger.exception('Unexpected error')
self.fail()
[/code]

If I start a unittest campaign like this :
[code]
if __name__ == "__main__":
#Add the test you want to run
suite = unittest.TestSuite()

#To choose a list of tests, comment those you don't want to run
suite.addTest(Test_loopRun_And_Timeout('test_002'))
suite.addTest(Test_loopRun_And_Timeout('test_001'))
unittest.TextTestRunner(verbosity=0).run(suite)
print 'done'
[/code]
the result give me 2 tests OK

Now If I launch this
[code]
if __name__ == "__main__":
#Add the test you want to run
suite = unittest.TestSuite()

#To choose a list of tests, comment those you don't want to run
suite.addTest(Test_loopRun_And_Timeout('test_001'))
suite.addTest(Test_loopRun_And_Timeout('test_002'))
unittest.TextTestRunner(verbosity=0).run(suite)
print 'done'
[/code]
1 OK (test_001)
1 Fail (test_002 goes on timeout)

And if I am using more than 3 Testcases, the code is going to run in
infinite loop (Ctrl+C or timeout decorator does not work, only a kill
works)


Is there an issue using the 'timeout' decorator with the loop.run() ?
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: with statement and context managers

2011-08-03 Thread Thomas Rachel

Am 03.08.2011 04:15 schrieb Steven D'Aprano:

> I'm not greatly experienced with context managers and the with
> statement, so I would like to check my logic.
>
> Somebody (doesn't matter who, or where) stated that they frequently
> use this idiom:
>
> spam = MyContextManager(*args)
> for ham in my_iter:
>  with spam:
>   # do stuff
>
>
> but to me that looks badly wrong. Surely the spam context manager
> object will exit after the first iteration, and always raise an
> exception on the second? But I don't quite understand context
> managers enough to be sure.

Depends on the implementation. As already stated, a 
contextlib.contextmanager will only run once. But it is easy to turn it 
into a persistent one - which internally initializes as often as needed.


class PersistentCM(object):
def __init__(self, target):
self.target = target
self.current = []
def __enter__(self):
cm = self.target()
cm.__enter__()
self.current.append(cm)
def __exit__(self, *e):
return self.current.pop(-1).__exit__(*e)

(untested)

This would allow for a CM to be used multiple times. It internally 
generates a new one on every __enter__() call and disposes of it on 
every __exit__(). As __enter__() and __exit__() are supposed to be used 
symmetrical, it should(!) work this way.


Many CMs don't allow your shown use, though, partly due to obvious reasons:

* The ...ing named ones do the action they are named after in the 
__exit__() part and do nothing in the __enter__() part. (E.g. closing.)


* The ...ed named ones do the named action in __enter__() and undo it in 
__exit__(). They may or may not work multiple times.


For example, take threading.Lock et al., which you can have locked 
several times.




spam = open('aaa')
for ham in range(5):

... with spam:
... print ham
...
0
Traceback (most recent call last):
   File "", line 2, in
ValueError: I/O operation on closed file


To be honest, this one I don't understand as well. __exit__() closes the 
file spam, but even before "print ham" can be executed for a second 
time, there is a check if the file is still open. What does __enter__() 
do here? Just check if it is closed, or is it more? Nevertheless, it 
makes sense.




# Slightly more complex example.


from contextlib import closing
import urllib
spam = closing(urllib.urlopen('http://www.python.org'))
for ham in range(5):

... with spam as page:
... print ham, sum(len(line) for line in page)
...
0 18486
1
Traceback (most recent call last):
   File "", line 3, in
   File "", line 3, in
   File "/usr/local/lib/python2.7/socket.py", line 528, in next
 line = self.readline()
   File "/usr/local/lib/python2.7/socket.py", line 424, in readline
 recv = self._sock.recv
AttributeError: 'NoneType' object has no attribute 'recv'


Here the above said applies: after closing happens in __exit__() and 
_sock gets set to None, the object is not usable any longer.




Am I right to expect that the above idiom cannot work? If not, what sort of
context managers do work as shown?


As said, take threading.Lock as an example: they can be acquire()d and 
release()d as often as you want, and these actions happen in __enter__() 
and __exit__(), respectively.



HTH,

Thomas
--
http://mail.python.org/mailman/listinfo/python-list


parsing in python

2011-08-03 Thread Jayron Soares
Hi folks,

I've created a simple method to grab files texts from directory by words
random, however I figure out that I need extract the content inside of each
file, in fact I believe  I have to create a parsing, nonetheless I don't
know how to create a parser.
Please some could share some tips to do it?
Cheers
Jayron
here is the code:

# -*- conding: utf-8 -*-


from subprocess import Popen, PIPE

pesquisa = raw_input(" Digite a pesquisa de interesse: ")

def Pesquisar(pesquisa):
p = Popen(["search", pesquisa],stdout=PIPE)
resultado = p.communicate()[0]
return resultado


print Pesquisar(pesquisa)




-- 
*" A Vida é arte do Saber...Quem quiser saber tem que Estudar!"*

http://bucolick.tumblr.com
http://artecultural.wordpress.com/
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: attach cString to memoryview?

2011-08-03 Thread Peter Otten
Phlip wrote:

> q = cStringIO.StringIO()
> p = Popen(cmd, shell=True, stdout=q, bufsize=4096)

> The Subject line problem refers to how to get rid of this:
> 
> AttributeError: 'cStringIO.StringO' object
>has no attribute 'fileno'

What's wrong with subprocess.PIPE?

>>> import subprocess as sp   
>>> p = sp.Popen(["ls", "/usr/lib/python2.6"], stdout=sp.PIPE)
>>> print p.communicate()[0]

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Grant Edwards
On 2011-08-03, Kushal Kumaran  wrote:
> On Wed, Aug 3, 2011 at 7:29 AM, Dan Stromberg  wrote:
>>
>> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans  wrote:
>>>
>>> On 02/08/11 11:32, loial wrote:
>>> > I am trying to hardlink all files in a directory structure using
>>> > os.link.
>>> >
>>> > However I do not think it is possible to hard link directories ?
>>
>> That is pretty true.?? I've heard of hardlinked directories on Solaris, but
>> that's kind of an exception to the general rule.
>
> In APUE, Richard Stevens says only root could do this,

Yep, in early versions of Solaris root could hard-link directories.
I did it once, and it's not something one did a second time.  fsck
couldn't deal with it and pretty much fell over.  IIRC, the only way
to recover was to clear several inodes manually and then let fsck
salvage things.

> if it is supported by the system at all.  In a footnote, he
> additionally mentions he screwed up his filesystem by creating a loop
> of hardlinked directories while writing that section of the book.

That sounds about right.

> I suppose it is a good thing systems don't allow that now.

It wouldn't be a problem, except there are some important places in
Unix where it is assume that filesystems are trees.  Hard linking
directories causes that assumption to be false.

-- 
Grant Edwards   grant.b.edwardsYow! Somewhere in DOWNTOWN
  at   BURBANK a prostitute is
  gmail.comOVERCOOKING a LAMB CHOP!!
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Thomas Jollans
On 03/08/11 16:22, Grant Edwards wrote:
> On 2011-08-03, Kushal Kumaran  wrote:
>> I suppose it is a good thing systems don't allow that now.
> 
> It wouldn't be a problem, except there are some important places in
> Unix where it is assume that filesystems are trees.  Hard linking
> directories causes that assumption to be false.
> 

It is generally assumed that ".." is clearly defined, i.e. that every
directory has a parent. With hard linked directories, this would break down.
-- 
http://mail.python.org/mailman/listinfo/python-list


python distributed computing

2011-08-03 Thread julien godin
Hello the list,

I have a, let say 'special' , request.
I would like to make a very specific type of distributed computing.
Let me explain myself :

I have a HUGE(everything is relative) amount of data coming from UDP/514
( good old syslog), and one process alone could not handle them ( it's
full of regex, comparison and counting. all this at about 100K lines of data
per seconds and each line is about 1500o long. I tried to handle it on one
process, 100% core time in htop. )

So i said to myself : why not distribute the computing ?
I looked for a good library that could do the distribution/process
management job for me (http://wiki.python.org/moin/ParallelProcessing).

What i found was not really usable by me ( distributed processing with a lot
of processing but almost no process communication wich is the opposite of
the thing i would like to do: distribute the events in real time so it is
processed in real time.).

My question is : is their any library or API or anything that is able to do
that ?

Sincerly
Julien

PS: This is my first time posting on any mailing list, my explanation/syntax
could look a little bite, messy.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: python distributed computing

2011-08-03 Thread Chris Angelico
On Wed, Aug 3, 2011 at 4:37 PM, julien godin  wrote:
> I have a HUGE(everything is relative) amount of data coming from UDP/514
> ( good old syslog), and one process alone could not handle them ( it's
> full of regex, comparison and counting. all this at about 100K lines of data
> per seconds and each line is about 1500o long. I tried to handle it on one
> process, 100% core time in htop. )
> So i said to myself : why not distribute the computing ?

You could brute-force it by forking out to multiple computers, but
that would entail network traffic, which might defeat the purpose. If
you have a multi-core CPU or multi-CPU computer, a better option would
be to look into the multiprocessing module for some simple ways to
divide the work between cores/CPUs. Otherwise, there's a few things to
try.

First and most important thing: Optimize your algorithms! Can you do
less work and still get the same result? For instance, can you replace
the regex with two or three simpler string methods? Profile your code
to find out where most of the time is being spent, and see if you can
recode those parts.

Second: Try PyPy or Cython for higher-performance Python code.

And third, if you still can't get the performance you need, consider
changing languages. Python isn't the best language for fast execution
- it's good for fast coding. Rewrite some or all of your code in C,
Pike, COBOL, raw assembly language... okay, maybe not those last two!
Since you profiled your code up in step 1, you'll know which parts are
the best candidates for C code.

I think there are quite a few options better than forking across
computers; although distributed computing IS a lot of fun.

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list


parsing function parameters

2011-08-03 Thread Lee Harr

I am trying to get some information about a function
before (and without) calling it.

Here is what I have so far. I chose to go with a
regular expression, so now I have 2 problems :o)



def pdict(f, pstr):
    '''given a function object and a string with the function parameters,
    return a dictionary of {parameter name: default value or None, ...}


    I don't really need the actual default values, just knowing that
    they exist or having their string representations would be fine.


    def foo(x, y, z=50):
    pass

    ps = 'x, y, z=50'

    pdict(foo, ps)
    {'x': None, 'y': None, 'z': 50}
    OR: {'x': '', 'y': '', 'z': '50'}
    OR: {'x': False, 'y': False, 'z': True}


    Note that the parameter list can be arbitrarily complex
    (as long as it is legal python)

    def bar(m, n=(1, 2, 3), o  =  ('n=5', 'or anything legal')):
    pass

    pps = 'm, n=(1, 2, 3), o  =  ('n=5', 'or anything legal')'

    pdict(bar, pps)
    {'m': None, 'n': '(1, 2, 3)', 'o': "('n=5', 'or anything legal')"}

    '''
    prs = f.func_code.co_varnames
    rxl = [r'(\s*%s\s*=(?=(?:\s*\S+))\s*\S+)\s*'%pr for pr in prs]
    rx = ','.join(rxl)
    print rx


    import re
    re.match(rx, pstr).groups()


This regex works in limited situations, namely when every parameter
has a default value, but I can't figure out how to also make it
match a parameter that has no default.


I am open to any solution, though I am reluctant to pull in any more
external libraries just for this one little enhancement. I have it
working by just splitting the pstr on comma, except of course that
it fails when the default value contains a comma (a tuple or list,
for instance).


Any assistance or ideas appreciated.

  
-- 
http://mail.python.org/mailman/listinfo/python-list


[job] multiple openings for CS, CE, EE, IT majors / graduates

2011-08-03 Thread AJ ONeal
I know of some job openings. Contact me off list for more details.


The positions are full-time (or at least 30hrs / week) paid internships that
should lead to a full-time salaried position in near future.


If a *mix of any* of the bullet points below *describes you* or *someone you
know*, please ask me some questions, send me a resume, and I'll get you in
touch with some great people who need you!


*All great candidates*

   - have taken their education into their own hands to some degree
   - learn quickly
   - have beyond-the-classroom experience / personal projects
   - know how to leverage tools and people in their circle (forums, interest
   groups, friends) to find answers to questions
   - are familiar with *nix and know their way around with find, grep, vim
   (or emacs)




*Event-driven Application Developer*

   - could write a basic evented messaging queueing and retrieval service in
   about a week

   - can solve fizzbuzz in less than an hour
   - has written a simple chat server
   - knows the difference between a markup language and an object notation

   - is familiar with one of node.js, python / twisted, ruby / eventmachine,
   go, erlang
   - has used a web framework of some sort
   - has used C and written a queue of some sort
   - enjoys regex

   - has a good understanding of http and general networking concepts
   - has used wireshark to examine http and tcp traffic
   - has written a native mobile app (iOS, Android)




*Gadget Engineer*

   - could build a rechargeable battery-powered usb hub in about a week

   - could reverse engineer a cheap usb hub from e-bay in about a day
   - could build a pringles cantenna in about a day


   - has used C and written a queue of some sort
   - can write object-oriented C (as if it were a programming language)


   - has played with Arduino, Bug, Beagle, Panda, or some other sort of
   project board

   - Envies those who own a Fluke multimeter (or owns one)




*HTML5 Web App Developer*

   - could write a basic client-side timesheet application in about a week

   - has a good handle on JavaScript
   - knows how to use CSS selectors
   - knows the difference between a markup language and an object notation
   - knows the difference (or relationship) between HTML and the DOM


   - can solve fizzbuzz in less than an hour
   - has written a simple chat server
   - can draw a graph on canvas
   - enjoys regex

   - has worked with APIs such as Google Apps, facebook, twitter, etc
   - has a good understanding of http and general networking concepts
   - has used wireshark or firebug to examine http traffic
   - has written a native mobile app (iOS, Android)




*Linux Techie++*

   - could set up a VPS with nginx, gitosis, git pull hooks, and
   reverse-proxied apps in about a week

   - has built a kernel
   - has built an out-of-tree kernel module

   - can use bash almost as if were a programming language (but has the good
   sense not to)
   - can write object-oriented C (as if it were a programming language)
   - enjoys regex

   - has used C and written a queue of some sort
   - has used wireshark to examine traffic
   - is familiar with one of node.js, python / twisted, ruby / eventmachine,
   go, erlang




AJ ONeal
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: parsing in python

2011-08-03 Thread Dan Stromberg
To just split lines into words, you could probably just use a regex.

If you need to match things, like quotes or brackets or parens, pyparsing is
pretty nice.

On Wed, Aug 3, 2011 at 6:26 AM, Jayron Soares wrote:

> Hi folks,
>
> I've created a simple method to grab files texts from directory by words
> random, however I figure out that I need extract the content inside of each
> file, in fact I believe  I have to create a parsing, nonetheless I don't
> know how to create a parser.
> Please some could share some tips to do it?
> Cheers
> Jayron
> here is the code:
>
> # -*- conding: utf-8 -*-
>
>
> from subprocess import Popen, PIPE
>
> pesquisa = raw_input(" Digite a pesquisa de interesse: ")
>
> def Pesquisar(pesquisa):
> p = Popen(["search", pesquisa],stdout=PIPE)
> resultado = p.communicate()[0]
> return resultado
>
>
> print Pesquisar(pesquisa)
>
>
>
>
> --
> *" A Vida é arte do Saber...Quem quiser saber tem que Estudar!"*
>
> http://bucolick.tumblr.com
> http://artecultural.wordpress.com/
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list


Equality check

2011-08-03 Thread Abhishek Jain
How to check equality of two nos. in python
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: parsing function parameters

2011-08-03 Thread Stefan Behnel

Lee Harr, 03.08.2011 18:02:

I am trying to get some information about a function
before (and without) calling it.

Here is what I have so far. I chose to go with a
regular expression, so now I have 2 problems :o)


Can't you just use runtime introspection? Take a look at the inspect module.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list


Re: Snippet: The leanest Popen wrapper

2011-08-03 Thread Peter Otten
Phlip wrote:

> Groupies:

I smell a slight misperception of the audience you are addressing ;)

> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
> 
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
> 
> def command(*cmdz):
> 
> process = Popen( flatten(cmdz),
>  shell= True,
>  stdout= subprocess.PIPE,
>  stderr= subprocess.PIPE,
>  bufsize= 4096 )

Protect your environment, don't let stderr pollute the nearby river ;)

> def line():
> return process.stdout.readline().rstrip()
> 
> def s():
> while True:
> l = line()

At that point l may be empty because you have read the output completely or 
because there was an empty line that you rstripped to look like the end of 
file.

> if not l:  break
> yield l
> 
> line.s = s
> 
> return line

I think you are overdoing that closure/function factory thing a bit...

Seriously, you should reread the subprocess documentation to learn how to 
avoid deadlocks.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Dan Stromberg
On Wed, Aug 3, 2011 at 2:47 AM, Thomas Jollans  wrote:

> Is it more portable? I don't actually have cpio installed on this
> system.


Interesting.  Of course, it's probably readily available to you.  What *ix
are you seeing that doesn't include cpio by default?


> Which implementations of cp don't implement -R and -l?


Probably most of them, except GNU and newer BSD.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Dan Stromberg
On Wed, Aug 3, 2011 at 12:04 AM, Nobody  wrote:

> On Tue, 02 Aug 2011 02:32:54 -0700, loial wrote:
>
> > However I do not think it is possible to hard link directories ?
>
> Modern Unices disallow hard links to directories, as it makes the
> directory "tree" not a tree, so anything which performs a recursive walk
> must explicitly check for cycles to avoid infnite recursion.
>
> Older systems which allowed hard links to directories required root
> privilege to do so.
>
>
Actually, I recently heard about a system with multiple Solaris Containers
set up, that were sharing their /usr via directory hardlinks.  The question
came up: How does one back up such a system?  But ufsdump worked.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: how to sort a hash list without generating a new object?

2011-08-03 Thread Daniel Stutzbach
On Tue, Aug 2, 2011 at 5:53 PM, Chris Rebert  wrote:

> If you /really/ need a sorted mapping datatype, google for
> "sorteddict" (which is quite distinct from OrderedDict).
> Or look for a binary search tree or skip list implementation of some
> sort; but these aren't commonly used in Python, so it may be hard to
> find a good one.
>

The blist package (I'm the author) provides a list-like type that has O(log
n) insertions and deletions.  It provides a sorteddict type that uses the
blist type under-the-hood.

blist's "sorteddict" supports the "key" parameter (which works like
list.sort's key parameter), which the original poster could use to maintain
the keys in reverse order.

http://pypi.python.org/pypi/blist/

There's no overhead to learn how to use the new types.  A blist works
exactly like a list but with different performance characteristics, and a
sorteddict works just like a dict but keeps the keys sorted.

-- 
Daniel Stutzbach
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Snippet: The leanest Popen wrapper

2011-08-03 Thread Thomas Jollans
On 03/08/11 17:29, Phlip wrote:
> Groupies:
> 
> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
> 
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
> 
> def command(*cmdz):
> 
> process = Popen( flatten(cmdz),
>  shell= True,
>  stdout= subprocess.PIPE,
>  stderr= subprocess.PIPE,
>  bufsize= 4096 )
> 
> def line():
> return process.stdout.readline().rstrip()
> 
> def s():
> while True:
> l = line()
> if not l:  break

This will ignore everything after a blank line. Intended?
It may be better not to use readline(), but to use the fact that it's an
iterable, and use next(process.stdout) to get each line. (and deal with
StopIteration accordingly -- or not)

> yield l
> 
> line.s = s
> 
> return line
> 
> That leads to some syntactic sugar. For example, one truly demented
> way to stream in an entire block and then treat it as one big string
> is this:
> 
> print '\n'.join(command('ls').s())
> 
> The point of the command() complex is the ability to start a long
> command and then fetch out individual lines from it:
> 
> line = command('find', '../..')
> 
> print 'lines'
> print line()
> print line()
> print line()
> 
> print 'all'
> print list(line.s())
> 
> If you need different pipe abilities, such as stdin, you can trivially
> add them to the contents of command() (it's not productized on
> purpose).
> 
> So I can take the line() functor and, for example, pin it to a View
> object, or put it in another thread now, right?
> 
> --
>   Phlip
>   http://bit.ly/ZeekLand

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Equality check

2011-08-03 Thread Thomas Jollans
On 03/08/11 18:22, Abhishek Jain wrote:
> How to check equality of two nos. in python
> 
> 

http://docs.python.org/py3k/reference/expressions.html#not-in
http://docs.python.org/py3k/tutorial/index.html
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: parsing function parameters

2011-08-03 Thread Peter Otten
Lee Harr wrote:

> I am trying to get some information about a function
> before (and without) calling it.

If you allow for the function arguments to be evaluated it's easy (requires 
Python 2.7):

>>> import inspect
>>> def get_args(*args, **kw):
... return args, kw
...
>>> argstr = "1, 2, z=42"
>>> def f(a, b, x=1, y=2, z=3):
... pass
...
>>> args, kw = eval("get_args(%s)" % argstr)
>>> inspect.getcallargs(f, *args, **kw)
{'a': 1, 'x': 1, 'b': 2, 'z': 42, 'y': 2}


-- 
http://mail.python.org/mailman/listinfo/python-list


Re: parsing in python

2011-08-03 Thread Jayron Soares
Hi Dan,

Thank you a lot =)
Cheers
Jayron

2011/8/3 Dan Stromberg 

>
> To just split lines into words, you could probably just use a regex.
>
> If you need to match things, like quotes or brackets or parens, pyparsing
> is pretty nice.
>
> On Wed, Aug 3, 2011 at 6:26 AM, Jayron Soares wrote:
>
>> Hi folks,
>>
>> I've created a simple method to grab files texts from directory by words
>> random, however I figure out that I need extract the content inside of each
>> file, in fact I believe  I have to create a parsing, nonetheless I don't
>> know how to create a parser.
>> Please some could share some tips to do it?
>> Cheers
>> Jayron
>> here is the code:
>>
>> # -*- conding: utf-8 -*-
>>
>>
>> from subprocess import Popen, PIPE
>>
>> pesquisa = raw_input(" Digite a pesquisa de interesse: ")
>>
>> def Pesquisar(pesquisa):
>> p = Popen(["search", pesquisa],stdout=PIPE)
>> resultado = p.communicate()[0]
>> return resultado
>>
>>
>> print Pesquisar(pesquisa)
>>
>>
>>
>>
>> --
>> *" A Vida é arte do Saber...Quem quiser saber tem que Estudar!"*
>>
>> http://bucolick.tumblr.com
>> http://artecultural.wordpress.com/
>>
>>
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>>
>


-- 
*" A Vida é arte do Saber...Quem quiser saber tem que Estudar!"*

http://bucolick.tumblr.com
http://artecultural.wordpress.com/
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Snippet: The leanest Popen wrapper

2011-08-03 Thread Chris Rebert
On Wed, Aug 3, 2011 at 8:29 AM, Phlip  wrote:
> Groupies:
>
> This is either a code snippet, if you like it, or a request for a
> critique, if you don't.
>
> I want to call a command and then treat the communication with that
> command as an object. And I want to do it as application-specifically
> as possible. Anyone could think of a way to productize this:
>
> def command(*cmdz):
>
>    process = Popen( flatten(cmdz),

flatten() being defined as...?

>                     shell= True,

I would strongly encourage you to avoid shell=True. You really don't
want to have to worry about doing proper shell escaping yourself.

>                     stdout= subprocess.PIPE,
>                     stderr= subprocess.PIPE,
>                     bufsize= 4096 )

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list


RE: parsing function parameters

2011-08-03 Thread Lee Harr

>> I am trying to get some information about a function
>> before (and without) calling it.


> how about


def pdict(f):

    parameter_defaults = {}

    defaults = f.func_defaults

    defaultcount = len(defaults)

    argcount = f.func_code.co_argcount

    for i in xrange(f.func_code.co_argcount):

    name = f.func_code.co_varnames[i]

    value = None

    if i >= argcount - defaultcount:

    value = defaults[i - (argcount - defaultcount)]

    parameter_defaults[name] = value

    return parameter_defaults




> No need for the string parameters.
>
> Tim




That's it!



I saw the func_defaults, but could not see how to

make them match up with the co_varnames. I forgot

that keyword args must follow the positional args (duh).



I think this is going to work perfectly.


Thanks to all for the suggestions!


  
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Snippet: The leanest Popen wrapper

2011-08-03 Thread Phlip
> flatten() being defined as...?

Python Cookbook recipe 4.6

def flatten(sequence):  #  CONSIDER:  Reconcile with utils...
for item in sequence:
if isinstance(item, (list, tuple)):
for subitem in flatten(list(item)):
yield subitem
else:
yield item

>>                     shell= True,

> I would strongly encourage you to avoid shell=True. You really don't
> want to have to worry about doing proper shell escaping yourself.

Tx for helping me avoid reading up on it. I just copied it in.

I keep getting "fab not found" etc. when running 'fab command' thru
it. So then I ditch to os.system().

The long-term solution would be 'bash', '-c', 'yack yack yack' if you
want truly shelly things!

-- 
  Phlip
  http://c2.com/cgi/wiki?ZeekLand
-- 
http://mail.python.org/mailman/listinfo/python-list


RE: parsing function parameters

2011-08-03 Thread Lee Harr

Needed to make one change... for functions with no default args:


def pdict(f):
    parameter_defaults = {}
    defaults = f.func_defaults
    if defaults is not None:
    defaultcount = len(defaults)
    else:
    defaultcount = 0
    argcount = f.func_code.co_argcount
    for i in xrange(f.func_code.co_argcount):
    name = f.func_code.co_varnames[i]
    value = None
    if i >= argcount - defaultcount:
    value = defaults[i - (argcount - defaultcount)]
    parameter_defaults[name] = value
    return parameter_defaults

  
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Thomas Jollans
On 03/08/11 18:29, Dan Stromberg wrote:
> 
> On Wed, Aug 3, 2011 at 2:47 AM, Thomas Jollans  > wrote:
> 
> Is it more portable? I don't actually have cpio installed on this
> system.
> 
> 
> Interesting.  Of course, it's probably readily available to you.  What
> *ix are you seeing that doesn't include cpio by default?

Arch Linux - the base install is quite minimal. I just discovered that I
have a program called bsdcpio which is used by mkinitcpio (and possibly
other system scripts); no need for the GNU cpio. Curious.

>  
> 
> Which implementations of cp don't implement -R and -l?
> 
> 
> Probably most of them, except GNU and newer BSD.

Okay. While GNU libc manuals usually document how portable functions are
in detail, that's not true for the GNU coreutils manuals.

Thomas
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Equality check

2011-08-03 Thread Dave Angel

On 01/-10/-28163 02:59 PM, Abhishek Jain wrote:

How to check equality of two nos. in python

Python doesn't have numbers, it has various numeric data types.  If all 
you're concerned with are int types, then just use the obvous:


 if  a == b:
dosomething

If one or both might be floats, you have to deal with the likelihood 
that one or both may have quantization error, either caused by 
computational approximations, precision limitations, or conversion back 
and forth between string and internal binary representation.


If one is in Decimal, you have to mostly deal with the first two.

If one is complex, ...

DaveA


--
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Steven D'Aprano
Chris Angelico wrote:

>> In CPython at least, local lookups are faster
>> than globals: locals are stored in a fixed array, and the function knows
>> the numeric offset of each variable.
> 
> Ah! I was not aware of this, and thought that locals were a dictionary
> too. Of course, it makes a lot of sense. In that case, the classic
> "grab it as a local" isn't just loading down the locals dictionary
> with more names and thus slower lookups.

Er, not quite. Hash tables don't get slower as you add more items in them.

The difference is between:

(1) Search for "name" in globals dict;
(2) If not found, search for "name" in builtins dict;

(both searches being O(1) constant time to a first approximation) 

versus:

(1) Look in slot 5 in the function table of local variables


Name lookups involve hashing the string, then jumping to that position in a
hash table, then checking the name equals the key actually found. On
average, that requires constant time regardless of how many keys are in the
hash table (although the hash calculation and the equality tests might
depend on the length of the name). So the whole thing is predictably fast.
But not quite as fast as a simple jump to an offset to a table.

In theory, if you can arrange matters so that the dict is full of keys which
all hash the same, then performance will fall to O(N) instead of O(1), but
that would be *very* hard to do by accident. (A malicious coder might find
a way to fill your globals with a whole lot of three-letter variables that
happen to hash equal to that of "len", but then a malicious coder has
better ways of making your life miserable than slowing down name lookups.)


>> But, are you *sure* that name lookups are *really* the bottleneck? Name
>> lookups are pretty fast. If you want them faster, turn them into a local
>> variable. It's not clear to me that syntax, or a pragma directive, or
>> some other form of magic, is better than an explicit assignment:
> 
> No, I'm not sure. Unfortunately I have no convenient way to compare;

Have you tried using the profiler?


> all I can do is compare with a completely different language, which is
> of course NEVER going to be fair. The only actual data I have on the
> subject is the perfect-numbers search the other day; Pike managed the
> same task in a fraction of the time that Python spent on it. Pike has
> a single integer type that quietly switches from small to large at
> 2**32, with a noticeable performance change. This is the same as
> Python 2, but could explain some of the Python 3 penalty. There's only
> two obvious possibilities (there may be others, of course): firstly,
> that the actual name lookup is expensive; 

I wouldn't imagine that's a big factor, but I could be wrong.

> and secondly, that Pike is 
> able to optimize integer arithmetic by knowing that the value in
> question is an int, and it will never be anything else.

Much more likely.

Or that Pike's programming model is simpler and therefore code is faster, or
it has a smarter compiler... there could be many, many different reasons.

Or you cheated and used a slightly different algorithm in Pike :)


[...]
> So which is the better idiom?
> 
> def func(x):
>_len = len  # bind at function call time
>for i in x:
>_len(i)  # lookups of _len will be faster than len

That's the standard way of doing, er, early binding as late as possible :)

To do early binding as early as possible:

def func(x, len=len):  # binds at function definition time
for i in x:
len(i)


> or:
> 
> def func(x):
>len = len  # localize len
>for i in x:
>len(i)  # use it exactly as you otherwise would

That can't work. The problem is that because len is assigned to in the body
of the function, the Python compiler treats it as a local variable. So the
line len=len is interpreted as len = len, which doesn't yet
exist. There's no way of saying len = len in the body of the
function.

So you must either:

(1) use a different name: length = len

(2) use a fully-qualified name: import builtins; len = builtins.len

(3) do the assignment as a default parameter, which has slightly different
binding rules: def func(x, len=len)

(4) manual lookup: len = builtins.__dict__['len']  # untested


I don't recommend that last one, unless you're deliberately trying to write
obfuscated code :)



-- 
Steven

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Ned Deily
In article ,
 Grant Edwards  wrote:
> On 2011-08-03, Kushal Kumaran  wrote:
> > On Wed, Aug 3, 2011 at 7:29 AM, Dan Stromberg  wrote:
> >>
> >> On Tue, Aug 2, 2011 at 3:13 AM, Thomas Jollans  wrote:
> >>>
> >>> On 02/08/11 11:32, loial wrote:
> >>> > I am trying to hardlink all files in a directory structure using
> >>> > os.link.
> >>> >
> >>> > However I do not think it is possible to hard link directories ?
> >>
> >> That is pretty true.?? I've heard of hardlinked directories on Solaris, but
> >> that's kind of an exception to the general rule.
> >
> > In APUE, Richard Stevens says only root could do this,
> 
> Yep, in early versions of Solaris root could hard-link directories.
> I did it once, and it's not something one did a second time.  fsck
> couldn't deal with it and pretty much fell over.  IIRC, the only way
> to recover was to clear several inodes manually and then let fsck
> salvage things.
> 
> > if it is supported by the system at all.  In a footnote, he
> > additionally mentions he screwed up his filesystem by creating a loop
> > of hardlinked directories while writing that section of the book.
> 
> That sounds about right.
> 
> > I suppose it is a good thing systems don't allow that now.
> 
> It wouldn't be a problem, except there are some important places in
> Unix where it is assume that filesystems are trees.  Hard linking
> directories causes that assumption to be false.

FWIW, Apple implemented directory hard links for HFS+ file systems in 
Mac OS X 10.5 specifically to make the Time Machine incremental backup 
system work efficiently.  The hard directory links are, of course, not 
meant to be used by the casual user.  The section "Directory Hard Links" 
in this blog entry from Amit Singh explains what restrictions are 
enforced on directory hard links to prevent cycles:

http://osxbook.com/blog/2008/11/09/hfsdebug-40-and-new-hfs-features/

-- 
 Ned Deily,
 n...@acm.org

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Alain Ketterlin
Chris Angelico  writes:

[...]
> The only actual data I have on the subject is the perfect-numbers
> search the other day; Pike managed the same task in a fraction of the
> time that Python spent on it. Pike has a single integer type that
> quietly switches from small to large at 2**32, with a noticeable
> performance change. This is the same as Python 2, but could explain
> some of the Python 3 penalty. There's only two obvious possibilities
> (there may be others, of course): firstly, that the actual name lookup
> is expensive; and secondly, that Pike is able to optimize integer
> arithmetic by knowing that the value in question is an int, and it
> will never be anything else.

Pike is (at least partly) statically typed. Most of the lookups are
solved at compile time, and have therefore zero overhead at run-time. So
your second possibility is the good one, but probably not because of
arithmetic optims, rather because of all the lookups Pike doesn't
perform dynamically.

-- Alain.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Snippet: The leanest Popen wrapper

2011-08-03 Thread Terry Reedy

On 8/3/2011 11:29 AM, Phlip wrote:


This is either a code snippet, if you like it, or a request for a
critique, if you don't.


A learning exercise but pretty useless otherwise. As others pointed out, 
immediately stripping off \n is a bug relative to *your* function 
description. Also, you yourself then give an example of joining with \n. 
But that does not restore the final \n. The rest duplicates the 
iteration ability of the .stdout file object.


For repeated execution of code like
  process.stdout.readline()
you can once create a 'packetized' bound method object like so
  cmdline = process.stdout.readline
and then, without repeating the attribute lookups, repeatedly call
  cmdline()
.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Dan Stromberg
On Wed, Aug 3, 2011 at 11:49 AM, Thomas Jollans  wrote:

>
> > Interesting.  Of course, it's probably readily available to you.  What
> > *ix are you seeing that doesn't include cpio by default?
>
> Arch Linux - the base install is quite minimal. I just discovered that I
> have a program called bsdcpio which is used by mkinitcpio (and possibly
> other system scripts); no need for the GNU cpio. Curious.
>

I guess that makes some sense.  If you want to really strip down an install,
removing cpio is a good candidate since it duplicates what's in tar, and tar
is more popular - especially for interactive use.

> Which implementations of cp don't implement -R and -l?
>
>
> Probably most of them, except GNU and newer BSD.

Okay. While GNU libc manuals usually document how portable functions are
> in detail, that's not true for the GNU coreutils manuals.
>

I don't think cpio is in GNU coreutils.  Also, I think GNU cpio is a
reimplementation, not the original.

cpio's been around since PWB/Unix, which sits between 6th Edition Unix and
7th Edition.  It should be in just about everything, unless a
vendor/distributor got pretty zealous about cutting duplicate utilities.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Thomas Jollans
On 03/08/11 23:25, Dan Stromberg wrote:
> 
> On Wed, Aug 3, 2011 at 11:49 AM, Thomas Jollans  > wrote:
> 
> 
> > Interesting.  Of course, it's probably readily available to you.  What
> > *ix are you seeing that doesn't include cpio by default?
> 
> Arch Linux - the base install is quite minimal. I just discovered that I
> have a program called bsdcpio which is used by mkinitcpio (and possibly
> other system scripts); no need for the GNU cpio. Curious.
> 
> 
> I guess that makes some sense.  If you want to really strip down an
> install, removing cpio is a good candidate since it duplicates what's in
> tar, and tar is more popular - especially for interactive use.
> 
>> Which implementations of cp don't implement -R and -l?
>>
>>
>> Probably most of them, except GNU and newer BSD.
> 
> Okay. While GNU libc manuals usually document how portable functions are
> in detail, that's not true for the GNU coreutils manuals.
> 
> 
> I don't think cpio is in GNU coreutils.  Also, I think GNU cpio is a
> reimplementation, not the original.

Indeed. But cp is in the coreutils, and that was what we were talking about.

As for GNU cpio, that's simply what /usr/bin/cpio, if present, is
expected to be on a GNU/Linux system.

> 
> cpio's been around since PWB/Unix, which sits between 6th Edition Unix
> and 7th Edition.  It should be in just about everything, unless a
> vendor/distributor got pretty zealous about cutting duplicate utilities.
> 


-- 
http://mail.python.org/mailman/listinfo/python-list


Inconsistent SMTP/Gmail connection drop

2011-08-03 Thread Astley Le Jasper
I have a laptop that wakes up and then runs a script that collects
info and sends an email with a spreadsheet (zipped and about 350KB)
report to a number of people. However, every time I get the following
error:


File "/usr/lib/python2.7/smtplib.py", line 343, in getreply
raise SMTPServerDisconnected("Connection unexpectedly closed")


- When I check the laptop and manually run the function that sends the
email and report, it works fine. I have never been able to replicate
the error when doing it manually.
- It can't be a problem with Gmail at that time in the morning or my
connection, because the script also sends me a copy of the log file
straight after, and that works.
- The same code has been working for years, it just that I have
recently rebuilt the machine so now it is using Python 2.7. I have
also change the references to use absolute file references, rather
than relative. I also have changed the code to run as root.
- I've checked the smtp debug report and everything seems ok. I just
never get the confirmation code (250) to say that it's completed.
- Even stranger ... it ran ok once. But I didn't change a thing and
the next day it wouldn't work.

The only thing I can think of is that when the file is transferred to
the reports directory, it somehow isn't been released quickly enough
and perhaps this is tripping up SMTP. But ... this is a bit of a guess
given I've tried to exhaust all the other options.

Any ideas?

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Inconsistent SMTP/Gmail connection drop

2011-08-03 Thread David Stanek
On Wed, Aug 3, 2011 at 5:46 PM, Astley Le Jasper
wrote:

>
> Any ideas?
>


Is it possible that the first email is sent before the network connection
has been properly established?

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Early binding as an option

2011-08-03 Thread Chris Angelico
On Wed, Aug 3, 2011 at 8:51 PM, Steven D'Aprano
 wrote:
> Chris Angelico wrote:
>
>> Ah! I was not aware of this, and thought that locals were a dictionary
>> too. Of course, it makes a lot of sense. In that case, the classic
>> "grab it as a local" isn't just loading down the locals dictionary
>> with more names and thus slower lookups.
>
> Er, not quite. Hash tables don't get slower as you add more items in them.

Whoops, sloppy English there. The locals dictionary isn't slower
because it gets more stuffed in it (well, maybe a dict will slow down
if you stuff a few million things in it vs having five, but not with
this scale), but that it's slower than alternatives.

>> No, I'm not sure. Unfortunately I have no convenient way to compare;
>
> Have you tried using the profiler?

I can profile the code the way it is. I can't profile the code the way
it isn't, with static lookups. I could compare global vs local, but
not global/local vs no lookup at all.

>> and secondly, that Pike is
>> able to optimize integer arithmetic by knowing that the value in
>> question is an int, and it will never be anything else.
>
> Much more likely.

Pike separates variables from their values, just as Python does. I've
actually stuffed strings into variables that are supposed to be int
only, and things work fine (a bit confusing for the human though!).
But you're right that it's probably simplifying other lookups, not the
actual variable name. I think the same consideration applies; if you
know exactly what you're working with, if you assume that x.__add__ is
not going to change, then you can optimize.

> Or you cheated and used a slightly different algorithm in Pike :)

Heh. No, I kept the algorithm exactly the same. I'll post the code if
you like :)

>> def func(x):
>>    len = len  # localize len
>>    for i in x:
>>        len(i)  # use it exactly as you otherwise would
>
> That can't work. The problem is that because len is assigned to in the body
> of the function, the Python compiler treats it as a local variable. So the
> line len=len is interpreted as len = len, which doesn't yet
> exist. There's no way of saying len = len in the body of the
> function.

Duh. Forgot that. Without convenient syntax like "len = ::len" to do
this, it's not advisable.

> (4) manual lookup: len = builtins.__dict__['len']  # untested
>
> I don't recommend that last one, unless you're deliberately trying to write
> obfuscated code :)

For that reason! Although using builtins in this way isn't a good idea
- there's no reason to do early binding late if you just bind to the
same thing that early binding early would have done. And
globals()["len"] doesn't work either, because builtins aren't
globals... blargh, there's really no simple way to do this. It's a
good thing I don't *need* to do anything like this, because it's not
the syntactically simple optimization that I thought it would be!

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Inconsistent SMTP/Gmail connection drop

2011-08-03 Thread Astley Le Jasper
Not sure how I'd test or determine that. I've set smtp debug to true
and the output looks all good. I suppose I could try sending an email
before the report is sent, as well as the email that is sent after it.

On Thu, Aug 4, 2011 at 12:17 AM, David Stanek  wrote:
> On Wed, Aug 3, 2011 at 5:46 PM, Astley Le Jasper 
> wrote:
>>
>> Any ideas?
>>
>
> Is it possible that the first email is sent before the network connection
> has been properly established?
>
> --
> David
> blog: http://www.traceback.org
> twitter: http://twitter.com/dstanek
> www: http://dstanek.com
>
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Equality check

2011-08-03 Thread Chris Angelico
On Wed, Aug 3, 2011 at 8:15 PM, Dave Angel  wrote:
> If one is complex, ...

The situation is complex enough without bringing complex numbers into it!

OP, I recommend reading this page and then re-asking your question:
http://www.catb.org/~esr/faqs/smart-questions.html

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Hardlink sub-directories and files

2011-08-03 Thread Dan Stromberg
On Wed, Aug 3, 2011 at 2:54 PM, Thomas Jollans  wrote:

>
> On 03/08/11 23:25, Dan Stromberg wrote:
> > > Interesting.  Of course, it's probably readily available to you.
>  What
> > > *ix are you seeing that doesn't include cpio by default?
> >
> > Arch Linux - the base install is quite minimal.
>

H...

So we're probably not talking about portability to Android...  Because it's
too minimal, too different?

>> Which implementations of cp don't implement -R and -l?
>>
>>
>> Probably most of them, except GNU and newer BSD.
>
> Okay. While GNU libc manuals usually document how portable functions
are
> in detail, that's not true for the GNU coreutils manuals.

You don't really need it in the coreutils doc.

I'd feel very safe assuming that 98+% of all *ix's (not counting Android)
either have cpio installed by default, or can easily get it via their native
package manager.

The same isn't true of a cp with new features.

> I don't think cpio is in GNU coreutils.  Also, I think GNU cpio is a
> reimplementation, not the original.

Indeed. But cp is in the coreutils, and that was what we were talking about.
>

Sorry.  Maybe you should describe what you believe we -are- talking about,
then?  I guess I missed that.

I thought it was portability.


> As for GNU cpio, that's simply what /usr/bin/cpio, if present, is
> expected to be on a GNU/Linux system.
>

Well, yes, but:
1) GNU/Linux is important
2) GNU/Linux isn't the end all and be all

(I wrote:)
> cpio's been around since PWB/Unix, which sits between 6th Edition Unix
> and 7th Edition.  It should be in just about everything, unless a
> vendor/distributor got pretty zealous about cutting duplicate utilities.

I guess this is boiling down to "A pure python solution might be better,
since some folks don't like to use their package managers"

Also, a pure python solution should work on windows - these shell commands
don't unless you have one of the many ports of *ix tools to windows
installed.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Inconsistent SMTP/Gmail connection drop

2011-08-03 Thread Dan Stromberg
Some things to consider:

1) You might see if there's something about the size of the message - is it
bigger after collecting data all night?  Is google disconnecting after a
maximum amount of data is transferred?

2) You might try sending a tiny test message at the beginning, just to
yourself, and seeing if that has the problem

3) You might try running your script under a system call tracer:
http://stromberg.dnsalias.org/~dstromberg/debugging-with-syscall-tracers.html-
and inspecting the list of syscalls and their return codes near when
the
process terminates

4) I get disconnects from gmail once in a while too, especially when
downloading a large list of new message headers; I've been just coding
around it by making my code idempotent and otherwise restartable.  This
seems to happen with both 2.x and 3.x, so I assume it's not really related
to the Python version, but I suppose I should try it on PyPy or Jython
someday.  Note that the error message in 3.x was better than in 2.x.

HTH

On Wed, Aug 3, 2011 at 2:46 PM, Astley Le Jasper
wrote:

> I have a laptop that wakes up and then runs a script that collects
> info and sends an email with a spreadsheet (zipped and about 350KB)
> report to a number of people. However, every time I get the following
> error:
>
>
> File "/usr/lib/python2.7/smtplib.py", line 343, in getreply
>raise SMTPServerDisconnected("Connection unexpectedly closed")
>
>
> - When I check the laptop and manually run the function that sends the
> email and report, it works fine. I have never been able to replicate
> the error when doing it manually.
> - It can't be a problem with Gmail at that time in the morning or my
> connection, because the script also sends me a copy of the log file
> straight after, and that works.
> - The same code has been working for years, it just that I have
> recently rebuilt the machine so now it is using Python 2.7. I have
> also change the references to use absolute file references, rather
> than relative. I also have changed the code to run as root.
> - I've checked the smtp debug report and everything seems ok. I just
> never get the confirmation code (250) to say that it's completed.
> - Even stranger ... it ran ok once. But I didn't change a thing and
> the next day it wouldn't work.
>
> The only thing I can think of is that when the file is transferred to
> the reports directory, it somehow isn't been released quickly enough
> and perhaps this is tripping up SMTP. But ... this is a bit of a guess
> given I've tried to exhaust all the other options.
>
> Any ideas?
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list


making my extensions work together

2011-08-03 Thread Mathew
This isn't exactly a Python question but maybe someone here has run into 
this.

I have 2 extensions and they both access a function in a (static) library. 
The function maintains state information using a static variable.

This doesn't work. When one of my extensions changes the variable value, the 
other extension does not see the change.

Would it work if I made my library dynamic?

This is on Windows XP compiling with MSVC 2008.

-Mathew 


-- 
http://mail.python.org/mailman/listinfo/python-list


Re: range() vs xrange() Python2|3 issues for performance

2011-08-03 Thread Steven D'Aprano
harrismh777 wrote:

> def perf(n):
>  sum = 0
>  for i in range(1, n):
>  if n % i == 0:
>  sum += i
>  return sum == n


A more effective way to speed that up is with a better algorithm:

def isperfect(n):
if n < 2: return False
total = 1
for i in range(2, int(n**0.5)+1):
a, b = divmod(n, i)
if b == 0:
total += a+i
return total == n
 
For n=33550336 it loops about 5791 times instead of 33550335, reducing the
amount of work done by about 99.98%. 

isperfect is fast enough to exhaustively test every number up to a low
limit: testing every number between 0 and 8128 inclusive took about 0.38
second in total on my PC. That's an average of 0.05 millisecond per number
tested. Unfortunately, it doesn't scale. isperfect(33550336) takes about
4.4 ms, and isperfect(8589869056) about 84 ms. 

However, it is still (barely) feasible to exhaustively test every number up
to a much higher range. I extrapolate that the time needed to check every
value between 0 and 33550336 would take about 30 hours.

If you're wondering why exhaustive testing is so expensive, it's because the
test code simply calls isperfect(n) for each n. The number of iterations in
each call to isperfect is approximately sqrt(n), so the total in any
exhaustive test is approximately sum(x**0.5 for x in range(upperlimit)), 
which gets rather large.


-- 
Steven

-- 
http://mail.python.org/mailman/listinfo/python-list


Community Involvement

2011-08-03 Thread Steve Holden
[Ccs appreciated]

After some three years labor I (@holdenweb) at last find myself approaching the 
completion of the Python Certificate Series with O'Reilly School of Technology 
(@OReillySchool).

At OSCON last week the team fell to talking about the final assignment 
(although the Certificate is not a certification, students only progress by 
answering real quiz questions, not the usual multiple-choice task). Success 
also requires that they complete a project at the end of each (of the ~60) 
lesson(s).

We would ideally like the last project to to be something that demonstrates at 
least some minimal involvement with the Python community. Something like "get a 
Python answer upvoted on StackOverflow", for example, or getting a question 
answered on c.l.p. At the same time it shouldn't be anything that places a 
burden on the community (otherwise the hundredth student would be abused and 
the thousandth murdered).

So I wondered if anyone had any good ideas.

regards
 Steve
-- 
Steve Holden
st...@holdenweb.com



-- 
http://mail.python.org/mailman/listinfo/python-list


Re: making my extensions work together

2011-08-03 Thread John Gordon
In  "Mathew"  writes:

> This isn't exactly a Python question but maybe someone here has run into 
> this.

> I have 2 extensions and they both access a function in a (static) library. 
> The function maintains state information using a static variable.

I'm not sure what you mean by "extension", but it might not be relevant.

Are these extensions, whatever they may be, part of the same program?
Or are they separate?

> This doesn't work. When one of my extensions changes the variable value, the 
> other extension does not see the change.

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Notifications when process is killed

2011-08-03 Thread chrisallick
On Aug 1, 11:39 am, Andrea Di Mario  wrote:
> Thanks Thomas, it is what i'm looking for.
>
> Regards
>
> --
> Andrea Di Mario

Catch a Kill:

def signal_handler(signal, frame):
print "Received exit command."
#server.running = False
sys.exit()

signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)

Find and Kill A Process:

import os, signal

process = "websocket.py"

found = False
for line in os.popen("ps ax | grep python"):
fields = line.split()
pid = fields[0]
for field in fields:
if field.find(process) >= 0:
print pid
print field
os.kill(int(pid), signal.SIGTERM)
found = True
break
if found == True:
break

if found == True:
print "found and killed web server process."
else:
print "could not find web server process.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Notifications when process is killed

2011-08-03 Thread chrisallick
On Aug 1, 5:56 am, Andrea Di Mario  wrote:
> Hi, i've created a twisted server application and i want that the
> server send me a message when someone stops or kills the process.
> I want to override reactor.stop(), but do this way send me message
> when the process is stopped by a system kill?
> Could you suggest me if there's a way to do this?
>
> Thanks, regards.
>
> --
> Andrea Di Mario

This will catch Ctrl+C and a Kill PID request:

# Add SIGINT handler for killing the threads
def signal_handler(signal, frame):
print "Received exit command."
server.running = False
sys.exit()

signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)

And this will find an destroy a process:

import os, signal

process = "websocket.py"

found = False
for line in os.popen("ps ax | grep python"):
fields = line.split()
pid = fields[0]
for field in fields:
if field.find(process) >= 0:
print pid
print field
os.kill(int(pid), signal.SIGTERM)
found = True
break
if found == True:
break

if found == True:
print "found and killed web server process."
else:
print "could not find web server process."
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: parsing in python

2011-08-03 Thread John L. Stephens
Depending on what you want to do, you might try looking at the pyparsing 
module.  I have used it to successfully parse sentences looking for 
keywords and structures.


On 8/3/2011 9:26 AM, Jayron Soares wrote:

Hi folks,

I've created a simple method to grab files texts from directory by 
words random, however I figure out that I need extract the content 
inside of each file, in fact I believe  I have to create a parsing, 
nonetheless I don't know how to create a parser.

Please some could share some tips to do it?
Cheers
Jayron
here is the code:

# -*- conding: utf-8 -*-


from subprocess import Popen, PIPE

pesquisa = raw_input(" Digite a pesquisa de interesse: ")

def Pesquisar(pesquisa):
p = Popen(["search", pesquisa],stdout=PIPE)
resultado = p.communicate()[0]
return resultado


print Pesquisar(pesquisa)




--
/" A Vida é arte do Saber...Quem quiser saber tem que *Estudar*!"/

http://bucolick.tumblr.com 
http://artecultural.wordpress.com/





-- 
http://mail.python.org/mailman/listinfo/python-list


python module to determine if a machine is idle/free

2011-08-03 Thread Danny Wong (dannwong)
Hi all,

I have 5 server machines that are using to process
information. I would like to write a quick server python script that
determines which of the machines are not in use. Any recommendations on
which python module I should use to detect if a machine is not
performing idle (ex. Some specific task is not running)?

-- 
http://mail.python.org/mailman/listinfo/python-list


HID Device Drivers for OS X

2011-08-03 Thread chrisallick
Hi,
   I spent almost all day trying to figure out how to capture a click
from this button:

http://www.dreamcheeky.com/big-red-button

I tried libusb, PyUSB, HIDAPI and probably a couple other things.  No
luck.  My understanding is that libusb won't allow me to get at the
interface and the example I found using PyUSB to read a magnetic card
reader errors with a segmentation fault in the libusb library.

Can anyone point me in the right direction?  I looked the OS X I/O Kit
at developer.apple.com, but it seems really confusing and over kill.
I have the vendor ID and product ID, it seems like it would be really
simple to read when the button is clicked.

Any thoughts?
-- 
http://mail.python.org/mailman/listinfo/python-list


Inconsistencies with tab/space indentation between Cygwin/Win32?

2011-08-03 Thread Christian Gelinek
Hi all,

I have a problem running some python program using version 2.6.4 (or version
2.7.2, I tried both) from the Win7 command line - it never finishes due to
an infinite loop. The thing is, when I run the same program through Cygwin
which uses Python version 2.6.5, it exits the loop at some point.

I came to try this after I realised that in some of the sources (it is a
rather big program using many source files, most of them being created by
others from a Linux environment), the indentation is mixed tabs/spaces where
the assumed tab size is 8 spaces.

Reading on the Python website, a tab size of 8 is default anyway, so I would
have assumed it should work... does that mean that one tab equals 2
indents?!? I myself never use tabs to indent Python code but let my editor
do a tab-to-4-spaces conversion when I press .

I find it at least confusing to read that Python expects a tab size of 8 but
at the same time uses 4 spaces for one indent level. Or maybe I am
completely on the wron track here?

Any ideas on how to get the thing to run under (real) Windows, hopefully
without having to edit existing sources of people who left our company ages
ago?


-- 
http://mail.python.org/mailman/listinfo/python-list


Re: python module to determine if a machine is idle/free

2011-08-03 Thread Chris Rebert
On Wed, Aug 3, 2011 at 9:06 PM, Danny Wong (dannwong)
 wrote:
> Hi all,
>
>     I have 5 server machines that are using to process
> information. I would like to write a quick server python script that
> determines which of the machines are not in use. Any recommendations on
> which python module I should use to detect if a machine is not performing
> idle (ex. Some specific task is not running)?

Yes, psutil:
http://code.google.com/p/psutil/

os.getloadavg() may or may not also be useful to you:
http://docs.python.org/library/os.html#os.getloadavg

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: Inconsistencies with tab/space indentation between Cygwin/Win32?

2011-08-03 Thread Thorsten Kampe
* Christian Gelinek (Thu, 4 Aug 2011 13:55:37 +0930)
> Any ideas on how to get the thing to run under (real) Windows,
> hopefully without having to edit existing sources of people who left
> our company ages ago?

python -t
"Issue a warning when a source file mixes tabs and spaces for
indentation in a way that makes it depend on the worth of a tab
expressed in spaces. Issue an error when the option is given twice."

Thorsten
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: making my extensions work together

2011-08-03 Thread Stefan Behnel

Mathew, 04.08.2011 03:19:

This isn't exactly a Python question but maybe someone here has run into
this.

I have 2 extensions


With "extensions", I assume you mean extension modules for the CPython 
runtime that are written in C? It would help if you were more specific in 
your problem description.




and they both access a function in a (static) library.


It would help if you mentioned the names of the modules (or packages) and 
of the external library.




The function maintains state information using a static variable.


That's bad design, but it happens.



This doesn't work. When one of my extensions changes the variable value, the
other extension does not see the change.


Are the two modules linked in any way or are they just arbitrary modules 
that happen to be installed at the same time, trying to use the same 
external C library?


If the former, consider letting them communicate with each other by making 
one depend on the other. If you control the source code, you may also 
consider wrapping the library only once and reusing that from the two 
modules. Or, just move the configuration part into a separate module and 
have both depend on that. Or, try to dump the dependency on the static 
variable.


If the latter, then, well, it depends on several environmental factors that 
you left out in your question.




Would it work if I made my library dynamic?


That's usually a good idea, but it's unlikely to help in this specific case.



This is on Windows XP compiling with MSVC 2008.


Others will know more here.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list