Re: Pickling over a socket

[Chris Angelico]

On Wed, Apr 20, 2011 at 4:44 PM, Bastian Ballmann  wrote:
> Yes pickle is like eval, but that doesnt mean that one should never
> ever use it over a socket connection.
> What about ssl sockets where client and server authenticate each other?
> Or you encrypt the pickle dump with symmetric encryption and only load
> it if you can decrypt it? There are ways to ensure that the data you
> get can be handled as trusted.

No, I disagree. And I'll cite Caesary as evidence of why.

Caesary is a multiplayer game that uses Flash as its client. (I'm told
the back end is Java, which would explain why it starts lagging
horribly when everyone's online at once.) It has some measure of
authentication of the client, but it's not difficult to spoof;
obviously you could go more elaborate and harder to spoof, but that
still doesn't solve the problem. Even public/private key systems won't
work here; someone could get hold of your client and its private key,
and poof.

Caesary uses an Adobe Message Format system, whereby complex objects
get serialized and transmitted in both directions. It's fundamentally
the same as pickling. When I started poking around with things, it
took me very little time to start transmitting my own requests to the
server; my requests were benign (asking it for information), but other
people figured out the same thing and were rather less ethical.

That's why I tend to use and create much simpler protocols for network
transmission. Also, I like to use a MUD client to test my servers,
ergo textual protocols similar to SMTP. Sure, it may be a tad more
verbose than some, but it's usually easy to parse and verify.

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

freesoftwaredownload

[need life partnerkumar]

http://123maza.com/25/line526/
-- 
http://mail.python.org/mailman/listinfo/python-list

Writing Exif File

[Jean-Pierre M]

I want to enter Comments of a picture in a JPeg file.

Is there  a library in Python which allow me to do that without  having to
reinvent the wheel?

The target is to read those comments in my private webSite using the php
exif_read_data function (http://php.net/manual/fr/book.exif.php)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Tim Roberts]

Chris Angelico  wrote:

>On Wed, Apr 20, 2011 at 1:45 PM, Chris Rebert  wrote:
>> Built-ins aren't quite the same as globals, but essentially yes:
>
>Sure. That might explain some of the weirdness, but it doesn't explain
>why things were still weird with the variable named posttype. 

It's because, unlike some other languages (like Pascal), Python doesn't
have infinitely recursive nested namespaces.  Glossing over details, there
is a global namespace, and there is a local namespace.  A new function gets
a new local namespace.  "posttype" is part of the local namespace of the
outer function, but it's not part of the local namespace of the lambda.

You can solve this through the common lamba idiom of a closure:

lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Pairwise count of frequency from an incidence matrix of group membership

[Shafique, M. (UNU-MERIT)]

Hi,
I have a number of different groups g1, g2, … g100 in my data. Each group is 
comprised of a known but different set of members from the population m1, m2, 
…m1000. The data has been organized in an incidence matrix:
g1g2g3g4g5
m01
m210010
m301100
m411011
m500110

I need to count how many groups each possible pair of members share (i.e., both 
are member of). 
I shall prefer the result in a pairwise edgelist with weight/frequency in a 
format like the following:
m1, m1, 4
m1, m2, 1
m1, m3, 2
m1, m4, 3
m1, m5, 1
m2, m2, 2
... and so on.

I shall highly appreciate if anybody could suggest/share some code/tool/module 
which could help do this.

Best regards,
Muhammad
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pickling over a socket

[Bastian Ballmann]

Am Wed, 20 Apr 2011 16:59:19 +1000
schrieb Chris Angelico :
 
> Even public/private key systems won't
> work here; someone could get hold of your client and its private key,
> and poof.

Oh yeah but than all kinds of trusted computing wont work. Sure
one can see it on the net these days looking at the rsa or commodo or
ps3 hack and the like.

No system is totally secure. You can _always_ poke around if a program
uses user input. For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters. Now would you say one shouldnt use sql
databases cause of that? ;)

My point is using ssl authentication / encryption together with another
symmetric encryption builds up two layers, which I would say is secure
enough to handle the data as trusted.

Greets

Basti


signature.asc
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Vectors

[Algis Kabaila]

Hi,

Are there any modules for vector algebra (three dimensional 
vectors, vector addition, subtraction, multiplication [scalar 
and vector]. Could you give me a reference to such module?

platform - ubuntu 10.10 (Linux), Python 3.1 or higher.

Thanks for your help to avoid re-invention of the wheel.

OldAl.

-- 
Algis
http://akabaila.pcug.org.au/StructuralAnalysis.pdf
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing Exif File

[Chris Rebert]

On Wed, Apr 20, 2011 at 12:00 AM, Jean-Pierre M
 wrote:
> I want to enter Comments of a picture in a JPeg file.
> Is there  a library in Python which allow me to do that without  having to
> reinvent the wheel?
> The target is to read those comments in my private webSite using the php
> exif_read_data function (http://php.net/manual/fr/book.exif.php)

Possibilities:
pyexif - http://pypi.python.org/pypi/pyexif/
pexif - http://pypi.python.org/pypi/pexif/

In the future, try searching PyPI, Python's rough equivalent of Perl's CPAN:
http://pypi.python.org/pypi

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Vectors

[Chris Rebert]

On Wed, Apr 20, 2011 at 12:47 AM, Algis Kabaila  wrote:
> Hi,
>
> Are there any modules for vector algebra (three dimensional
> vectors, vector addition, subtraction, multiplication [scalar
> and vector]. Could you give me a reference to such module?

Dunno if it has 3D-specific features, but NumPy is the gold standard
for matrix-y stuff in Python:
http://numpy.scipy.org/

And apparently it's even Python 3.1-compatible now; that's quite a feat.

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Vectors

[Anssi Saari]

Algis Kabaila  writes:

> Are there any modules for vector algebra (three dimensional 
> vectors, vector addition, subtraction, multiplication [scalar 
> and vector]. Could you give me a reference to such module?

NumPy has array (and matrix) types with support for these basic
operations you mention. See the tutorial at http://numpy.scipy.org/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pairwise count of frequency from an incidence matrix of group membership

[Peter Otten]

Shafique, M. (UNU-MERIT) wrote:

> Hi,
> I have a number of different groups g1, g2, … g100 in my data. Each group
> is comprised of a known but different set of members from the population
> m1, m2, …m1000. The data has been organized in an incidence matrix:
> g1g2g3g4g5
> m01
> m210010
> m301100
> m411011
> m500110
> 
> I need to count how many groups each possible pair of members share (i.e.,
> both are member of).
> I shall prefer the result in a pairwise edgelist with weight/frequency in
> a format like the following:
> m1, m1, 4
> m1, m2, 1
> m1, m3, 2
> m1, m4, 3
> m1, m5, 1
> m2, m2, 2
> ... and so on.
> 
> I shall highly appreciate if anybody could suggest/share some
> code/tool/module which could help do this.

Homework? What have you tried?

One strategy is to create a list of sets containing the groups from the 
initial matrix

matrix = [
[1, 1, 1, 0, 1],
[1, 0, 0, 1, 0],
]

sets = [ # zero-based indices
   set([0,1,2,4]),
   set([0,3]),
   ...
]

The enumerate() builtin may help you with the conversion. You can then find 
the shared groups with set arithmetic:

sets[0] & sets[1] #m1/m2


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pickling over a socket

[Thomas Rachel]

Am 20.04.2011 09:34, schrieb Bastian Ballmann:


No system is totally secure. You can _always_ poke around if a program
uses user input.


It depends on what the program does with the input. If it treats it 
appropriately, nothing can happen.




For example one can totally own a complete computer by
nothing more than a single sql injection attack even if the programmer
implemented some filters.


What do yu want with filters here? Not filtering is appropriate against 
SQL injection, but escaping.


If Little Bobby Tables is really called "Robert'); DROP TABLE STUDENTS; 
--", it is wrong to reject this string - instead, all dangerous 
characters inside it must be quoted (in this case: ') and then it does 
not harm at all.



> Now would you say one shouldnt use sql

databases cause of that? ;)


No, just beware of what can happen and use the dbs and its functions 
appropriately.



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Chris Angelico]

On Wed, Apr 20, 2011 at 5:16 PM, Tim Roberts  wrote:
> It's because, unlike some other languages (like Pascal), Python doesn't
> have infinitely recursive nested namespaces.  Glossing over details, there
> is a global namespace, and there is a local namespace.  A new function gets
> a new local namespace.  "posttype" is part of the local namespace of the
> outer function, but it's not part of the local namespace of the lambda.

Okay, well that saves me from going insane anyhow! (Although it may be
too late to save me from that.) I skimmed your email and then tried to
explain it all to my boss, who has some programming experience but
only rudimentary Python, and after ten or fifteen minutes we concluded
that I needed to put some comments in the code.

> You can solve this through the common lamba idiom of a closure:
>
> lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)

Seems a little odd, but sure. I guess this means that a function's
default arguments are evaluated in the parent context, but the body is
evaluated in its own context?

>From what I understand, though, from __future__ import nested_scopes
became a permanent part of the language long before 2.6.6, and PEP 227
describes this exact situation of lambda functions:
http://www.python.org/dev/peps/pep-0227/ gives an example involving
Tkinter. Unfortunately googling for 'python nested scope' mainly turns
up old information from 2.1 and thereabouts; has anything changed
since then?

Infinitely-nested scopes is definitely how I, as a programmer, tend to
think about my code. Any place where the computer's interpretation of
the code differs from mine is a place where either the computer needs
to change its thinking, or I do. And it's usually easier to change me,
except that I use so many languages. :)

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

[OT] Re: Pickling over a socket

[Bastian Ballmann]

Am Wed, 20 Apr 2011 10:25:14 +0200
schrieb Thomas Rachel
:

> It depends on what the program does with the input. If it treats it 
> appropriately, nothing can happen.

Yes, but the question seems to be what is appropriately.
 

> What do yu want with filters here? Not filtering is appropriate
> against SQL injection, but escaping.

Escaping in strings, filtering with numbers etc.

 
> If Little Bobby Tables is really called "Robert'); DROP TABLE
> STUDENTS; --", it is wrong to reject this string - instead, all
> dangerous characters inside it must be quoted (in this case: ') and
> then it does not harm at all.

Well you forgot to escape ; and \ but this seems to slide into OT ;)
Greets

Basti


signature.asc
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pickling over a socket

[Chris Angelico]

On Wed, Apr 20, 2011 at 7:17 PM, Bastian Ballmann  wrote:
> Well you forgot to escape ; and \ but this seems to slide into OT ;)

The semicolon doesn't need to be escaped in a quoted string, and the
backslash does only if it's the escape character. The
string-safetifier function that I used with DB2 was called "dblapos"
because it simply doubled every apostrophe - nothing else needed. On
the other hand, mysql_real_escape_string will escape quite a few
characters, for convenience in reading dumps.

> Am Wed, 20 Apr 2011 18:43:01 +1000
> schrieb Chris Angelico :
>
>> So, like Jean-Paul said, you simply do not trust anything that comes
>> from the network. Ever.
>
> If you generalize it in this way you should never trust any user input
> regardless if it comes from the net or from local or the environment
> etc.

Yes, but the other half of the issue is that you have to treat
anything that comes over the network as "user input", even if you
think it's from your own program that you control.

>> Urrrm. You can "own" a "complete computer" with SQL injection? Then
>> someone has some seriously weak protection.
>
> Yes and the database is poorly protected, but this happens way too
> often.

That's just *sad*.

>> SQL injection is easier to
>> protect against than buffer overruns, and with a lot of function
>> libraries
>
> I totally disagree. Buffer overflow is just a matter of size checking,
> but sql injection is a matter of syntax. It's more than just throwing
> the input into a magic auto-escape function.

Buffer overruns can happen in all sorts of places; SQL injection can
only happen where you talk to the database. And it IS just a matter of
using a magic auto-escape function, if your library is set up right -
unless, of course, you allow your users to submit SQL themselves (eg a
WHERE clause). That's almost impossible to sanitize, which is why I
would never EVER allow such a thing unless it's actually a trusted
environment (eg PHPMyAdmin - anyone who has access to PMA has access
to the database anyway).

> We both agree that one should never trust user input blindly and we also
> seem to conform to one can use user input in a appropriate way that's
> not the case, but if i read your mail i think you want to tell me one
> should never ever use the internet or only write programs without user
> input at all.

Not at all; just never *trust* user input. Where thou typest foo,
someone someday will type...

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pickling over a socket

[Bastian Ballmann]

Am Wed, 20 Apr 2011 19:26:44 +1000
schrieb Chris Angelico :

> Yes, but the other half of the issue is that you have to treat
> anything that comes over the network as "user input", even if you
> think it's from your own program that you control.

Sure.

 
> Buffer overruns can happen in all sorts of places; SQL injection can
> only happen where you talk to the database. And it IS just a matter of
> using a magic auto-escape function, if your library is set up right -

No. Not all data is strings.


> Not at all; just never *trust* user input. Where thou typest foo,
> someone someday will type...

I never *trust* the user *blindly* as you do with your
magic-escape-function so where do we disagree?
Greets

Basti


signature.asc
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Language & lib reference in man format ?

[Andre Majorel]

If someone has ever written a script to convert the Python
Language Reference and Library Reference to man format, I'm
interested.

Thanks in advance.

-- 
André Majorel http://www.teaser.fr/~amajorel/
"The object of this year's expedition is to see if we can find
trace of last year's expedition."
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: meteclasses 2.x/3.x compatibility

[Steven D'Aprano]

On Wed, 20 Apr 2011 11:56:35 +1000, James Mills wrote:

> Hi all,
> 
> Is there a compatible way to use meteclasses in both Python 2.x (2.6 to
> 2.7) and Python 3.x (3.0 to 3.2).


Untested:

if sys.version >= "3":
kw = {'metaclass': MyMetaClass}
else:
kw = {}

class Foo(object, **kw):
if sys.version < "3":
__metaclass__ = MyMetaClass


Inelegant as it gets, but it should work.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Sherm Pendley]

Grant Edwards  writes:

> I'm trying to implement a device discovery/configuration protocol that
> uses UDP broadcast packets to discover specific types of devices on
> the local Ethernet segment.  The management program broadcasts a
> discovery command to a particular UDP port.  All devices who get that
> packet are expected to answer regardless of thier current IP address.

Have you looked at the source for Apple's Bonjour?

  
  

Might be interesting to see how it does announcement/discovery. Or maybe
just use it directly, if this happens to be a case of "gee, I didn't
know that wheel had already been invented." :-)

sherm--

-- 
Sherm Pendley
   
Cocoa Developer
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: No more Python support in NetBeans 7.0

[Markus]

I read it too.
I always preferred Netbeans + their Python plugin over Eclipse and
PyDev.

Perhaps I have another look for working with Aptana + PyDev for my web
development stuff, but I am afraid this enviroment (and the base,
Eclipse as it's main reason) is as user unfriendly as it always was.
But long years there was no other choice than Eclipse - Netbeans wasnt
the IDE it is now - and so the people started to build plugins for
Eclipse even being aware of its shortcomings.

I think I check out Activestates Komodo IDE.
I recognized that I use their free Komodo Edit more often lately and
liked the Editor already, perhaps the IDE will be a good choice.

I've seen right now, they put on the topic:
http://www.activestate.com/blog/2011/03/netbeans-drops-python-support-komodo-adds-more

Infoworld awarded it as best Python IDE, testing: Boa Constructor,
Eric, ActiveState's Komodo, Oracle's NetBeans, Aptana's Pydev,
PyScripter, SPE, Spyder, and WingWare's Wing IDE.

And if all fails, remember, there is always vim to fallback :)



On Mar 24, 4:32 pm, Kees Bakker  wrote:
> Hi,
>
> Sad news (for me, at least), in the upcoming version 7.0 of NetBeans
> there will be no Python plugin anymore.
>
> I have been using NetBeans for Python development for a while now
> and I was very happy with it.
>
> See this archive for 
> details:http://netbeans.org/projects/www/lists/nbpython-dev/archive/2010-11/m...http://netbeans.org/projects/www/lists/nbpython-dev/archive/2011-01/m...
> --
> Kees


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Steven D'Aprano]

On Wed, 20 Apr 2011 13:10:21 +1000, Chris Angelico wrote:

> Context: Embedded Python interpreter, version 2.6.6
> 
> I have a list of dictionaries, where each dictionary has a "type"
> element which is a string. I want to reduce the list to just the
> dictionaries which have the same "type" as the first one.

[snip discussion and code]

It should, and does, work as expected, both in the global scope:


>>> lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"}, 
...{42: None, "type": "CALC"}]
>>> t = lst[0]["type"].lower()
>>> 
>>> filter(lambda x: x["type"].lower() == t, lst)
[{'type': 'calc'}, {42: None, 'type': 'CALC'}]
>>> [i for i in lst if i["type"].lower() == t]
[{'type': 'calc'}, {42: None, 'type': 'CALC'}]

and in a function:

>>> def test():
... lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},
...{42: None, "type": "CALC"}]
... t = lst[0]["type"].lower()
... print filter(lambda x: x["type"].lower() == t, lst)
... print [i for i in lst if i["type"].lower() == t]
...
>>> test()
[{'type': 'calc'}, {42: None, 'type': 'CALC'}]
[{'type': 'calc'}, {42: None, 'type': 'CALC'}]


[...]
> If I use the filter() method, the resulting list is completely empty. If
> I use the list comprehension, it works perfectly. Oddly, either version
> works in the stand-alone interpreter.

Let me guess... you're using an IDE?

There's your problem. IDEs often play silly buggers with the environment 
in order to be "clever". You've probably found a bug in whatever IDE 
you're using.

And this is why I won't touch the buggers with a 30 ft pole, at least not 
for anything serious.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Teaching Python

[Steven D'Aprano]

On Wed, 20 Apr 2011 10:06:27 +1000, Ben Finney wrote:

> Dan Stromberg  writes:
> 
>> On Tue, Apr 19, 2011 at 4:03 PM, geremy condra 
>> wrote:
>> > When you say 'hacking', you mean ?
>>
>> Presumably he meant the real meaning of the word, not what the press
>> made up and ran with.
> 
> To be fair, the press already had its own pejorative meaning of “hack”
> before the engineering and computing term, so the association was
> probably inevitable.

It's hardly just the press. "Hack" is a fine old English word:

"The jungle explorer hacked at the undergrowth with his machete."

"I was so hungry, I didn't take the time to neatly slice up the meat, but 
just hacked off a chunk and stuffed it in my mouth."

"Good lord, have you seen the completely botched job that carpenter has 
done? He's such a hack!"

Given the wide range of pejorative meanings of "hack" going back at least 
to the 19th century (to cut roughly without skill, a mediocre and 
talentless writer, a person engaged to perform unskilled and boring 
labour, a broken-down old horse, etc.), what's remarkable is that anyone 
decided to start use "hack" in a non-pejorative sense.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Thomas Heller]

Am 20.04.2011 00:21, schrieb Grant Edwards:

I'm have problems figuring out how to receive UDP broadcast packets on
Linux.

[...]


On the receiving machine, I've used tcpdump to verify that broadcast
packets are being seen and have a destination IP of 255.255.255.255 and
destination MAC of ff:ff:ff:ff:ff:ff

[...]

But, the receiving Python program never sees any packets unless the
_source_ IP address in the packets is on the same subnet as the
receiving machine.  In this test case, the receiving machine has an IP
address of 172.16.12.34/16.  If I change the receiving machine's IP
address to 10.0.0.123, then the receiving program sees the packets.

Even though the destination address is 255.255.255.255, the receiving
machine appears to discard the packets based on the _source_ IP.  Can
anybody provide example Python code for Linux that receives UDP
broadcast packets regardless of their source IP address?

This probably is more of a Linux networking question than a Python
question, but I'm hoping somebody has solved this problem in Python.



You must set the network interface to promiscous mode on the receiving side:

os.system("ifconfig eth0 promisc")

Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Adam Tauno Williams]

On Wed, 2011-04-20 at 06:07 -0400, Sherm Pendley wrote:
> Grant Edwards  writes:
> > I'm trying to implement a device discovery/configuration protocol that
> > uses UDP broadcast packets to discover specific types of devices on
> > the local Ethernet segment.  The management program broadcasts a
> > discovery command to a particular UDP port.  All devices who get that
> > packet are expected to answer regardless of thier current IP address.
> Have you looked at the source for Apple's Bonjour?

On LINUX this is called "avahi", which has Python bindings.  Avahi
auto-configuration / discovery works very well.






See also:

at least you may be able to lift code from them (License is non-viral
MIT)

> Might be interesting to see how it does announcement/discovery. Or maybe
> just use it directly, if this happens to be a case of "gee, I didn't
> know that wheel had already been invented." :-)

-- 
Adam Tauno Williams  LPIC-1, Novell CLA

OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: No more Python support in NetBeans 7.0

[Paul Rubin]

Markus  writes:
> Infoworld awarded it as best Python IDE, testing: Boa Constructor,
> Eric, ActiveState's Komodo, Oracle's NetBeans, Aptana's Pydev,
> PyScripter, SPE, Spyder, and WingWare's Wing IDE.

I saw somebody using Geany recently and it looked pretty impressive.
For Python gui debuggers, winpdb.org is great.
-- 
http://mail.python.org/mailman/listinfo/python-list

PyCon Australia 2011: registrations now open

[Ryan Kelly]

Hi Everyone,


I'm pleased to announce that registrations are now open for PyCon
Australia 2011.

PyCon Australia is Australia's only conference dedicated exclusively to
the Python programming language, and will be held at the Sydney Masonic
Center over the weekend of August 20 and 21.  See below for more
information and updates on:

1. Registration is now open
2. Classroom Track
3. Call For Proposals deadline approaching
4. Sponsors Announced

Please pass this message on to those you feel may be interested.



Registration Is Now Open


We offer three levels of registration for PyCon Australia 2011.
Registration provides access to two full days of technical content
presented by Python enthusiasts from around the country, as well as the
new classroom track and a seat at the conference dinner.

We are currently offering a limited number of early-bird tickets, but
get in quick because they are selling fast! 

  Corporate - $440
If your company is paying for you to attend PyCon, please register
at the corporate rate. You'll be helping to keep the conference
affordable for all.
  
  Full (Early Bird) - $165
This is the registration rate for regular attendees. We are offering
a limited Early Bird rate for the first 50 registrations until the
end of May.

  Student - $44
For students able to present a valid student card we're offering
this reduced rate, which does not include the conference dinner.

All prices include GST.  For more information or to register, please
visit the conference website.

Register here: http://pycon-au.org/reg



Classroom Track
===

In addition to the standard technical talks, this year's conference
will feature a "Classroom Track" designed specifically for tutorial
style presentations.

If you need to get up to speed on some of the latest language features
and tools, this will be a great opportunity to learn fast in a
supportive environment.



Call For Proposals
==

We've had some great initial responses to the Call For Proposals, but
there's still time left and plenty of program to fill.

Remember, the deadline for proposal submission is the 2nd of May.
That's just under two weeks away!

We are looking for proposals for talks on all aspects of Python
programming from novice to advanced levels; applications and frameworks,
or how you have been involved in introducing Python into your
organisation. We're especially interested in short presentations that
will teach conference-goers something new and useful. Can you show
attendees how to use a module? Explore a Python language feature?
Package an application?

We welcome first-time speakers; we are a community conference and we
are eager to hear about your experience. If you have friends or
colleagues who have something valuable to contribute, twist their arms
to tell us about it! Please also forward this Call for Proposals to
anyone that you feel may be interested.

The earlier you submit your proposal, the more time we will have to
review and give you feedback before the program is finalised.

Speakers receive free registration for the conference, including a seat
at the conference dinner.  Don't miss out, submit your proposal today! 

http://pycon-au.org/cfp




Sponsors Announced
==

We are happy to announce our first set of sponsors.  Thank you to the
following companies for their continuing support of Python and for
helping to make PyCon Australia 2011 a reality:


   Gold:  Google  
   Gold:  Microsoft   

   Silver:  Anchor
   Silver:  Enthought 
   Silver:  Python Software Foundation


Thanks also to Linux Australia, who provide the overarching legal and
organisational structure for PyCon Australia.



   Ryan Kelly
   PyCon Australia 2011





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Peter Otten]

Chris Angelico wrote:

> Context: Embedded Python interpreter, version 2.6.6
> 
> I have a list of dictionaries, where each dictionary has a "type"
> element which is a string. I want to reduce the list to just the
> dictionaries which have the same "type" as the first one.
> 
> lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]
> 
> I'm seeing a weird difference between two otherwise-equivalent-looking
> ways of doing the job.
> 
> type=lst[0]["type"].lower()
> 
> lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one
> type
> 
> lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one
> type
> 
> If I use the filter() method, the resulting list is completely empty.
> If I use the list comprehension, it works perfectly. Oddly, either
> version works in the stand-alone interpreter.
> 
> I have no idea where to start looking for the problem. Hints, please!
> 
> Chris Angelico

The assignment writes to the local namespace, the lambda function reads from 
the global namespace; this will only work as expected if the two namespaces 
are the same:

>>> exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}
[]
>>> ns = {}
>>> exec """type = 42; print filter(lambda x: x == type, [42])""" in ns
[42]

The list comprehension doesn't introduce another namespace, at least in 2.x:

$ python2.7 -c 'exec("type = 42; print([x for x in [42] if x == type])", {}, 
{})'
[42]
$ python3.2 -c 'exec("type = 42; print([x for x in [42] if x == type])", {}, 
{})'
[]

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Chris Angelico]

On Wed, Apr 20, 2011 at 8:16 PM, Steven D'Aprano
 wrote:
> There's your problem. IDEs often play silly buggers with the environment
> in order to be "clever". You've probably found a bug in whatever IDE
> you're using.
>
> And this is why I won't touch the buggers with a 30 ft pole, at least not
> for anything serious.

Not an IDE, but it's running in an embedded interpreter. Which is why
I looked there for issues, but I can't find any. (Checking sys.version
and friends shows that it's the same version of Python either way.) I
described the embed environment above, in the hopes that someone would
spot an "obvious error", and if your instant suspicion is an IDE, then
that's probably it.

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: My stupidity / strange inconsistency overriding class methods

[andrew cooke]

Thanks for finding that reference in the data model docs!  I was about to post 
a bug report because in PEP 3119 it says otherwise:

> The primary mechanism proposed here is to allow overloading the built-in 
> functions isinstance() and issubclass(). The overloading works as follows: 
> The call isinstance(x, C) first checks whether C.__instancecheck__ exists, 
> and if so, calls C.__instancecheck__(x) instead of its normal implementation. 

http://www.python.org/dev/peps/pep-3119/

But that's now what's implemented in Issue http://bugs.python.org/issue1708353 
which behaves as you quoted (only on the metaclass).

Cheers,
Andrew
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: My stupidity / strange inconsistency overriding class methods

[andrew cooke]

I didn't phrase that very well.  I do see the point about this being "an 
instance lookup on a class"...
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: meteclasses 2.x/3.x compatibility

[andrew cooke]

What I do in Lepl is use two stages.  The first calls the type/metaclass 
directly and the second subclasses that.  This avoids using the "sugar" that 
changes between 2 and 3.

So, for example, in 
http://code.google.com/p/lepl/source/browse/src/lepl/matchers/matcher.py#40 I 
have

  _Matcher = ABCMeta('_Matcher', (object, ), {})

and then

  class Matcher(_Matcher):
  ...

Andrew
-- 
http://mail.python.org/mailman/listinfo/python-list

py32 on windows - input() includes trailing \r

[Frank Millman]

Hi all

On linux, python 3.2 -


x = input()

xyz

len(x)

3

x

'xyz'

on windows, python 3.2 -


x = input()

xyz

len(x)

4

x

'xyz\r'

Is this expected behaviour?

Frank Millman


--
http://mail.python.org/mailman/listinfo/python-list

Re: IDLE bug

[rantingrick]

On Apr 18, 11:16 pm, James Mills  wrote:
> On Tue, Apr 19, 2011 at 2:05 PM, harrismh777  wrote:
> > Are bug reports wanted here, or just in issue tracker?
>
> Pretty sure they're wanted in the Issue Tracker.

My opinion is to report bugs on the tracker AND here since this list
is viewed by more people than the bug tracker is. And the wider
community needs to know where the shortcomings of this language exist.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python IDE/text-editor

[Teemu Likonen]

* 2011-04-18T21:17:17-07:00 * Westley Martínez wrote:

> On Tue, 2011-04-19 at 06:51 +0300, Teemu Likonen wrote:
>> * 2011-04-19T00:40:09+10:00 * Alec Taylor wrote:
>>> Please continue recommending
>> 
>> Vim.
>> 
>> * 2011-04-19T02:41:11+10:00 * Alec Taylor wrote:
>>> Please continue suggesting Python IDEs and/or fixes for the above
>>> Cons.
>> 
>> Vim.
>> 
>> * 2011-04-19T13:44:29+10:00 * Alec Taylor wrote:
>>> Please continue with your recommendations.
>> 
>> Vim.
>
> /trollface.jpg

You can recommend Vim, and I think that's an excellent option, but
please don't edit the contents of other people's text. Quotes are
quotes. This didn't cause any real harm but it's the principle.
-- 
http://mail.python.org/mailman/listinfo/python-list

dictionary size changed during iteration

[Laszlo Nagy]

Given this iterator:

class SomeIterableObject(object):



def __iter__(self):
ukeys = self.updates.keys()
for key in ukeys:
if self.updates.has_key(key):
yield self.updates[key]
for rec in self.inserts:
yield rec



How can I get this exception:

RuntimeError: dictionary changed size during iteration


It is true that self.updates is being changed during the iteration. But 
I have created the "ukeys" variable solely to prevent this kind of 
error. Here is a proof of correctness:



 d = {1:1,2:2}
 k = d.keys()
 del d[1]
 k

[1, 2]

 k is d.keys()

False

So what is wrong with this iterator? Why am I getting this error message?

Thanks,

   Laszlo

--
http://mail.python.org/mailman/listinfo/python-list

Re: py32 on windows - input() includes trailing \r

[Peter Otten]

Frank Millman wrote:

> On linux, python 3.2 -
> 
 x = input()
> xyz
 len(x)
> 3
 x
> 'xyz'
> 
> on windows, python 3.2 -
> 
 x = input()
> xyz
 len(x)
> 4
 x
> 'xyz\r'
> 
> Is this expected behaviour?

No, that's a bug:

http://bugs.python.org/issue11272 

IMO it's severe enough to warrant a brown-bag release...

-- 
http://mail.python.org/mailman/listinfo/python-list

Non hashable object (without __dict__)

[Arthur Mc Coy]

Hello,


I have a C++ application, I used SWIG to call the python code. I pass
myModule.myObject object to the method of python code and what to
store it in JSON format.

The problem is when I do:

o = myModule.myObject()


inside python, the o object has __dict__ property, but if I take the
passed object o like:

def myMethod(self, objects):
for o in objects:
print o.__dict__


it fails saying no SwigPyObject does not have __dict__ property. The
only way to use o properties is to call it by name like here:

print o.MyProperty


But in order to use json.dump( objects, file, default =
self.MyJSONEncode ) I MUST to pass to MyJSONEncode method an object
that has __dict__ as its property.



So, what is the best way to correct my objects, which are coming from C
++ code to make them available for putting in JSON format ?


Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: py32 on windows - input() includes trailing \r

[Frank Millman]

"Peter Otten" <__pete...@web.de> wrote in message 
news:iomla6$p8f$1...@dough.gmane.org...

Frank Millman wrote:


On linux, python 3.2 -


x = input()

xyz

len(x)

3

x

'xyz'

on windows, python 3.2 -


x = input()

xyz

len(x)

4

x

'xyz\r'

Is this expected behaviour?


No, that's a bug:

http://bugs.python.org/issue11272

IMO it's severe enough to warrant a brown-bag release...



Thanks, Peter.

I guess I should have checked the bug tracker first.

FWIW, I have found a workaround that seems to work on both platforms -


x = input().rstrip()

xyz

len(x)

3

x

'xyz'

Obviously this will also strip trailing spaces, but it will do for my 
purposes.


Frank


--
http://mail.python.org/mailman/listinfo/python-list

[OT] Disable creation of pyc files in DrPython

[craf]

Hi.

I wonder if anyone uses Python DrPython as editor.
I need to know if you can disable the creation of
Pyc files created by the program. In the Geany editor you can 
add the parameter -B, but not if it can in this editor.

Thanks in advance.

Regards

Cristian Abarzúa F

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary size changed during iteration

[Peter Otten]

Laszlo Nagy wrote:

> Given this iterator:
> 
> class SomeIterableObject(object):
>  
>  
> 
>  def __iter__(self):
>  ukeys = self.updates.keys()
>  for key in ukeys:
>  if self.updates.has_key(key):
>  yield self.updates[key]
>  for rec in self.inserts:
>  yield rec
>  
>  
> 
> How can I get this exception:
> 
> RuntimeError: dictionary changed size during iteration
> 
> 
> It is true that self.updates is being changed during the iteration. But
> I have created the "ukeys" variable solely to prevent this kind of
> error. Here is a proof of correctness:
> 
  d = {1:1,2:2}
  k = d.keys()
  del d[1]
  k
> [1, 2]
  k is d.keys()
> False
> 
> So what is wrong with this iterator? Why am I getting this error message?

The keys() method which used to return a list in 2.x was changed in 3.x to 
return a view object and to become more or less the equivalent of the old 
dict.iterkeys():

>>> d = dict(a=1)
>>> keys = d.keys()
>>> keys
dict_keys(['a'])
>>> for k in keys:
... d["b"] = 42
...
Traceback (most recent call last):
  File "", line 1, in 
RuntimeError: dictionary changed size during iteration
>>> keys
dict_keys(['a', 'b'])

You now have to create the list explicitly to avoid the error:

>>> d = dict(a=1)
>>> keys = list(d.keys())
>>> for k in keys:
... d["b"] = 42
...
>>> d
{'a': 1, 'b': 42}
>>> keys
['a']


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary size changed during iteration

[Peter Otten]

Peter Otten wrote:

> Laszlo Nagy wrote:
> 
>> Given this iterator:
>> 
>> class SomeIterableObject(object):
>>  
>>  
>> 
>>  def __iter__(self):
>>  ukeys = self.updates.keys()
>>  for key in ukeys:
>>  if self.updates.has_key(key):

Hm, I see you are using has_key() which is 2.x-only and invalidates my 
explanation :(

>>  yield self.updates[key]
>>  for rec in self.inserts:
>>  yield rec
>>  
>>  
>> 
>> How can I get this exception:
>> 
>> RuntimeError: dictionary changed size during iteration
>> 
>> 
>> It is true that self.updates is being changed during the iteration. But
>> I have created the "ukeys" variable solely to prevent this kind of
>> error. Here is a proof of correctness:
>> 
>  d = {1:1,2:2}
>  k = d.keys()
>  del d[1]
>  k
>> [1, 2]
>  k is d.keys()
>> False
>> 
>> So what is wrong with this iterator? Why am I getting this error message?
> 
> The keys() method which used to return a list in 2.x was changed in 3.x to
> return a view object and to become more or less the equivalent of the old
> dict.iterkeys():
> 
 d = dict(a=1)
 keys = d.keys()
 keys
> dict_keys(['a'])
 for k in keys:
> ... d["b"] = 42
> ...
> Traceback (most recent call last):
>   File "", line 1, in 
> RuntimeError: dictionary changed size during iteration
 keys
> dict_keys(['a', 'b'])
> 
> You now have to create the list explicitly to avoid the error:
> 
 d = dict(a=1)
 keys = list(d.keys())
 for k in keys:
> ... d["b"] = 42
> ...
 d
> {'a': 1, 'b': 42}
 keys
> ['a']


-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Vectors

[Andreas Tawn]

> Algis Kabaila  writes:
> 
> > Are there any modules for vector algebra (three dimensional
> > vectors, vector addition, subtraction, multiplication [scalar
> > and vector]. Could you give me a reference to such module?
> 
> NumPy has array (and matrix) types with support for these basic
> operations you mention. See the tutorial at http://numpy.scipy.org/

You might also want to consider http://code.google.com/p/pyeuclid/

Cheers,

Drea
-- 
http://mail.python.org/mailman/listinfo/python-list

renaming files in OS X

[jyoung79]

Hello,

I'm considering using os.rename or shutil for renaming 
files on OS X (Snow Leopard).  However, I've read that 
shutil doesn't copy the resource fork or metadata for 
the files on OS X.  I'm not sure about os.rename though.  
I need to keep the resource fork and metadata.  Is it 
better if I just use os.system('mv …') or is os.rename 
safe to use?

Thanks.

Jay
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Mel]

Chris Angelico wrote:
> On Wed, Apr 20, 2011 at 5:16 PM, Tim Roberts  wrote:
>> You can solve this through the common lamba idiom of a closure:
>>
>> lst=filter(lambda x,posttype=posttype: x["type"].lower()==posttype,lst)
> 
> Seems a little odd, but sure. I guess this means that a function's
> default arguments are evaluated in the parent context, but the body is
> evaluated in its own context?

The operation of calling a function has to evaluate arguments provided in 
the caller's namespace and assign them to variables in the called function's 
namespace.  Yes.

Mel.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary size changed during iteration

[Mel]

Laszlo Nagy wrote:

> Given this iterator:
> 
> class SomeIterableObject(object):
>  
>  
> 
>  def __iter__(self):
>  ukeys = self.updates.keys()
>  for key in ukeys:
>  if self.updates.has_key(key):
>  yield self.updates[key]
>  for rec in self.inserts:
>  yield rec
>  
>  
> 
> How can I get this exception:
> 
> RuntimeError: dictionary changed size during iteration
> 
> 
> It is true that self.updates is being changed during the iteration. But
> I have created the "ukeys" variable solely to prevent this kind of
> error. Here is a proof of correctness:
> 
  d = {1:1,2:2}
  k = d.keys()
  del d[1]
  k
> [1, 2]
  k is d.keys()
> False
> 
> So what is wrong with this iterator? Why am I getting this error message?

`ukeys` isn't a different dictionary from `self.updates.keys`  I'ts merely 
another name referring to the same dict object.  I think

ukeys = dict (self.updates.keys)

would do what you want.

Mel.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Dan Stromberg  wrote:
> On Tue, Apr 19, 2011 at 8:12 PM, Dan Stromberg  wrote:
>> I agree though that you're kind of pushing IP in a direction it wasn't
>> intended to go.
>
> It just occurred to me: You might get some additional mileage out of
> popping the network adapter into promiscuous mode.  In fact, it Might
> be necessary irrespective of the rest of your approach.

The network adapter is already receiving all the packets I want to
receive, so putting it into promiscuous mode would only increase the
number of unwanted packets.

-- 
Grant Edwards   grant.b.edwardsYow! Send your questions to
  at   ``ASK ZIPPY'', Box 40474,
  gmail.comSan Francisco, CA 94140,
   USA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary size changed during iteration

[Mel]

Mel wrote:
> Laszlo Nagy wrote:
> `ukeys` isn't a different dictionary from `self.updates.keys`  I'ts merely
> another name referring to the same dict object.  I think
> 
> ukeys = dict (self.updates.keys)
> 
> would do what you want.

Sorry.  Belay that.  Thought I'd had enough coffee.

Mel.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Roy Smith  wrote:
> In article ,
>  Grant Edwards  wrote:
>
>> I'm trying to implement a device discovery/configuration protocol that
>> uses UDP broadcast packets to discover specific types of devices on
>> the local Ethernet segment.  The management program broadcasts a
>> discovery command to a particular UDP port.  All devices who get that
>> packet are expected to answer regardless of thier current IP address.
>
> Have you considered what will happen if you have, say, 1000 such 
> devices, and they all respond at the same time?

Yes.  Firstly, there will very rarely be more than a handful of such
devices.  Secondly, I plan on inserting a small, psuedo-random delay
before replying.  Thirdly, the management program typically repeats
the discovery process a few times in case any packets get dropped.



-- 
Grant Edwards   grant.b.edwardsYow! Maybe we could paint
  at   GOLDIE HAWN a rich PRUSSIAN
  gmail.comBLUE --
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: No more Python support in NetBeans 7.0

[alister ware]

On Wed, 20 Apr 2011 03:24:00 -0700, Paul Rubin wrote:

> Markus  writes:
>> Infoworld awarded it as best Python IDE, testing: Boa Constructor,
>> Eric, ActiveState's Komodo, Oracle's NetBeans, Aptana's Pydev,
>> PyScripter, SPE, Spyder, and WingWare's Wing IDE.
> 
> I saw somebody using Geany recently and it looked pretty impressive. For
> Python gui debuggers, winpdb.org is great.

I use Geany all the time & it is a good little editor.
nice & lightweight for my linux netbook, but still with the features i 
need. I am sure a full IDE offers more but i never did get around to 
finding out what.



-- 
You fill a much-needed gap.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Heiko Wundram  wrote:
> Am 20.04.2011 01:54, schrieb Grant Edwards:
>> I guess the problem is that I expected to receive a packet on an
>> interface anytime a packet was received with a destination IP address
>> that matched that of the the interface.  Apprently there's some
>> filtering in the network stack based on the _source_ address as well
>> (that seems very counter-intuitive to me).
>
> Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
> there's a filtering done (at least under Linux, and I'd guess something
> similar on xBSD too) to packets based on the source address coming in on
> an interface, and it's called the reverse path filter and is on by
> default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).

Brilliant!  While I had determined that such filtering took place, I'd
been unable to figure out if it was configurable.

> The idea behind the reverse path filter is that your machine won't
> accept packets coming in over an interface when a return packet (i.e.,
> the presumed response) won't be routed over the same interface, and from
> what I gather, this is what makes the TCP/IP stack drop the packets
> because your machine will not route packets to 192.168.x.x over the same
> interface it sees the packet coming in. This is a _security_ feature,
> because it makes address spoofing harder.

And it's an eminently sensible feature.

> If you need to see the packets regardless, either use a promiscuous mode
> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
> using SOCK_RAW, capturing packets at the ethernet level), or add a route
> on your system for the 192.168.x.x network on the same interface.

I've thought about the SOCK_RAW option, but the CPU load of looking
all received Ethernet packets in user-space would be a big down-side.

Adding the route isn't an option since 1) the device doesn't know what
route to add, and 2) adding such a route could  break the normal
networking operation.

-- 
Grant Edwards   grant.b.edwardsYow! My haircut is totally
  at   traditional!
  gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Dan Stromberg  wrote:
> On Tue, Apr 19, 2011 at 6:15 PM, Grant Edwards  
> wrote:
>
>>> Or can you simply use a stupid netmask like /1 that picks up all the
>>> IP ranges? That way, the source-IP check wouldn't fail.
>>
>> That would require that the device somehow knows that it's not
>> configured correctly and should change the netmask to /1. ?The device
>> doesn't have any way to know that, and it must respond to the
>> discovery commands both before and after it's properly configured.
>
> - Actually, you Might be able to configure your device to have a
> netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
> 255.255.255.255.
> - I've seen something a bit similar used for detecting IP address
> conflicts automatically.
> - A network guru I used to work with told me that you could configure
> a machine with a broadcast of 255.255.255.255 more simply than messing
> around with the netmask, while still achieving the same result for
> general purpose networking.

I'll look into that.

>> I've reread the protocol documentation and noticed that the device has
>> to respond not only to broadcasts to 255.255.255.255 but also to
>> subnet broadcasts send to subnets it's not on. ?That pretty much
>> clinches the requirement to use a raw socket. :/
>
> With a netmask of 0.0.0.0, I suspect you will receive all broadcasts
> on the wire, given appropriate listening code.

That might be an option as well, as long as it doesn't disrupt normal
operation of the interface.

-- 
Grant Edwards   grant.b.edwardsYow! Remember, in 2039,
  at   MOUSSE & PASTA will
  gmail.combe available ONLY by
   prescription!!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Thomas Heller  wrote:
> Am 20.04.2011 00:21, schrieb Grant Edwards:
>> I'm have problems figuring out how to receive UDP broadcast packets on
>> Linux.
> [...]
>>
>> On the receiving machine, I've used tcpdump to verify that broadcast
>> packets are being seen and have a destination IP of 255.255.255.255 and
>> destination MAC of ff:ff:ff:ff:ff:ff
> [...]
>> But, the receiving Python program never sees any packets unless the
>> _source_ IP address in the packets is on the same subnet as the
>> receiving machine.  In this test case, the receiving machine has an IP
>> address of 172.16.12.34/16.  If I change the receiving machine's IP
>> address to 10.0.0.123, then the receiving program sees the packets.
>>
>> Even though the destination address is 255.255.255.255, the receiving
>> machine appears to discard the packets based on the _source_ IP.  Can
>> anybody provide example Python code for Linux that receives UDP
>> broadcast packets regardless of their source IP address?
>>
>> This probably is more of a Linux networking question than a Python
>> question, but I'm hoping somebody has solved this problem in Python.
>>
>
> You must set the network interface to promiscous mode on the
> receiving side:
>
> os.system("ifconfig eth0 promisc")

Why?

The network interface is already receiving the packets I want, since
they're beign sent with a destination MAC of ff:ff:ff:ff:ff:ff.

-- 
Grant Edwards   grant.b.edwardsYow! Spreading peanut
  at   butter reminds me of
  gmail.comopera!!  I wonder why?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Ian Kelly]

On Wed, Apr 20, 2011 at 4:41 AM, Peter Otten <__pete...@web.de> wrote:
> The assignment writes to the local namespace, the lambda function reads from
> the global namespace; this will only work as expected if the two namespaces
> are the same:
>
 exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}
> []
 ns = {}
 exec """type = 42; print filter(lambda x: x == type, [42])""" in ns
> [42]

That must be a quirk of exec, because it works just fine without using
exec, both in and out of functions, either from the interactive
interpreter or from a script:

Python 2.7.1 (r271:86832, Apr  2 2011, 19:44:19)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
... t = 42
... print filter(lambda x: x == t, [42])
...
>>> f()
[42]
>>> t = 43
>>> print filter(lambda x: x == t, [43])
[43]

So, the question for the OP:  Is this file being run with execfile?

Cheers,
Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Heiko Wundram]

Am 20.04.2011 16:30, schrieb Grant Edwards:
>> If you need to see the packets regardless, either use a promiscuous mode
>> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
>> using SOCK_RAW, capturing packets at the ethernet level), or add a route
>> on your system for the 192.168.x.x network on the same interface.
> 
> I've thought about the SOCK_RAW option, but the CPU load of looking
> all received Ethernet packets in user-space would be a big down-side.

Not necessarily: instead of using UDP datagrams to send the data, use
ethernet datagrams (without any IP/UDP header) with your own
ethernet-type (there is a range of "local" types that you can use for
your own local use-case), and then simply create a RAW socket that only
listens on packets that have the specified ethernet types. We use
something similar at work for a high-availability application.

The server-side looks something like:

"""
PKT_TYPE = 0x1234 # My very own ethertype.

sock = socket(AF_PACKET,SOCK_DGRAM,htons(PKT_TYPE))
sock.bind(("ethxyz",PKT_TYPE))

while True:
data, (_, _, _, _, addr) = sock.recvfrom(1500)
print "I got:", repr(data), "from etheraddr:", addr
"""

The client-side looks similar.

Because you're using UDP broacast, you have unreliable transport anyway,
and if the client-side supports sending ethernet datagrams (with a
broadcast address), I'd rather advise to use that for your use case.
This makes you independent of IP configuration (and as I can see, you're
actually not interested in the "routing" that IP gives you, but rather
interested in contacting all nodes on a local ethernet; why not use
ethernet directly?).

-- 
--- Heiko.
-- 
http://mail.python.org/mailman/listinfo/python-list

SLOW AND FAST FUNCTIONS

[Martin]

  //\ PROJF
  //P\ SLOW VER
  //  GDRAW PROJF DEMO P
  //
  //  P //  XEQ GDRAW
  //
  //P   \   PROJF
  //
  //  \  FAST VER
  //  @ domain [http://meami.org/fastslow.htm]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: renaming files in OS X

[Gnarlodious]

Not a Python question. You should go over to
http://groups.google.com/group/comp.sys.mac.system/ and ask.

-- Gnarlie

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: renaming files in OS X

[jyoung79]

> Not a Python question. You should go over to
> http://groups.google.com/group/comp.sys.mac.system/ and ask.

> -- Gnarlie

What do you mean it's not a python question?  os.rename is 
python syntax…  how does it work on OS X?  Is it the same as 
the 'mv' command, etc?

Jay
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Heiko Wundram  wrote:
> Am 20.04.2011 16:30, schrieb Grant Edwards:
>>> If you need to see the packets regardless, either use a promiscuous mode
>>> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
>>> using SOCK_RAW, capturing packets at the ethernet level), or add a route
>>> on your system for the 192.168.x.x network on the same interface.
>> 
>> I've thought about the SOCK_RAW option, but the CPU load of looking
>> all received Ethernet packets in user-space would be a big down-side.
>
> Not necessarily: instead of using UDP datagrams to send the data,

There's the rub.  The requirement is to use UDP.

> use ethernet datagrams (without any IP/UDP header) with your own
> ethernet-type (there is a range of "local" types that you can use for
> your own local use-case), and then simply create a RAW socket that
> only listens on packets that have the specified ethernet types. We
> use something similar at work for a high-availability application.

Same here.  What I'm working on is a replacement for just such a
protocol.  The problem is that this whole scheme has to work easily
with the management program running on Windows. While doing a custom
Ethernet protocol on Linux is dead-simple (it does require either
UID==0 or CAP_NET_RAW), doing it on Windows is painful in the extreme.

Thus the requirement to use something that works using "normal UDP" on
the manager end of things (even if it requires jumping through some
hoops on the device end).  

[Insert standard whinge about how once again, Microsoft royally
botches something and we non-Windows people have to suffer.]

-- 
Grant Edwards   grant.b.edwardsYow! ... I want to perform
  at   cranial activities with
  gmail.comTuesday Weld!!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Grant Edwards  wrote:
> On 2011-04-20, Heiko Wundram  wrote:

>>> I've thought about the SOCK_RAW option, but the CPU load of looking
>>> all received Ethernet packets in user-space would be a big down-side.
>>
>> Not necessarily: instead of using UDP datagrams to send the data,
>> use ethernet datagrams (without any IP/UDP header) with your own
>> ethernet-type (there is a range of "local" types that you can use for
>> your own local use-case), and then simply create a RAW socket that
>> only listens on packets that have the specified ethernet types. We
>> use something similar at work for a high-availability application.
>
> Same here. What I'm working on is a replacement for just such a
> protocol.

Now that I think of it, I added raw socket support to Python just so I
could implement that protocol in Python!

[A rather vain attempt to inject some Python content back into the
thread.]

-- 
Grant Edwards   grant.b.edwardsYow! Awright, which one of
  at   you hid my PENIS ENVY?
  gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Vectors

[RJB]

On Apr 20, 6:43 am, Andreas Tawn  wrote:
> > Algis Kabaila  writes:
>
> > > Are there any modules for vector algebra (three dimensional
> > > vectors, vector addition, subtraction, multiplication [scalar
> > > and vector]. Could you give me a reference to such module?
>
> > NumPy has array (and matrix) types with support for these basic
> > operations you mention. See the tutorial athttp://numpy.scipy.org/
>
> You might also want to considerhttp://code.google.com/p/pyeuclid/
>
> Cheers,
>
> Drea

Pyeuclid docs don't mention dot or cross products.
RJB

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Adam Tauno Williams  wrote:
> On Wed, 2011-04-20 at 06:07 -0400, Sherm Pendley wrote:
>> Grant Edwards  writes:
>> > I'm trying to implement a device discovery/configuration protocol that
>> > uses UDP broadcast packets to discover specific types of devices on
>> > the local Ethernet segment.  The management program broadcasts a
>> > discovery command to a particular UDP port.  All devices who get that
>> > packet are expected to answer regardless of thier current IP address.
>> Have you looked at the source for Apple's Bonjour?
>
> On LINUX this is called "avahi", which has Python bindings.  Avahi
> auto-configuration / discovery works very well.
>
>
>
>
>
>
> See also:
>
> at least you may be able to lift code from them (License is non-viral
> MIT)

I've looked at those protocols and they aren't particularly suited to
what we want to do.  One of our requirements is to be able to discover
a device with a static IP address (or no IP address?) that isn't on
the same subnet -- and I don't see how those protocols can do that. 

Since the management program already has to implement the UDP based
protocol I'm working on, it would be a big win if I didn't have to
make the management program add support for a second protocol.

-- 
Grant Edwards   grant.b.edwardsYow! Are we THERE yet?
  at   My MIND is a SUBMARINE!!
  gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Dan Stromberg  wrote:

> - Actually, you Might be able to configure your device to have a
>   netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
>   255.255.255.255.

255.255.255.255 isn't allowed as an IP address.

I tried a netmask of 0.0.0.0, and it didn't make any differnce.

-- 
Grant Edwards   grant.b.edwardsYow! My EARS are GONE!!
  at   
  gmail.com
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Vectors

[Andreas Tawn]

> On Apr 20, 6:43 am, Andreas Tawn  wrote:
> > > Algis Kabaila  writes:
> >
> > > > Are there any modules for vector algebra (three dimensional
> > > > vectors, vector addition, subtraction, multiplication [scalar
> > > > and vector]. Could you give me a reference to such module?
> >
> > > NumPy has array (and matrix) types with support for these basic
> > > operations you mention. See the tutorial athttp://numpy.scipy.org/
> >
> > You might also want to considerhttp://code.google.com/p/pyeuclid/
> >
> > Cheers,
> >
> > Drea
> 
> Pyeuclid docs don't mention dot or cross products.
> RJB

http://partiallydisassembled.net/euclid/vector-classes.html#SECTION00222

Bottom of the page.

Cheers,

Drea

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-20, Grant Edwards  wrote:
> On 2011-04-20, Heiko Wundram  wrote:
>> Am 20.04.2011 01:54, schrieb Grant Edwards:
>>> I guess the problem is that I expected to receive a packet on an
>>> interface anytime a packet was received with a destination IP address
>>> that matched that of the the interface.  Apprently there's some
>>> filtering in the network stack based on the _source_ address as well
>>> (that seems very counter-intuitive to me).
>>
>> Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
>> there's a filtering done (at least under Linux, and I'd guess something
>> similar on xBSD too) to packets based on the source address coming in on
>> an interface, and it's called the reverse path filter and is on by
>> default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).
>
> Brilliant!  While I had determined that such filtering took place, I'd
> been unable to figure out if it was configurable.

Bingo!

Turning off reverse-path filtering solved my original problem (not
receiving broadcasts sent to 255.255.255.255 because the source IP
wasn't reachable via the receiving interface).

It doesn't help with the "new" requirement of receiving subnet
broadcasts that don't match the interface on which they're received,
since it's destination IP filtering that's dropping those.

Apparently that requirement is due to the fact that on some OSes (some
flavors of BSD and Windows) applications can't broadcast to
255.255.255.255, they can only do subnet broadcasts.

-- 
Grant Edwards   grant.b.edwardsYow! I wish I was a
  at   sex-starved manicurist
  gmail.comfound dead in the Bronx!!
-- 
http://mail.python.org/mailman/listinfo/python-list

Why doesn't this asyncore.dispatcher.handle_read() get called?

[Dun Peal]

Hi,

I'm writing and testing an asyncore-based server. Unfortunately, it
doesn't seem to work. The code below is based on the official docs and
examples, and starts a listening and sending dispatcher, where the
sending dispatcher connects and sends a message to the listener - yet
Handler.handle_read() never gets called, and I'm not sure why. Any
ideas?

Thanks, D.


import asyncore, socket, sys

COMM_PORT = 9345

class Handler(asyncore.dispatcher):
def handle_read(self):
print 'This never prints'

class Listener(asyncore.dispatcher):
def __init__(self, port=COMM_PORT):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.set_reuse_addr()
self.bind(('', port))
self.listen(5)

def handle_accept(self):
client, addr = self.accept()
print 'This prints.'
return Handler(client)

class Sender(asyncore.dispatcher):
def __init__(self, host):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.connect( (host, COMM_PORT) )
self.buffer = 'Msg\r\n'

def handle_connect(self):
pass

def writable(self):
return len(self.buffer) > 0

def handle_write(self):
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]

def test_communication():
from multiprocessing import Process
def listener():
l = Listener()
asyncore.loop(timeout=10, count=1)
lis = Process(target=listener)
lis.start()
def sender():
s = Sender('localhost')
asyncore.loop(timeout=10, count=1)
sen = Process(target=sender)
sen.start()
lis.join()

test_communication()
-- 
http://mail.python.org/mailman/listinfo/python-list

mystery string code - help!

[Uncle Ben]

I found this in one of the online cookbooks:

#Raghunath Reddy Peesari 6 years, 3 months ago  # | flag
#There is more simple way. ###

a = 'abcdefghi'
a = a[::-1]

print a

>>> 'ihgfedcba'

As a newbie Pythoner, I understand [] -1]
but would some tell me how '::' does its magic?

Uncle Ben
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mystery string code - help!

[Dan M]

> As a newbie Pythoner, I understand [] -1] but would some tell me how
> '::' does its magic?
> 
> Uncle Ben

The -1 is the "stride" or "step" argument. It's described at 
http://docs.python.org/release/2.3.5/whatsnew/section-slices.html

Dan
-- 
http://mail.python.org/mailman/listinfo/python-list

learnpython.org - an online interactive Python tutorial

[Ron]

Hey everyone.

I've written an online interactive Python tutorial atop Google App Engine: 
http://www.learnpython.org.

All you need to do is log in using your Google account and edit the wiki to add 
your tutorials.

Read more on the website.

Thanks for your help, and I would appreciate if you help me spread the word, 
and give me feedback on the website.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: renaming files in OS X

[Philip Semanchuk]

On Apr 20, 2011, at 10:02 AM,   wrote:

> Hello,
> 
> I'm considering using os.rename or shutil for renaming 
> files on OS X (Snow Leopard).  However, I've read that 
> shutil doesn't copy the resource fork or metadata for 
> the files on OS X.  I'm not sure about os.rename though.  
> I need to keep the resource fork and metadata.  Is it 
> better if I just use os.system('mv …') or is os.rename 
> safe to use?

Hi Jay,
I don't know if os.rename() does what you want, but why don't you try a simple 
test and find out? Surely an empirical test is at least as useful as an answer 
from someone like me who may or may not know what he's talking about. =)

The OS X command xattr  shows whether or not a file has extended attributes, 
which are what I think you're referring to when you say "metadata". xattr is 
written (badly) in Python; on my system it lives in /usr/bin/xattr-2.6

You might also find this helpful:
http://jonsview.com/mac-os-x-resource-forks


Hope this helps
Philip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: learnpython.org - an online interactive Python tutorial

[Matty Sarro]

Awesome project, I really like it. I'll see if I can't help adding
some material that's missing when I get on the train.
Keep up the great work!

On Wed, Apr 20, 2011 at 1:15 PM, Ron  wrote:
> Hey everyone.
>
> I've written an online interactive Python tutorial atop Google App Engine: 
> http://www.learnpython.org.
>
> All you need to do is log in using your Google account and edit the wiki to 
> add your tutorials.
>
> Read more on the website.
>
> Thanks for your help, and I would appreciate if you help me spread the word, 
> and give me feedback on the website.
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [OT] Disable creation of pyc files in DrPython

[Alexander Kapps]

On 20.04.2011 15:21, craf wrote:

Hi.

I wonder if anyone uses Python DrPython as editor.
I need to know if you can disable the creation of
Pyc files created by the program. In the Geany editor you can
add the parameter -B, but not if it can in this editor.


I don't know DrPython, but Python itself checks for the 
$PYTHONDONTWRITEBYTECODE environment variable. Perhaps you can run 
DrPython with a command like:


PYTHONDONTWRITEBYTECODE=1 drpython


HTH
--
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Chris Angelico]

On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly  wrote:
> So, the question for the OP:  Is this file being run with execfile?
>

Not execfile per se; the code is fetched from the database and then
executed with:

PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

Is Py_file_input the problem?

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: learnpython.org - an online interactive Python tutorial

[Ron]

Thanks! :)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: learnpython.org - an online interactive Python tutorial

[FELD Boris]

Excellent idea,

I've some ideas on specific subjects misunderstood by beginners.

One idea for facilitating the contribution, create a mercurial repository (or a 
git), everyone has not a google account and your contributors will be 
developers so they should use a SCM.

Once again, it's an excellent idea and, when the tutorials were expanded 
enough, i should be good to integrate it with python.org website (but it's more 
a dream than something else...)

One question, if i want to write a tutorial about import mechanism, how can i 
manage the file system ?

Nice work !
-- 
FELD Boris
Sent with Sparrow
On mercredi 20 avril 2011 at 19:57, Matty Sarro wrote: 
> Awesome project, I really like it. I'll see if I can't help adding
> some material that's missing when I get on the train.
> Keep up the great work!
> 
> On Wed, Apr 20, 2011 at 1:15 PM, Ron  wrote:
> > Hey everyone.
> > 
> > I've written an online interactive Python tutorial atop Google App Engine: 
> > http://www.learnpython.org.
> > 
> > All you need to do is log in using your Google account and edit the wiki to 
> > add your tutorials.
> > 
> > Read more on the website.
> > 
> > Thanks for your help, and I would appreciate if you help me spread the 
> > word, and give me feedback on the website.
> > --
> > http://mail.python.org/mailman/listinfo/python-list
> -- 
> http://mail.python.org/mailman/listinfo/python-list
> 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [OT] Disable creation of pyc files in DrPython

[craf]

On 20.04.2011 15:21, craf wrote:
> Hi.
>
> I wonder if anyone uses Python DrPython as editor.
> I need to know if you can disable the creation of
> Pyc files created by the program. In the Geany editor you can
> add the parameter -B, but not if it can in this editor.

>I don't know DrPython, but Python itself checks for the 
>$PYTHONDONTWRITEBYTECODE environment variable. Perhaps you can run 
>DrPython with a command like:

>PYTHONDONTWRITEBYTECODE=1 drpython


>HTH

Hi. Alexander.

Thanks for the information.


I'll prove it

Regards.

Cristian Abarzúa F

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: List comprehension vs filter()

[Ian Kelly]

On Wed, Apr 20, 2011 at 12:03 PM, Chris Angelico  wrote:
> On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly  wrote:
>> So, the question for the OP:  Is this file being run with execfile?
>>
>
> Not execfile per se; the code is fetched from the database and then
> executed with:
>
> PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);
>
> Is Py_file_input the problem?

Not specifically.  The problem is that you're execing the code, and
the locals and globals are two different namespaces, as Peter
suggested.  Since the locals namespace is not associated with a
function, the compiler evidently doesn't generate closures for it, and
thus the lambda (which has its own locals namespace) can't see it.

That's my take, anyway.  Somebody else may have better insight than me.

Cheers,
Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Tutor] NLP

[James Thornton]

http://scikit-learn.sourceforge.net/

On Fri, Apr 8, 2011 at 6:52 AM, Ranjith Kumar  wrote:
> Hi all,
>     Can anyone suggest me any best Natural Language Processing in
> python other than nltk.
> --
> Cheers,
> Ranjith Kumar K,
> Chennai.
> http://ranjithtenz.wordpress.com
>
>
>
>
> ___
> Tutor maillist  -  tu...@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>



-- 
Latest Blog: http://jamesthornton.com/blog/how-to-get-to-genius
-- 
http://mail.python.org/mailman/listinfo/python-list

TestFixtures 1.9.2 Released!

[Chris Withers]

Hi All,

I'm happy to announce a new release of TestFixtures.

This release adds a utcnow method to test_datetime that behaves 
identically to the now method:


http://packages.python.org/testfixtures/api.html#testfixtures.tdatetime.utcnow

The package is on PyPI and a full list of all the links to docs, issue 
trackers and the like can be found here:


http://www.simplistix.co.uk/software/python/testfixtures

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
   - http://www.simplistix.co.uk
--
http://mail.python.org/mailman/listinfo/python-list

RE: renaming files in OS X

[jyoung79]

>> I'm considering using os.rename or shutil for renaming 
>> files on OS X (Snow Leopard)...

> Hi Jay,
> I don't know if os.rename() does what you want, but why 
> don't you try a simple test and find out? Surely an 
> empirical test is at least as useful as an answer from 
> someone like me who may or may not know what he's 
> talking about. =)

> The OS X command xattr  shows whether or not a file has 
> extended attributes, which are what I think you're 
> referring to when you say "metadata". xattr is written 
> (badly) in Python; on my system it lives in 
> /usr/bin/xattr-2.6

> You might also find this helpful:
> http://jonsview.com/mac-os-x-resource-forks


Hi Philip,

Thank you very much for your reply.  This actually does 
help a lot!  I had done a bit of testing earlier, but
I admit I'm quite ignorant on Apples file structure.
Your excellent link had some great info and I did a
few tests with what it had to say before and after I
ran os.rename on a file.  I also ran some applescript
code to get the file/creator type before and after
os.name (although I don't believe these types are
actually part of the resource fork, they're still used
by some scripts we run).  So far, after running
os.name, the file still contains everything I can think
of so far.

Again, thank you for your help with this!

Jay
-- 
http://mail.python.org/mailman/listinfo/python-list

Groups in regular expressions don't repeat as expected

[John Nagle]

Here's something that surprised me about Python regular expressions.

>>> krex = re.compile(r"^([a-z])+$")
>>> s = "abcdef"
>>> ms = krex.match(s)
>>> ms.groups()
('f',)

The parentheses indicate a capturing group within the
regular expression, and the "+" indicates that the
group can appear one or more times.  The regular
expression matches that way.  But instead of returning
a captured group for each character, it returns only the
last one.

The documentation in fact says that, at

http://docs.python.org/library/re.html

"If a group is contained in a part of the pattern that matched multiple 
times, the last match is returned."


That's kind of lame, though. I'd expect that there would be some way
to retrieve all matches.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: Groups in regular expressions don't repeat as expected

[Neil Cerutti]

On 2011-04-20, John Nagle  wrote:
> Here's something that surprised me about Python regular expressions.
>
> >>> krex = re.compile(r"^([a-z])+$")
> >>> s = "abcdef"
> >>> ms = krex.match(s)
> >>> ms.groups()
> ('f',)
>
> The parentheses indicate a capturing group within the
> regular expression, and the "+" indicates that the
> group can appear one or more times.  The regular
> expression matches that way.  But instead of returning
> a captured group for each character, it returns only the
> last one.
>
> The documentation in fact says that, at
>
> http://docs.python.org/library/re.html
>
> "If a group is contained in a part of the pattern that matched multiple 
> times, the last match is returned."
>
> That's kind of lame, though. I'd expect that there would be some way
> to retrieve all matches.

.findall

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: renaming files in OS X

[Ned Deily]

In article <280cb56a-89b8-4d62-9374-d769b3acf...@semanchuk.com>,
 Philip Semanchuk  wrote:
> On Apr 20, 2011, at 10:02 AM,   
> wrote:
> > I'm considering using os.rename or shutil for renaming 
> > files on OS X (Snow Leopard).  However, I've read that 
> > shutil doesn't copy the resource fork or metadata for 
> > the files on OS X.  I'm not sure about os.rename though.  
> > I need to keep the resource fork and metadata.  Is it 
> > better if I just use os.system('mv Š') or is os.rename 
> > safe to use?

os.rename() is a simple wrapper around the standard rename system call 
(man 2 rename) so it has the same semantics.  Extended attributes, 
including resource forks, are preserved by rename(2).  Note that the 
system call only works for renames within one file system.  The mv(1) 
program handles cross-system renames by copying and unlinking and the 
Apple-supplied version does copy extended attribute metadata in that 
case.  As documented, none of the shutil copy functions do that.

> I don't know if os.rename() does what you want, but why don't you try a 
> simple test and find out? Surely an empirical test is at least as useful as 
> an answer from someone like me who may or may not know what he's talking 
> about. =)
> 
> The OS X command xattr  shows whether or not a file has extended attributes, 

The 'ls -l' command does as well:

$ ls -l a.jpg
-rw-r--r--@  1 nad  staff  2425268 Apr  4 16:30 a.jpg
$ ls -l@ a.jpg
-rw-r--r--@  1 nad  staff  2425268 Apr  4 16:30 a.jpg
   com.apple.FinderInfo  32

-- 
 Ned Deily,
 n...@acm.org

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Questions about GIL and web services from a n00b

[Chris H]

On 4/19/11 3:48 AM, Lamont Nelson wrote:

  >  1. Are you sure you want to use python because threading is not
good due

to the Global Lock (GIL)?  Is this really an issue for multi-threaded
web services as seems to be indicated by the articles from a Google
search?  If not, how do you avoid this issue in a multi-threaded process
to take advantage of all the CPU cores available?

To take advantage of the cores on your server you'll want to consider
a multi-process design instead of multi-threading. You can achieve
this with something like http://projects.unbit.it/uwsgi/, which will
allow you to manage the processes. I've used this behind apache
successfully.


2. Are there good web services frameworks available for building a REST
based service?  I admit I have looked at web2py, Django, pyramid/pylons,
and a few others.  SOAP seems to be pretty well supported but I'm not
finding the same for quick development of REST based services for
exchanging JSON or XML formatted data.  This is probably just my n00b
status, but what tools are best for building a simple REST data exchange
API?

I've personally used Pyramid to implement REST web services with
multiple data transport formats (json, xml) for the same endpoints
with minimal fuss. It's basically as simple as returning an object
from your view and defining a renderer that knows how to translate
this object to the desired format. Look at
http://docs.pylonsproject.org/projects/pyramid/1.0/narr/renderers.html#views-which-use-a-renderer
and 
http://docs.pylonsproject.org/projects/pyramid/1.0/narr/viewconfig.html#view-config-chapter
for more information.

Lamont


Thanks to everyone who has replied.  I have a better understanding of 
the limitations around the GIL as well as the relatively architectures 
for avoiding the issue.  Now to find the right framework for a simple 
web service.  So far I've heard pyramid the most, but others we are 
looking into include rest.ish.io, web2py, and flask.  Anyone with 
experience across these as to what is best for someone starting from 
scratch now?


Chris

--
http://mail.python.org/mailman/listinfo/python-list

Re: Groups in regular expressions don't repeat as expected

[MRAB]

On 20/04/2011 20:20, John Nagle wrote:

Here's something that surprised me about Python regular expressions.

 >>> krex = re.compile(r"^([a-z])+$")
 >>> s = "abcdef"
 >>> ms = krex.match(s)
 >>> ms.groups()
('f',)

The parentheses indicate a capturing group within the
regular expression, and the "+" indicates that the
group can appear one or more times. The regular
expression matches that way. But instead of returning
a captured group for each character, it returns only the
last one.

The documentation in fact says that, at

http://docs.python.org/library/re.html

"If a group is contained in a part of the pattern that matched multiple
times, the last match is returned."

That's kind of lame, though. I'd expect that there would be some way
to retrieve all matches.


You should take a look at the regex module on PyPI. :-)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't this asyncore.dispatcher.handle_read() get called?

[Jean-Paul Calderone]

On Apr 20, 12:25 pm, Dun Peal  wrote:
> Hi,
>
> I'm writing and testing an asyncore-based server. Unfortunately, it
> doesn't seem to work. The code below is based on the official docs and
> examples, and starts a listening and sending dispatcher, where the
> sending dispatcher connects and sends a message to the listener - yet
> Handler.handle_read() never gets called, and I'm not sure why. Any
> ideas?
>
> Thanks, D.
>
> import asyncore, socket, sys
>
> COMM_PORT = 9345
>
> class Handler(asyncore.dispatcher):
>     def handle_read(self):
>         print 'This never prints'
>
> class Listener(asyncore.dispatcher):
>     def __init__(self, port=COMM_PORT):
>         asyncore.dispatcher.__init__(self)
>         self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
>         self.set_reuse_addr()
>         self.bind(('', port))
>         self.listen(5)
>
>     def handle_accept(self):
>         client, addr = self.accept()
>         print 'This prints.'
>         return Handler(client)
>
> class Sender(asyncore.dispatcher):
>     def __init__(self, host):
>         asyncore.dispatcher.__init__(self)
>         self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
>         self.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>         self.connect( (host, COMM_PORT) )
>         self.buffer = 'Msg\r\n'
>
>     def handle_connect(self):
>         pass
>
>     def writable(self):
>         return len(self.buffer) > 0
>
>     def handle_write(self):
>         sent = self.send(self.buffer)
>         self.buffer = self.buffer[sent:]
>
> def test_communication():
>     from multiprocessing import Process
>     def listener():
>         l = Listener()
>         asyncore.loop(timeout=10, count=1)
>     lis = Process(target=listener)
>     lis.start()
>     def sender():
>         s = Sender('localhost')
>         asyncore.loop(timeout=10, count=1)
>     sen = Process(target=sender)
>     sen.start()
>     lis.join()
>
> test_communication()

You didn't let the program run long enough for the later events to
happen.  loop(count=1) basically means one I/O event will be processed
- in the case of your example, that's an accept().  Then asyncore is
done and it never gets to your custom handle_read.

So you can try passing a higher count to loop, or you can add your own
loop around the loop call.  Or you can switch to Twisted which
actually makes testing a lot easier than this - no need to spawn
multiple processes or call accept or recv yourself.  Here's a somewhat
equivalent Twisted-based version of your program:

from twisted.internet.protocol import ServerFactory, Protocol
from twisted.internet import reactor

factory = ServerFactory()
factory.protocol = Protocol

reactor.listenTCP(0, factory)
reactor.run()

It's hard to write the equivalent unit test, because the test you
wrote for the asyncore-based version is testing lots of low level
details which, as you can see, don't actually appear in the Twisted-
based version because Twisted does them for you already.  However,
once you get past all that low-level stuff and get to the part where
you actually implement some of your application logic, you might have
tests for your protocol implementation that look something like this:

from twisted.trial.unittest import TestCase
from twisted.test.proto_helpers import StringTransport

from yourapp import Handler # Or a better name

class HandlerTests(TestCase):
def test_someMessage(self):
"""
When the "X" message is received, the "Y" response is sent
back.
"""
transport = StringTransport()
protocol = Handler()
protocol.makeConnection(transport)
protocol.dataReceived("X")
self.assertEqual(transport.value(), "Y")

Hope this helps,
Jean-Paul

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: renaming files in OS X

[jyoung79]

> In article <280CB56A-89B8-4D62-9374-D769B3ACFEBB at semanchuk.com>,
>  Philip Semanchuk  wrote:
> > On Apr 20, 2011, at 10:02 AM,   > kc.rr.com> 
> > wrote:
> > > I'm considering using os.rename or shutil for renaming 
> > > files on OS X (Snow Leopard)…

> os.rename() is a simple wrapper around the standard rename system call 
> (man 2 rename) so it has the same semantics.  Extended attributes, 
> including resource forks, are preserved by rename(2).  Note that the 
> system call only works for renames within one file system.  The mv(1) 
> program handles cross-system renames by copying and unlinking and the 
> Apple-supplied version does copy extended attribute metadata in that 
> case.  As documented, none of the shutil copy functions do that.

> > The OS X command xattr  shows whether or not a file has extended 
> > attributes, 

> The 'ls -l' command does as well:

> $ ls -l a.jpg
> -rw-r--r--@  1 nad  staff  2425268 Apr  4 16:30 a.jpg
> $ ls -l@ a.jpg
> -rw-r--r--@  1 nad  staff  2425268 Apr  4 16:30 a.jpg
>com.apple.FinderInfo  32


Hi Ned,

Thanks so much for this detail!  I didn't even realize there was a standard
rename system call - it's always nice learning something new about this
system.

Again, thank you for taking the time to share your knowledge.  This is
exactly what I was looking for!

Jay
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Language & lib reference in man format ?

[andrew cooke]

(1) Python's docs use Sphinx, which uses restructured text as a markup.  You 
can generate man pages from restructured text using rst2man (which is installed 
on my computer, probably as part of python/docutils).

HOWEVER I imagine it's not going to work very well, if at all, because Sphinx 
uses lots of fancy extensions.  But...

(2) Sphinx itself has a "builder" for man pages.  It's described at 
http://sphinx.pocoo.org/builders.html#sphinx.builders.manpage.ManualPageBuilder 
 So you should be able to configure Sphinx to use that.  So you will need to 
download the docs package, install Sphinx, and then tweak the Sphinx 
configuration as described at the link above and at 
http://sphinx.pocoo.org/config.html#confval-man_pages

This second approach should work.

Andrew
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Groups in regular expressions don't repeat as expected

[John Nagle]

On 4/20/2011 12:23 PM, Neil Cerutti wrote:

On 2011-04-20, John Nagle  wrote:

Here's something that surprised me about Python regular expressions.


krex = re.compile(r"^([a-z])+$")
s = "abcdef"
ms = krex.match(s)
ms.groups()

('f',)

The parentheses indicate a capturing group within the
regular expression, and the "+" indicates that the
group can appear one or more times.  The regular
expression matches that way.  But instead of returning
a captured group for each character, it returns only the
last one.

The documentation in fact says that, at

http://docs.python.org/library/re.html

"If a group is contained in a part of the pattern that matched multiple
times, the last match is returned."

That's kind of lame, though. I'd expect that there would be some way
to retrieve all matches.


.findall



Findall does something a bit different. It returns a list of
matches of the entire pattern, not repeats of groups within
the pattern.

Consider a regular expression for matching domain names:

>>> kre = re.compile(r'^([a-zA-Z0-9\-]+)(?:\.([a-zA-Z0-9\-]+))+$')
>>> s = 'www.example.com'
>>> ms = kre.match(s)
>>> ms.groups()
('www', 'com')
>>> msall = kre.findall(s)
>>> msall
[('www', 'com')]

This is just a simple example.  But it illustrates an unnecessary
limitation.  The matcher can do the repeated matching; you just can't
get the results out.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: who moved reload?

[mark]

On Tue, 2011-04-19 at 12:41 -0400, Calvin Spealman wrote:
> I have a great solution : stop using reload. It often dangerous and
> more often silly. 
> 

Yeah, I'm there. Some of this is just learning a new way of thinking
about the language. I can see that not using reload at all makes more
sense (Run --> Run Module) from the editor window which does what I want
with a mouse click... so I'm done with reload...

In the past I didn't use IDLE... just vi, reload(yadda yadda) in the
terminal, repeat till bored. There were some inconsistencies with 
from yadda yadda import *...  and it seems that reload was deliberately
hidden in imp to solve some of that problem. And we can still use
imp.reload() to reload the module without stopping the interpreter, so
all is good.   

Thanks.



-- 
Kind regards,
M Harris

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why doesn't this asyncore.dispatcher.handle_read() get called?

[Dun Peal]

On Apr 20, 3:01 pm, Jean-Paul Calderone 
wrote:
> You didn't let the program run long enough for the later events to
> happen.  loop(count=1) basically means one I/O event will be processed
> - in the case of your example, that's an accept().  Then asyncore is
> done and it never gets to your custom handle_read.

Wow, a response from a Twisted founder and core developer =)

You were right, of course. Incrementing the count to 2 on the
asyncore.loop() calls makes the snippet work as expected.

I'd definitely rather use Twisted. It's unclear why transmitting a
bytestream message between two servers should require low-level socket
incantations such as `self.create_socket(socket.AF_INET,
socket.SOCK_STREAM)` or `self.setsockopt(socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1)`. The stdlib should offer a higher-level
asynchronous communication abstraction to support such a
straightforward usecase. Twisted does provide that, but my humble
needs can't justify the extra dependency cost.

Thanks a lot, D.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Teaching Python

[Terry Reedy]

On 4/20/2011 6:17 AM, Steven D'Aprano wrote:


It's hardly just the press. "Hack" is a fine old English word:

"The jungle explorer hacked at the undergrowth with his machete."

"I was so hungry, I didn't take the time to neatly slice up the meat, but
just hacked off a chunk and stuffed it in my mouth."

"Good lord, have you seen the completely botched job that carpenter has
done? He's such a hack!"

Given the wide range of pejorative meanings of "hack" going back at least
to the 19th century (to cut roughly without skill, a mediocre and
talentless writer, a person engaged to perform unskilled and boring
labour, a broken-down old horse, etc.), what's remarkable is that anyone
decided to start use "hack" in a non-pejorative sense.


How about "The indefatigable exploror hacked through the seemingly 
impenetrable jungle for a month to arrive at the well-hidden ancient 
temple. Since it was itself covered in overgrowth, he hacked away 
another month to reveal it in its ancient glory." Make the appropriate 
substution of code jungles and hard-won prize.


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: learnpython.org - an online interactive Python tutorial

[Terry Reedy]

On 4/20/2011 1:15 PM, Ron wrote:


I've written an online interactive Python tutorial atop Google App Engine:


 http://www.learnpython.org.

Currently giving 500 server error. Hope something clears up.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

multiprocessing in subpackage on windows

[Darren Dale]

I have two really simple scripts:

C:\Python27\Scripts\foo
---
if __name__ == '__main__':
import bar
bar.main()

C:\Python27\Lib\site-packages\bar.py
---
from multiprocessing import Pool

def task(arg):
return arg

def main():
pool = Pool()
res = pool.apply_async(task, (3.14,))

print res.get()

if __name__ == '__main__':
main()


If I run "python C:\[...]bar.py", 3.14 is printed. If I run "python C:\
[...]foo", I get a long string of identical errors:

 File "", line 1 in 
 File "C:\Python27\lib\multiprocessing\forking.py", line 346, in main
 prepare(preparation_data)
 File "C:\Python27\lib\multiprocessing\forking.py", line 455, in
prepare
 file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named foo


This same scheme works on Linux. What step have I missed to allow a
script to run code from a subpackage that uses multiprocessing?

Thanks,
Darren
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Grant Edwards]

On 2011-04-19, Grant Edwards  wrote:

> I'm have problems figuring out how to receive UDP broadcast packets on
> Linux.

Thanks to everybody for their help with what turned out to have little
or nothing to do with Python itself.

The main problem was reverse-path filtering (which is enabled by
default on Linux). Disabling it allowed me to receive packets sent to
255.255.255.255 regardless of the sender's IP address.

Another minor issue that cropped up was that I ended up having to use
two sockets: an Rx socket bound to ('',port) and a Tx socket bound to
(myip,port).

Even though the protocol spec said that subnet broadcasts were used,
tracing a few transactions between existing implementations showed
that only global broadcasts were used. More testing showed that
existing implementations won't respond to subnet broadcasts unless
they're already configured to be on the proper subnet.  So, receiving
subnet broadcasts that don't match an interface turned out to be a
wild goose chase.

It's all working now...

-- 
Grant Edwards   grant.b.edwardsYow! Everywhere I look I
  at   see NEGATIVITY and ASPHALT
  gmail.com...
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: meteclasses 2.x/3.x compatibility

[James Mills]

On Wed, Apr 20, 2011 at 10:18 PM, andrew cooke  wrote:
> What I do in Lepl is use two stages.  The first calls the type/metaclass 
> directly and the second subclasses that.  This avoids using the "sugar" that 
> changes between 2 and 3.
>
> So, for example, in 
> http://code.google.com/p/lepl/source/browse/src/lepl/matchers/matcher.py#40 I 
> have
>
>  _Matcher = ABCMeta('_Matcher', (object, ), {})
>
> and then
>
>  class Matcher(_Matcher):
>      ...

Thank Andrew. I like this approach Elegance wins for me :)

cheers
James

-- 
-- James Mills
--
-- "Problems are solved by method"
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: mystery string code - help!

[Uncle Ben]

On Apr 20, 1:01 pm, Dan M  wrote:
> > As a newbie Pythoner, I understand [] -1] but would some tell me how
> > '::' does its magic?
>
> > Uncle Ben
>
> The -1 is the "stride" or "step" argument. It's described 
> athttp://docs.python.org/release/2.3.5/whatsnew/section-slices.html
>
> Dan

Very helpful!

Ben
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem receiving UDP broadcast packets.

[Dan Stromberg]

On Wed, Apr 20, 2011 at 7:21 AM, Grant Edwards  wrote:
> On 2011-04-20, Dan Stromberg  wrote:
>> On Tue, Apr 19, 2011 at 8:12 PM, Dan Stromberg  wrote:
>>> I agree though that you're kind of pushing IP in a direction it wasn't
>>> intended to go.
>>
>> It just occurred to me: You might get some additional mileage out of
>> popping the network adapter into promiscuous mode.  In fact, it Might
>> be necessary irrespective of the rest of your approach.
>
> The network adapter is already receiving all the packets I want to
> receive, so putting it into promiscuous mode would only increase the
> number of unwanted packets.

I think tcpdump and tshark (was tethereal) will put the interface into
promiscuous mode so it can see more traffic; on OSF/1 (Tru64), we had
to do this manually for said programs to see all that was possible
(barring the presence of a switch not repeating packets the way
routers and hubs would).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Teaching Python

[Chris Angelico]

On Wed, Apr 20, 2011 at 8:17 PM, Steven D'Aprano
 wrote:
> It's hardly just the press. "Hack" is a fine old English word:
>

"Can you teach me how to hack?"

"Sure. Go to the tobacconists and buy him out, then smoke the lot.
You'll be hacking like a pro in no time!"

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Teaching Python

[Yico Gaga]

 well...
 they are freshmen wanna know how to learn programming , actually ,python
will be such a great programming language
 funded their hobbies ，you are looking for material and some website ，yeah
，why not give this problem to your students ，ask the freshmen to prepare for
some material and website ，this is a assignment 。you will get a surprise ，i
guess 。



2011/4/20 Passiday 

> Hello,
>
> I am planning to teach Python to a group of high school students, who have
> in-depth interest in programming, hacking etc.
>
> I am looking for some good material, what I could use as a basic guide when
> preparing the classes plan for the course - website or book, what would roll
> out the topic methodologically gradually. The target audience is someone who
> knows most basics of the programming, but doesn't mind being reminded about
> them now and then.
>
> Thanks for any suggestions!
>
> Passiday
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Teaching Python

[harrismh777]

Chris Angelico wrote:

Hacking??

1) Tinkering, programming, building furniture with an axe.

2) Breaking and entering in the electronic world.


Not so much.

In the comp.lang.python community hacking is most easily identified with 
the many one-liners that show up... that is the underlying spirit of 
hacking.


 block quote from RMS "On Hacking"===

It is hard to write a simple definition of something as varied as 
hacking, but I think what these activities have in common is 
playfulness, cleverness, and exploration. Thus, hacking means exploring 
the limits of what is possible, in a spirit of playful cleverness. 
Activities that display playful cleverness have "hack value".


Hackers typically had little respect for the silly rules that 
administrators like to impose, so they looked for ways around. For 
instance, when computers at MIT started to have "security" (that is, 
restrictions on what users could do), some hackers found clever ways to 
bypass the security, partly so they could use the computers freely, and 
partly just for the sake of cleverness (hacking does not need to be 
useful).


 / block quote from RMS "On Hacking"===

You can find the entire article here:

http://stallman.org/cgi-bin/showpage.cgi?path=/articles/on-hacking.html&term=hacking&type=norm&case=0


kind regards,
m harris
--
http://mail.python.org/mailman/listinfo/python-list