ANN: rom 0.16 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.16

[added] Thanks to a feature request from https://github.com/jrsmith ,
Boolean
column support is now available.
[added] DateTime, Date, and Time column types because I was already adding
Boolean columns.
#--- 0.15

[fixed] Thanks to https://github.com/MickeyKim who fixed Json columns so
that
they round-trip after multiple saves.
#--- 0.14

[fixed] Thanks to https://github.com/mayfield the manifest is fixed for the
source installation.
#--- 0.13

[fixed] updating a model will no longer unindex the model on attributes that
weren't updated. Thanks to https://github.com/mayfield for the bug
report,
analysis, and initial pull request that ultimately resulted in the fix.
[fixed] pip requires, versioning, etc., thanks to fixes from
https://github.com/mayfield
[changed] Model.get_by() will now work on any type of indexed columns.
Unique
columns work as before, and other columns get pass-through to the
Model.query interface.
[changed] Model.get_by() and Model.query.filter() will both accept single
numbers as the value to match on (you don't need to specify a range if
you
are querying by equality).
[changed] all changes will be documented in this changelog.txt file to keep
a
list of everything that is going on with the project.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] PEP 350: Codetags

[Josiah Carlson]

"Phillip J. Eby" <[EMAIL PROTECTED]> wrote:
> At 03:35 PM 9/26/2005 -0700, Micah Elliott wrote:
> >Please read/comment/vote.  This circulated as a pre-PEP proposal
> >submitted to c.l.py on August 10, but has changed quite a bit since
> >then.  I'm reposting this since it is now "Open (under consideration)"
> >at .
> 
> This seems a little weird to me.  On the one hand, seems like a cool idea 
> if you aren't using Eclipse or another IDE that tracks this stuff, but 
> still need some kind of tracking system.  But, if that is the case, the 
> notation seems a little bit overkill, especially with respect to tracking 
> who's responsible - i.e., just you.

There are various Python editors which have had support for a similar
style of tags for quite a while.  Some allow #: ,
others allow #  : , even others allow
more or less.  Some even count exclamation points as an indicator of
severity.

Personally, though I use tags in some of the code I write, and though
the editor I use (and wrote) supports tags, I'm of the opinion that an
unofficial spec is sufficient.  See koders.com and search for 'fixme' to
see some common variants.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] PEP 350: Codetags

[Josiah Carlson]

Micah Elliott <[EMAIL PROTECTED]> wrote:
> Josiah> an unofficial spec is sufficient.  See koders.com and search
> Josiah> for 'fixme' to see some common variants.
> 
> But that's the problem -- there are already a bunch of "unofficial"
> specs, which don't serve much purpose as such.  It's a cool site.  I
> spent some time browsing and I do see a lot of codetags in use (many
> thousands in Python code alone; I'm not sure if the number represented
> strings or files).  But I'll argue that this just reinforces the need
> for an *official* spec/guideline, so that the tools can do something
> with them.

Defining a spec for code tags doesn't mean that people will start using
them.  Why?  Because it is a documentation spec.  From my experience,
documentation specs are only adhered to by the organizations (companies,
groups, etc.) which the code is produced by and for, and they generally
define the code specs for their organization.

Further, even if it becomes a spec, it doesn't guarantee implementation
in Python editors (for which you are shooting for). Take a wander
through current implementations of code tags in various editors to get a
feel for what they support.  I've read various posts about what code
tags could support, but not what editors which implement code tags
and/or its variants actually currently support; which is a better
indication of what people want than an informal survey via email of
python-dev, python-list, and/or the PEP submission process.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Dr. Dobb's Python-URL! - weekly Python news and links (Jan 9)

[Josiah Carlson]

QOTW:  Jim Fulton: "[What's] duck typing?"
Andrew Koenig: "That's the Australian pronunciation of 'duct taping'."

"I'm thinking that the I-Ching is a vast untapped resource for programming
wisdom, plus it makes it funny." -- Mark Carter


Nick Coghlan brings up the 'lambdas are going away in 3.0' topic, which
has been discussed before:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/81e17b1d3ccba538

A user asks about a non-generator-based method for iteration using class
__iter__ and next() methods:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/6336a00ad217888a

Daniel Bickett asks about getting Twisted and wxPython working together,
and receives both a threaded and non-threaded version:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/73f1e7758225afc3

A question about iterating over floats...be careful!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/1278866159ac4429

A Boogieman asks whether Python is easily learned and what can be done
with it.  Quick answer: yes, and anything you are capable of doing:
This Boogieman later asks about GUI toolkits:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/e67b776df72eb336

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/d502698b4adacd01

Erik Bethke writes Majong game after using Python for a month.  Ahh, will
the wonders of Python never cease?

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/6d710e35007f0f32

Kamilche asks about getting warnings when not calling a function, and gets
pointed to pychecker, something more of us should be using:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/dff810b7dacd247c

Are people allowed to build commercial applications in Python?  YES!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/aa953388db68b196

Questions about asyncore brings up a web server in asyncore from _Python
Web Programming_, and a pointer to Twisted.  Alternatively, one could look
at the asynchat or smtpd modules in the standard library as sources of
inspiration:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/81360ec06013e36d


Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects references
to all sorts of Python resources.
http://www.vex.net/~x/parnassus/   

Much of Python's real work takes place on Special-Interest Group
mailing lists
http://www.python.org/sigs/

The Python Business Forum "further[s] the interests of companies
that base their business on ... Python."
http://www.python-in-business.org

Python Success Stories--from air-traffic control to on-line
match-making--can inspire you or decision-makers to whom you're
subject with a vision of what the language makes practical.
http://www.pythonology.com/success

The Python Software Foundation (PSF) has replaced the Python
Consortium as an independent nexus of activity.  It has official
responsibility for Python's development and maintenance. 
http://www.python.org/psf/
Among the ways you can support PSF is with a donation.
http://www.python.org/psf/donate.html

Kurt B. Kaiser publishes a weekly report on faults and patches.
http://www.google.com/groups?as_usubject=weekly%20python%20patch
   
Cetus collects Python hyperlin

Dr. Dobb's Python-URL! - weekly Python news and links (Jan 9)

[Josiah Carlson]

QOTW:  Jim Fulton: "[What's] duck typing?"
Andrew Koenig: "That's the Australian pronunciation of 'duct taping'."

"I'm thinking that the I-Ching is a vast untapped resource for programming
wisdom, plus it makes it funny." -- Mark Carter


Nick Coghlan brings up the 'lambdas are going away in 3.0' topic, which
has been discussed before:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/81e17b1d3ccba538

A user asks about a non-generator-based method for iteration using class
__iter__ and next() methods:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/6336a00ad217888a

Daniel Bickett asks about getting Twisted and wxPython working together,
and receives both a threaded and non-threaded version:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/73f1e7758225afc3

A question about iterating over floats...be careful!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/1278866159ac4429

A Boogieman asks whether Python is easily learned and what can be done
with it.  Quick answer: yes, and anything you are capable of doing:
This Boogieman later asks about GUI toolkits:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/e67b776df72eb336

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/d502698b4adacd01

Erik Bethke writes Majong game after using Python for a month.  Ahh, will
the wonders of Python never cease?

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/6d710e35007f0f32

Kamilche asks about getting warnings when not calling a function, and gets
pointed to pychecker, something more of us should be using:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/dff810b7dacd247c

Are people allowed to build commercial applications in Python?  YES!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/aa953388db68b196

Questions about asyncore brings up a web server in asyncore from _Python
Web Programming_, and a pointer to Twisted.  Alternatively, one could look
at the asynchat or smtpd modules in the standard library as sources of
inspiration:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/81360ec06013e36d


Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects references
to all sorts of Python resources.
http://www.vex.net/~x/parnassus/   

Much of Python's real work takes place on Special-Interest Group
mailing lists
http://www.python.org/sigs/

The Python Business Forum "further[s] the interests of companies
that base their business on ... Python."
http://www.python-in-business.org

Python Success Stories--from air-traffic control to on-line
match-making--can inspire you or decision-makers to whom you're
subject with a vision of what the language makes practical.
http://www.pythonology.com/success

The Python Software Foundation (PSF) has replaced the Python
Consortium as an independent nexus of activity.  It has official
responsibility for Python's development and maintenance. 
http://www.python.org/psf/
Among the ways you can support PSF is with a donation.
http://www.python.org/psf/donate.html

Kurt B. Kaiser publishes a weekly report on faults and patches.
http://www.google.com/groups?as_usubject=weekly%20python%20patch
   
Cetus collects Python hyperlin

Re: IDLE Problem in Windows XP

[Josiah Carlson]

Branden Smith <[EMAIL PROTECTED]> wrote:
> 
> Hi,
> 
> I am a teaching assistant for an introductory course at Georgia Tech
> which uses Python, and I have a student who has been unable to start
> IDLE on her Windows XP Home Edition machine. Clicking on the shortcut
> (or the program executable) causes the hourglass to appear momentarily
> (and the process to momentarily appear in the process monitor), but
> nothing happens thereafter.
...

> Does anyone have any ideas as to what might cause this problem? It shows
> up with both Python 2.4 and 2.3. Version 2.2 works as it should.

It is probably the socket issue.  To get past the socket issue,
according to the idle docs:

Running without a subprocess:

If IDLE is started with the -n command line switch it will run in a
single process and will not create the subprocess which runs the RPC
Python execution server.  This can be useful if Python cannot create
the subprocess or the RPC socket interface on your platform.  However,
in this mode user code is not isolated from IDLE itself.  Also, the
environment is not restarted when Run/Run Module (F5) is selected.  If
your code has been modified, you must reload() the affected modules and
re-import any specific items (e.g. from foo import baz) if the changes
are to take effect.  For these reasons, it is preferable to run IDLE
with the default subprocess if at all possible.



That is, have the student modify the shortcut to pass a '-n' argument 
(without the quotes) to the command.  If it works, great, if it doesn't,
a traceback would be helpful.

 - Josiah

--
http://mail.python.org/mailman/listinfo/python-list

Dr. Dobb's Python-URL! - weekly Python news and links (Jan 23)

[Josiah Carlson]

QOTW:  "XML with elementtree is what makes me never have [to] think about
XML again." -- Istvan Albert

"'Plays well with others' was a strong motivator for Python's design, and
that often means playing by others' rules." -- Tim Peters


Type mutability, and why some types are immutable.  This has been
discussed before, and should be a FAQ topic (if it isn't already):

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/f05c9cbb7e58db9b
http://www.python.org/tim_one/000195.html

Caching instances of classes for reuse:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/9178beed046956d2

Storing data in a persistant fashion.  One link discussing concurrency,
the other discussing data storage mechanisms.  A link to PyTables as a
way of storing and indexing large amounts of data, and the announcement
for Dejavu 1.3, which also stores persistant data:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/072e6da39a4e8760

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/7f2849d3099abfdc
http://pytables.sourceforge.net/html/WelcomePage.html

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/01a77dc70a55fc94

So, you want to use SFTP in Python?  Paramiko is the answer (or so I read):

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/e95e8482c023fe25
http://www.lag.net/paramiko/

Circular iteration, or any strange iteration tasks?  Try itertools!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/42bd800632199d2a

What do PyChecker and Pylint have to do with each other?
Are they configurable?  And do *you* know a half-dozen 
distinct formats for printing a multi-line text?

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/2ff633d4474e52e7/

Jeremy Bowers and Dan Stromberg know that some applications
should do the right thing, whether they happen to have a
bit-mapped user interface at run time or not:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/9ff27f0e5fea2045/

The Bangalore Python Meetup first convened on 22 January 2005:
http://python.meetup.com/158/events/?eventId=3974233&action=pastdetail

Why generator expressions sometimes don't beat list comprehensions, and
fitting Python to requirements.  Let not the anti-functional fashion
obscure that filter() still sometimes makes for better coding than the
comparable list comprehension:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/0155347dab27026d

http://groups-beta.google.com/group/comp.lang.python/msg/b15bab678ad9b2dc

Security is hard.  Even the narrow aspect of cryptography is
hard, if only for the complication state politics makes of the
mathematics and engineering involved.  Paul Rubin, Nick Craig-
Wood, and others present a few details:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/fcf8b0764d369cfa

No, really--Python does *that*, too.  Although many people
regard hardware interfacing, and its associated bit-twiddling,
as outside Python's purview, they're ... wrong.  See for 
yourself how Python makes for *clearer* codings than C or
Java or Assembler or ...:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/ea293235bef39473/


Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects refe

Dr. Dobb's Python-URL! - weekly Python news and links (Jan 23)

[Josiah Carlson]

QOTW:  "XML with elementtree is what makes me never have [to] think about
XML again." -- Istvan Albert

"'Plays well with others' was a strong motivator for Python's design, and
that often means playing by others' rules." -- Tim Peters


Type mutability, and why some types are immutable.  This has been
discussed before, and should be a FAQ topic (if it isn't already):

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/f05c9cbb7e58db9b
http://www.python.org/tim_one/000195.html

Caching instances of classes for reuse:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/9178beed046956d2

Storing data in a persistant fashion.  One link discussing concurrency,
the other discussing data storage mechanisms.  A link to PyTables as a
way of storing and indexing large amounts of data, and the announcement
for Dejavu 1.3, which also stores persistant data:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/072e6da39a4e8760

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/7f2849d3099abfdc
http://pytables.sourceforge.net/html/WelcomePage.html

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/01a77dc70a55fc94

So, you want to use SFTP in Python?  Paramiko is the answer (or so I read):

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/e95e8482c023fe25
http://www.lag.net/paramiko/

Circular iteration, or any strange iteration tasks?  Try itertools!

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/42bd800632199d2a

What do PyChecker and Pylint have to do with each other?
Are they configurable?  And do *you* know a half-dozen 
distinct formats for printing a multi-line text?

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/2ff633d4474e52e7/

Jeremy Bowers and Dan Stromberg know that some applications
should do the right thing, whether they happen to have a
bit-mapped user interface at run time or not:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/9ff27f0e5fea2045/

The Bangalore Python Meetup first convened on 22 January 2005:
http://python.meetup.com/158/events/?eventId=3974233&action=pastdetail

Why generator expressions sometimes don't beat list comprehensions, and
fitting Python to requirements.  Let not the anti-functional fashion
obscure that filter() still sometimes makes for better coding than the
comparable list comprehension:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/0155347dab27026d

http://groups-beta.google.com/group/comp.lang.python/msg/b15bab678ad9b2dc

Security is hard.  Even the narrow aspect of cryptography is
hard, if only for the complication state politics makes of the
mathematics and engineering involved.  Paul Rubin, Nick Craig-
Wood, and others present a few details:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/fcf8b0764d369cfa

No, really--Python does *that*, too.  Although many people
regard hardware interfacing, and its associated bit-twiddling,
as outside Python's purview, they're ... wrong.  See for 
yourself how Python makes for *clearer* codings than C or
Java or Assembler or ...:

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/ea293235bef39473/


Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects refe

Re: [Python-Dev] PEP-343 - Context Managment variant

[Josiah Carlson]

falcon <[EMAIL PROTECTED]> wrote:
> It's idea was token from Ruby. But I think, good idea is good whatever it 
> came from.
> It can be not Pythonic.

Just because something may be a good idea, doesn't mean that the idea is
Pythonic.  The only person who can truely say is Guido, but you can gain
some insight by reading the zens (from the console, 'import this').

The fact is, context management has a syntax that has been accepted. 
Your first example:

> def Synhronised(lock,func):
> lock.acquire()
> try:
> func()
> finally:
> lock.release()
> 
> lock=Lock()
> def Some():
> local_var1=x
> local_var2=y
> local_var3=Z
> def Work():
> global local_var3
> local_var3=Here_I_work(local_var1,local_var2,local_var3)
> Synhronised(lock,Work)
> return asd(local_var3)

... becomes ...

def Synchronized(lock):
lock.acquire()
try:
yield None
finally:
lock.release()

def Some():
...
with Synchronized(lock):
local_var3 = Here_I_work(local_var1,local_var2,local_var3)
...

Which is quite close to what you offer:

> Synhronised(lock,block)
> local_var3=Here_I_work(local_var1,local_var2,local_var3)


It is of my opinion that your proposal is destined to fail for at least
two reasons; 1) context management already has an accepted syntax 2)
ruby blocks were found (due to lack of evidence to the contrary) to gain
Python users very little (if anything) over the accepted context
management syntax.


To note; unless your post is of the utmost importance, cross-posting to
python-list and python-dev seems quite rude to me.  Further, a variant
of Ruby block syntax was offered way back in February and March (read
the python-dev archives available at
http://mail.python.org/pipermail/python-dev/ ).  Reading what was
already discussed may offer you insight into what (if any) open issues
were remaining, and precisely why the context management syntax was
supported over anonymous blocks/thunks/Ruby blocks.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Python and games

[Josiah Carlson]

Perhaps some of you would get a kick out of this.
 - Josiah


Like XML, scripting was extremely useful as both a mod tool and an
internal development tool.  If you don't have any need to expose code
and algorithms in a simple and safe way to others, you can argue that
providing a scripting language is not worth the effort.  However, if you
do have that need, as we did, scripting is a no brainer, and it makes
complete sense to use a powerful, documented, cross-platform standard
such as Python.
Python, like many good technologies, soon spreads virally throughout
your development team and finds its way into all sorts of applications
and tools.  In other words, Python begins to feel like a big hammer and
coding tasks look like nails.

 - Mustafa Thamer, Firaxis Games (talking about Civilization IV)
   Page 18, August 2005 Game Developer magazine
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: network programming without goto

[Josiah Carlson]

kent sin <[EMAIL PROTECTED]> wrote:
> 
> But that will make too many uncessary connection:
> 
>one connection can do a number of search before it
> got timeout, so I want to do as many search as
> possible before it got timeout. I think the connection
> cost is high here, and it also got some load on the
> remote servers which I do not want.
> 
> Thanks for the suggestion.

Indeed it will do many connections, but you weren't terribly specific
with what you wanted.

In any case, if you want a total of 3 failures under any situation, try
the following.

queries = [buildquery(t) for t in targets]

tries = 3
for host in hostlist:
q = iter(queries)
try:
cq = q.next()
except StopIteration:
continue
for i in xrange(tries):
conn = zoom.Connecton(host.ip, host.port)
fail = 0
while 1:
try:
r = conn.query(cq)
#deal with r
cq = q.next()
except StopIteration:
break
except:
fail = 1
break
if not fail:
break
try:
conn.close()
except:
    pass


 - Josiah


> 
> --- Josiah Carlson <[EMAIL PROTECTED]> wrote:
> 
> > 
> > kent sin <[EMAIL PROTECTED]> wrote:
> > > 
> > > Please help:
> > > 
> > > I was really blocked here. without goto I really
> > do
> > > not known how to do it.
> > > 
> > > The problem is to use PyZ3950 to consult a lists
> > of
> > > hosts and for each of them to search for a list of
> > > targets. Since the network is undetermined, there
> > were
> > > always some exceptions: I would like to allow it
> > to
> > > retry for 3 times. Moreover, during the query
> > process,
> > > the conn will timeout (set by the remote server).
> > > Reconnect is preferred before fail the current
> > search,
> > > but the reconnect may fail even if the first try
> > is
> > > succeed.
> > 
> > The trick with getting it to do what you want it to
> > do is to understand
> > what you want it to do.
> > 
> > Being that I can't understand what you want it to
> > do, the following is
> > just an interpretation of what I think you want it
> > to do.  If you want
> > it to do something different, then be clearer with
> > what you want it to
> > do and ask again.  Note that depending on the
> > semantics of your
> > connection, this could probably be made more
> > efficient.
> > 
> > 
> > queries = [buildquery(t) for t in targets]
> > 
> > for host in hostlist:
> > tries_remaining = 3 #tunable parameter
> > for q in queries:
> > for i in xrange(tries_remaining+1):
> > if i == tries_remaining:
> > break
> > try:
> > conn = zoom.Connecton(host.ip,
> > host.port)
> > r = conn.search(q)
> > conn.close()
> > break
> > except:
> > continue
> > 
> > #handle result r
> > break
> > 
> > tries_remaining -= i
> > 
> > if tries_remaining == 0:
> > break
> > 
> >  - Josiah
> > 
> > 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: wxStyledTextCtrl problem ?

[Josiah Carlson]

Paul Robson <[EMAIL PROTECTED]> wrote:
> 
> I have a wierd thing with a little editor app I'm writing.
> 
> I'm using wxStyledTextCtrl ; the wrapper for Scintilla. When I create the
> frame in wxPython, I use self.Editor.SetFocus() to put the focus initially
> on the editor.
> 
> This works fine.
> 
> Odd thing is though, switching back and forwards between applications
> alternately hides and reappears the caret.
> 
> Hit Alt-Tab twice and it disappears
> Hit Alt-Tab twice and it comes back.
> 
> It still works fine - there's just no caret. I've tried various things in
> EVT_ACTIVATE including Moving the Caret on screen, and forcing it to be
> visible - they are being called but it makes no difference.
> 
> Any ideas ?

I don't know why the problem you are having happens, but I would just
make a call to self.Editor.SetFocus() on an EVT_ACTIVATE event.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python API wrapper for C++ API

[Josiah Carlson]

"Mark Doberenz" <[EMAIL PROTECTED]> wrote:
> I'm fairly new at Python, but I've got a project I really want to do.
>  
> I'm working with a real-time 3D program called Deep Creator, and it has
> a C++ API written for it.  I'm wondering what it would take to create a
> Python wrapper that would allow me to write Python programs that would
> call the C++ API.
>  
> The main issue I'm running into is that I need to create a DLL that I
> load as a plugin into Deep Creator.  I then will specify a python
> program inside Deep Creator that I want to run, and the DLL plugin will
> be the link between the python program and Deep Creator.
>  
> I'd really appreciate any ideas or issues that people see with this set
> up.

The first thing you need to decide is whether you want to call C++ from
Python, Python from C++, both, or whether you want to embed a Python
interpreter inside your application.

The standard places to look for this kind of thing are ctypes, SWIG, and
Boost.Python.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Few things

[Josiah Carlson]

[EMAIL PROTECTED] (bearophile) wrote:
> 
> Thank you for the comments and answers, and sorry for my answering
> delay...
> 
> Josiah Carlson:
> 
> >Decorators can do this without additional syntax. Think @accepts and
> @returns.<
> 
> The purpose of those pre-post is to write something simile and very
> *clean* that states what inputs and outputs must be. This is an
> example of a pre-post conditional for a sorting function taken from
> that site (all this is inside the docstring of the function):
> 
> pre:
> # must be a list
> isinstance(a, list)
> 
> # all elements must be comparable with all other items
> forall(range(len(a)),
>lambda i: forall(range(len(a)),
> lambda j: (a[i] < a[j]) ^ (a[i] >= a[j])))
> 
> post[a]:
> # length of array is unchanged
> len(a) == len(__old__.a)
> 
> # all elements given are still in the array
> forall(__old__.a, lambda e: __old__.a.count(e) == a.count(e))
> 
> # the array is sorted
> forall([a[i] >= a[i-1] for i in range(1, len(a))])

That is simple and clean?  In my opinion, if one wants to write such
complicated pre and post conditions, one should have to write the pre
and post condition functions that would do the test, and either use
decorators, or use calls within the function to do the tests.  That is
the way it is done now, and I personally don't see a good reason to make
a change.  Then again, I document and test, and haven't used pre/post
conditions in 5+ years.


> Surely such things can be passed (at least as strings) to the @accepts
> and @returns decorators (using a "decorate" identifier instead of @ is
> probably nicer, because the @ makes Python look more like Perl, but
> I've seen that lots of people have already discussed such topic). Such
> testing performed by such decorators can be "switched off" with a
> global boolean flag when the program is debugged and tested.
> So now someone can write and let standardise a couple of good @accepts
> and @returns decorators/functors :-]

Discussion of the @ decorator syntax is a moot point.  Python 2.4 final
was released within the last couple days and uses @, which Guido has
decided is the way it will be.  You are around 6 months too late to have
anything to say about what syntax is better or worse.  It is done.


> >Having a 'faq' for permutation and combination generation would be
> 99% of the way there.<
> 
> Uh, I'm sorry, but I don't understand :-]
> Aren't such functions quite well defined?

Think of it like an 'example' in the documentation, where the code is
provided for doing both permutations and combinations.  There exists a
FAQ for Python that addresses all sorts of "why does Python do A and not
B" questions.  Regardless, both are offered in the Python cookbook.


> >[Fixed] Quickselect, really, doesn't gain you a whole lot. Sure, it's
> a log factor faster to select a median, but many algorithms involving
> selecting medians (at least the ones that I run into in CS theory) end
> up repeatedly (logn times) selecting the 'kth' smallest element
> (varying k's), where sorting would actually run slightly faster.<
> 
> I've done some tests with a Quickselect that I have essentially
> translated and adapted to pure Python from "Numerical Recipes" (it
> seems a bit faster than the Quickselect coded by Raymond Hettinger
> that can be seen in the cookbook). I have seen that on my PC, on
> random sequence of FP numbers, a *single* Quickselect (to find just
> the median) is faster than the standard sort for lists longer than
> about 3 million elements. So it's often useless.
> But using Psyco, that Quickselect becomes 5-6 times faster (for long
> lists), so it beats the (good) standard Sort for lists longer than
> 600-3000 elements. If the Quickselect works in place (as the sort)
> then it returns a partially ordered list, and you can use it to
> quickly select other positions (so for close positions, like the
> computing of the two central values for the median, the complexity of
> the second select is nearly a constant time).
> So coding the Quickselect in C/Pyrex can probably make it useful.
> If you are interested I can give the Python Quickselect code, etc.

No thank you, I have my own.


> Nick Coghlan>I believe the OP was objecting to the spelling of "this
> integer literal is hex" and "this integer literal is octal".<
> 
> Right.
> 
> 
> Josiah Carlson>Regardless, I also don't believe the "I don't like
> this" without "this is the way it should be" will result in anything.<
> 
> You are

Re: decorators ?

[Josiah Carlson]

km <[EMAIL PROTECTED]> wrote:
> 
> Hi all,
>  was going thru the new features introduced into python2.4 version.
> i was stuck with 'decorators' -  can someone explain me the need of such a 
> thing called decorators ? 

Decorators are not 'needed'.  They are a convenience.

Among the things that they are meant to make more convenient are the
following...

class foo:
@staticmethod
def goo(arg, ...):
pass

rather than...

class foo:
def goo(arg, ...):
pass
goo = staticmethod(goo)

And other things of its ilk.  Where they will be used the most right off
is in function signatures for PyObjC bindings in (I believe) OSX by Bob
Ippolito.  Using his example:

class FooClass(NSObject):
 def returnsAnInteger(self):
 return 1
 returnsAnInteger = objc.selector(returnsAnInteger, signature='i@:')

 def returnsVoidTakesAnInteger_(self, anInteger):
 pass
 returnsVoidTakesAnInteger_ = \
 objc.selector(returnsVoidTakesAnInteger_, signature='v@:i')

...becomes...

def signature(sig):
 def _signature(fn):
 return objc.selector(fn, signature=sig)
 return _signature

class FooClass(NSObject):
 @signature('i@:')
 def returnsAnInteger(self):
 return 1

 @signature('v@:i')
 def returnsVoidTakesAnInteger_(self, anInteger):
 pass


 - Josiah

--
http://mail.python.org/mailman/listinfo/python-list

Re: A way to wait Python event

[Josiah Carlson]

[EMAIL PROTECTED] (Chang LI) wrote:
> 
> I tried to launch "python.exe test.py" in another program. After the
> launch the console was showed and exited on Windows. I want the
> console stay there. Is there a Python statement to wait an event loop
> like Tcl's "after forever"?

I'm a fan of...

_ = raw_input('press enter to continue...')


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: decorators ?

[Josiah Carlson]

Jacek Generowicz <[EMAIL PROTECTED]> wrote:
> 
> BJörn Lindqvist <[EMAIL PROTECTED]> writes:
> 
> > I think the essence of decorators is that it makes it possible to do
> > in Python what you in other languages do with method qualifiers.
> 
> I find it fascinating that the addition of a bit of syntax sugar gives
> the perception that a whole range of new and previously unthinkable
> possibilities have opened themselves before us.
> 
> @X
> def Y...
> ...
> 
> 
> is merely syntax sugar (or syntax ammonia, for some) for
> 
> 
> def Y...
> ...
> Y = X(Y)
> 
> Anything you can do with decorators, you could do before (with the
> exception of rebinding the __name__ of functions).
> 
> And yet, that bit of syntax sugar really _does_ make a big difference
> to the lengths that people are prepared to take the possibilities that
> the underlying feature affords them.


Technically, everything can be performed in assembly.  The point of
syntactic sugar (or ammonia) is to make things less painful.  While
everything was possible before, adding the decorators /after/ defining
the function hid the decorators, and was susceptible to mistyping.

I previously posted about Philip Eby's use of decorators in PyObjC.  In
his case, it saved him from typing 40-character function names 3
times.

 - Josiah

--
http://mail.python.org/mailman/listinfo/python-list

Re: using cmd.exe as a telnet client

[Josiah Carlson]

Donnal Walter <[EMAIL PROTECTED]> wrote:
> 
> Several months ago I tried using the telnet module (on Windows XP) to 
> communicate with a proprietary host on our network. This was 
> unsuccessful due to problems with "option negotiation", and I gave up on 
> the project for a while. I still have need for this, however, so I 
> recently started thinking about alternatives. I suppose I could dig deep 
> enough into option negotiation to use the socket module (with telnet as 
> a guide), but I am hoping to find a way to use fewer synapses.
> 
> Using the Windows C:> prompt (cmd.exe) I can telnet into this host with 
> no problems (the Windows telnet client performs option negotiation just 
> fine). Is there a straightforward way of using os.popen() (or os.fork() 
> or os.exec*() or os.wait*()) to connect to the host from Python via the 
> Windows telnet client?

I am sure it is possible with a little bit of pywin32 business.  You may
have better luck with the new subprocess module in Python 2.4, as it is
written with communication to/from console applications in mind.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Semaphore or what should I use?

[Josiah Carlson]

Pierre Barbier de Reuille <[EMAIL PROTECTED]> wrote:
> 
> Ville Vainio a écrit :
> >>"Bastian" == Bastian Hammer <[EMAIL PROTECTED]> writes:
> > 
> > 
> > Bastian> Now I have to make sure, that both threads are
> > Bastian> synchronal, 1 thread edits something and the other is
> > Bastian> blocked until the first thread is ready.
> > 
> > Bastian> Isn´t it a good idea to do this with a semaphore?
> > 
> > Semaphore will do, but this is a classical use case for
> > threading.Lock.
> > 
> > There should be lots of stuff regarding locks (or more googleably,
> > "mutexes") on the net.
> > 
> 
> I don't agree. Mutexes (or locks) are best suited for critical sections 
> (ie. sections that cannot be run by many thread at the same time). The 
> kind of synchonisation Bastian want is not really semaphore either but 
> more event. This python "Event" object is described in the section 7.5.5 
> of the documentation of Python 2.3. There is no example, but I think 
> Event are quite strait forward : you creates it, then some thread block, 
> waiting the event to occure while some other thread execute until it set 
> the event, allowing the blocked thread to go on its own execution :)

You can agree or disagree as much as you want.  Fundamentally, they are
all equivalent.

The only thing that makes mutex 'special' is that one can have an
optional 'call this function with this argument when it gets the lock',
but that can be implemented with a standard Lock, Condition, Event,
Semaphore, etc.


 - Josiah

--
http://mail.python.org/mailman/listinfo/python-list

Re: string slicing

[Josiah Carlson]

"Fredrik Lundh" <[EMAIL PROTECTED]> wrote:
> 
> Ishwor wrote:
> 
> >  I am trying some interactive examples here where i have come across
> > inconsistencies??? :O
> 
> obsession with implementation artifacts is a premature optimization,
> and should be avoided.

[snip]

> a suggestion: if you really want to be productive in python, forget about
> "is" for a while.

The above two tips are in my "top-10 things to tell newbs" list.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: string slicing

[Josiah Carlson]

Ishwor <[EMAIL PROTECTED]> wrote:
> 
> On Sun, 05 Dec 2004 10:31:12 -0800, Josiah Carlson <[EMAIL PROTECTED]> wrote:
> > 
> > "Fredrik Lundh" <[EMAIL PROTECTED]> wrote:
> 
> > > a suggestion: if you really want to be productive in python, forget about
> > > "is" for a while.
> 
> I know what Fredrik means here (Thanx Frederick :) ) but IMHO if the
> pattern of forgetting something is taken everytime someone tries to
> scratch under the hood, then seriously it'll take ages before someone
> seriously learns a language!
>  
> > The above two tips are in my "top-10 things to tell newbs" list.
> 
> You should consider having your own *nice* nifty things to say about
> newbs (like me) rather than try to point at someone else's
> intellectual property. If you are running out of ideas consider
> reading one of those "Complete Idiot's guide to " :)

Goodness, you got right snippy with the 'intellectual property' thing.
I'm sorry if it sounded as if I had meant "I came up with this before",
I did not mean it that way.  What I meant was "good advice", "I've seen
this advice before", and "I've previously taken note of it".  As an
aside, I also believe that if Fredrik were concerned about my supposed
theft of his 'advice IP', he would mention it


Confusions about 'is' are quite common.  Perhaps every week a new thread
in regards to object identity is offered, usually asking about integer,
string or list identity.  I'm not having much luck with the google
incantation right now, but my email archive is spitting back a thread on
"interning strings" from early November, which discusses strings and
identity.

Really, everyone who is thinking of using 'is' should ask themselves
whether or not it is truely important to have the same object.  In some
cases it matters, and in others cases it doesn't matter.  The more
experience one has with Python and the implementations of algorithms,
the more obvious such choices will become.

A perfect example of a case when using 'is' is proper is by Barry Warsaw
(http://mail.python.org/pipermail/python-dev/2004-March/043224.html):

missing = object()

if d.get('somekey', missing) is missing:
   # it ain't there


In the case of it not mattering, there was a series of threads in March
of this year in python-dev, discussing replacing the 'is' operator
strictly comparing object identity with a 'can be replaced by' operator,
spelled 'is'.  See the threads entitled "redefining is" here:
http://mail.python.org/pipermail/python-dev/2004-March/thread.html
Of course it came out that it didn't happen ('is' is much faster, and
doesn't require objects to define a mechanism to get at their contents,
etc.), but it is not uncommon for people who think they want 'is' to
really want 'can be replaced by', especially in the case of tuples:

>>> a = tuple(list('hello'))
>>> b = tuple(list('hello'))
>>> a == b
True
>>> a is b
False
>>> CanBeReplacedBy(a, b)
True
>>> 

(of course you need a proper implementation of CanBeReplacedBy, of which
a few are offered in the previously mentioned 'replacing is' threads).


Now, whether or not knowing what 'is' does, or whether knowing the
internals in regards to how object identity works, is necessary for
learning the language, I don't know.  It's been a few years since I was
a newb, but considering all of the questions regarding 'is' usually
resulting with python-list saying "you want '=='", I'd offer, 'it is not
necessary'.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mean, median, and mode

[Josiah Carlson]

[EMAIL PROTECTED] (Sean McIlroy) wrote:
> 
> >>> median = lambda x: (max(x)-min(x))/2

That is /not/ the median in the general case.

median = lambda x: x.sort() or x[len(x)//2]

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Deadlock detection

[Josiah Carlson]

Duncan Grisby <[EMAIL PROTECTED]> wrote:
> 
> Hi,
> 
> Does anyone know of a deadlock detector for Python?  I don't think it
> would be too hard to hook into the threading module and instrument
> mutexes so they can be tested for deadlocks. I've googled around but I
> haven't found anything.

I'm not aware of a deadlock dector for any language.  I propose that a
completely working deadlock detector is NP-Complete, as it would, I
believe, necessitate solving the halting problem.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mean, median, and mode

[Josiah Carlson]

"Robert Brewer" <[EMAIL PROTECTED]> wrote:
> 
> Josiah Carlson wrote:
> > [EMAIL PROTECTED] (Sean McIlroy) wrote:
> > > 
> > > >>> median = lambda x: (max(x)-min(x))/2
> > 
> > That is /not/ the median in the general case.
> > 
> > median = lambda x: x.sort() or x[len(x)//2]
> 
> That...is a really sneaky use of null return values. I like. :)

Thank you, I'm just using a paradigm (exploiting lambdas) that I picked
up while going through various functional programming modules.

If you're really good, you can do crazy stuff with list comprehensions.
I'll leave that exercise up to the reader, as many of my examples are
in for-pay code.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Mean, median, and mode

[Josiah Carlson]

[EMAIL PROTECTED] (Michael Fuhr) wrote:
> 
> Josiah Carlson <[EMAIL PROTECTED]> writes:
> 
> 
> > "Robert Brewer" <[EMAIL PROTECTED]> wrote:
> > > 
> > > Josiah Carlson wrote:
> > > > 
> > > > median = lambda x: x.sort() or x[len(x)//2]
> > > 
> > > That...is a really sneaky use of null return values. I like. :)
> >
> > Thank you, I'm just using a paradigm (exploiting lambdas) that I picked
> > up while going through various functional programming modules.
> 
> print median([1, 2, 3, 4, 5, 6])
> 4
> 
> Shouldn't the median be 3.5?

The wikipedia entry (as mentioned by Peter Hanson in another post) is
incomplete. There are 3 valid medians for even-lengthed sequences:
sorted(seq)[len(seq)//2]
sorted(seq)[len(seq)//2-1]
sum(sorted(seq)[len(seq)//2-1:len(seq)//2+2])/2.0

All are correct, what is /desired/ is generally application specific. If
you want the statistical median, 3.5 is the answer.  If you want a
computer science median (for quicksort, etc.), 3 or 4 is sufficient.  I
chose the shortest implementation (using the author's style, without
sorted, because I forgot about the new builtin), which gives the greater
(or equal) of the two possible computer science median values.


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Dr. Dobb's Python-URL! - weekly Python news and links (Dec 10)

[Josiah Carlson]

QOTW:  "I still think this is a silly idea, but at least it doesn't track mud
all over Python's nice clean rugs." -- Michael J. Fromberger

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/dde861393aa5a68/eb3a5e53f9743413

"Basically, in tk, canvases are for vector drawing; in other toolkits, they're
more for bitmap drawing. And this makes quite a difference..." -- Eric Brunel


Lovers of the eric3 editor get a new 3.5.1 release.
http://mail.python.org/pipermail/python-list/2004-November/252450.html

Lovers of the Wing IDE editor get a new 2.0.1 release.
http://mail.python.org/pipermail/python-list/2004-December/253698.html

Sometimes the `struct` module's use and structure aren't obvious to the
new user.  Tim Peters and Peter Hanson help clarify.
http://mail.python.org/pipermail/python-list/2004-November/252741.html

A company in New York is looking for a Python deveoper for Zope work.
I know there has to be someone in NYC with that kind of experience.
http://mail.python.org/pipermail/python-list/2004-December/252840.html

Locks, mutexes, Semaphores - Oh My!  Discussion of when to use what.
Hint: they are basically the same, only speed and internal implementation
really set them apart.  RLock, Condition, Event, etc., add functionality.
http://mail.python.org/pipermail/python-list/2004-December/252869.html

Gordon Williams asks about set intersections.  Raymond Hettinger
confirms that set intersection with the `set` module does the right
thing.
http://mail.python.org/pipermail/python-list/2004-December/253142.html

A nice one-liner for DNA recombinations.  Be careful with the size of
your input, you don't want to go and segfault with a 512M element list...
http://mail.python.org/pipermail/python-list/2004-December/252970.html

Armin Ringo releases Psyco 1.3, for those of us who like our Python to
execute faster.  Support for Python 2.1-2.4 is included, as well as more
support for local variables.
http://mail.python.org/pipermail/python-list/2004-December/253234.html

Alia Khouri asks "why no python setup.py uninstall?" and gets basically
no response.  It brings up the topic of Python package management, which
has been discussed before.
http://mail.python.org/pipermail/python-list/2004-December/253316.html
http://python.org/peps/pep-0301.html

Andre Roberge asks for advice on naming a new (claimed better) version
of "Guido van Robot", which is an instructional tool for new programmers.
http://mail.python.org/pipermail/python-list/2004-December/253453.html



Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects references
to all sorts of Python resources.
http://www.vex.net/~x/parnassus/   

Much of Python's real work takes place on Special-Interest Group
mailing lists
http://www.python.org/sigs/

The Python Business Forum "further[s] the interests of companies
that base their business on ... Python."
http://www.python-in-business.org

Python Success Stories--from air-traffic control to on-line
match-making--can inspire you or decision-makers to whom you're
subject with a vision of what the language makes practical.
http://www.pythonology.com/success

The Python Software Foundation (PSF) has replaced the Python
Consortium as an independent nexus of activity.  It has official
responsibility for Python's development and maintenance. 
http://www.python.org/psf/
Among the ways y

Dr. Dobb's Python-URL! - weekly Python news and links (Dec 10)

[Josiah Carlson]

QOTW:  "I still think this is a silly idea, but at least it doesn't track mud
all over Python's nice clean rugs." -- Michael J. Fromberger

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/dde861393aa5a68/eb3a5e53f9743413

"Basically, in tk, canvases are for vector drawing; in other toolkits, they're
more for bitmap drawing. And this makes quite a difference..." -- Eric Brunel


Lovers of the eric3 editor get a new 3.5.1 release.
http://mail.python.org/pipermail/python-list/2004-November/252450.html

Lovers of the Wing IDE editor get a new 2.0.1 release.
http://mail.python.org/pipermail/python-list/2004-December/253698.html

Sometimes the `struct` module's use and structure aren't obvious to the
new user.  Tim Peters and Peter Hanson help clarify.
http://mail.python.org/pipermail/python-list/2004-November/252741.html

A company in New York is looking for a Python deveoper for Zope work.
I know there has to be someone in NYC with that kind of experience.
http://mail.python.org/pipermail/python-list/2004-December/252840.html

Locks, mutexes, Semaphores - Oh My!  Discussion of when to use what.
Hint: they are basically the same, only speed and internal implementation
really set them apart.  RLock, Condition, Event, etc., add functionality.
http://mail.python.org/pipermail/python-list/2004-December/252869.html

Gordon Williams asks about set intersections.  Raymond Hettinger
confirms that set intersection with the `set` module does the right
thing.
http://mail.python.org/pipermail/python-list/2004-December/253142.html

A nice one-liner for DNA recombinations.  Be careful with the size of
your input, you don't want to go and segfault with a 512M element list...
http://mail.python.org/pipermail/python-list/2004-December/252970.html

Armin Ringo releases Psyco 1.3, for those of us who like our Python to
execute faster.  Support for Python 2.1-2.4 is included, as well as more
support for local variables.
http://mail.python.org/pipermail/python-list/2004-December/253234.html

Alia Khouri asks "why no python setup.py uninstall?" and gets basically
no response.  It brings up the topic of Python package management, which
has been discussed before.
http://mail.python.org/pipermail/python-list/2004-December/253316.html
http://python.org/peps/pep-0301.html

Andre Roberge asks for advice on naming a new (claimed better) version
of "Guido van Robot", which is an instructional tool for new programmers.
http://mail.python.org/pipermail/python-list/2004-December/253453.html



Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects references
to all sorts of Python resources.
http://www.vex.net/~x/parnassus/   

Much of Python's real work takes place on Special-Interest Group
mailing lists
http://www.python.org/sigs/

The Python Business Forum "further[s] the interests of companies
that base their business on ... Python."
http://www.python-in-business.org

Python Success Stories--from air-traffic control to on-line
match-making--can inspire you or decision-makers to whom you're
subject with a vision of what the language makes practical.
http://www.pythonology.com/success

The Python Software Foundation (PSF) has replaced the Python
Consortium as an independent nexus of activity.  It has official
responsibility for Python's development and maintenance. 
http://www.python.org/psf/
Among the ways y

Dr. Dobb's Python-URL! - weekly Python news and links (Dec 25)

[Josiah Carlson]

QOTW:  "Python, the language that wraps tightly around a problem and
swallows it whole." -- Raymond Hettinger



Because it is so important, this link is re-posted for the future
(originally appeared in the December 15 Python-Url! posting by
Cameron Laird): Martin Bless summarizes in an extremely valuable
post what you need to know to generate Python-related executable
binaries targeted for Windows*.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/73f29284d1e031c7

Mark English asks about running a tk GUI using a background thread.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/1f8d0a962f3fba49

You can register for PyCon DC 2005 on the web.
http://www.python.org/pycon/2005/register.html

A reminder to those who don't restart their editors while running
their scripts:  you are probably looking for reload().

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/9cd52e7b2dc4e278

Pointers to a handful of "graphics drawing" toolkits, hopefully to
result in a learning tool for sorting algorithms in Python.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/f8ed1f1e9b7e8063

Importing binary modules from ZIPs by Thomas Heller.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/bf37adb498126c0a

For the masochistic among us, a 120+ message thread on the history
of BASIC, Python, and why BASIC is probably not the first language
one should learn.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/5730ea9521effd09

timbot exhibits a concrete instance of the long-conjectured 
universal follow-up, that is, a post which mis-answers all
confusion equally well.  Merry Christmas to you, too.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/a472b72906740e8f

Diez B. Roggisch asks about decorator peculiarities and receives
a clarified version of his decorators in function and class form.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/079f86a5f2af15a9

Mike Meyer proposes a rational number module for Python.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/2074c5329e4004eb

Stephen Kellet announces a thread and lock validation tool for
Python and its test-oriented fans.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/6dfa265b1bd16048

SPE version 0.7.0.a is released.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/d71e84bb54aa4d9d

PythonCAD 20 appears.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/af83a14bdbd806e2

Python is now available on NokiaS60 phones.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/4bb1044d96b83962





Everything Python-related you want is probably one or two clicks away in
these pages:

Python.org's Python Language Website is the traditional
center of Pythonia
http://www.python.org
Notice especially the master FAQ
http://www.python.org/doc/FAQ.html

PythonWare complements the digest you're reading with the
marvelous daily python url
 http://www.pythonware.com/daily  
Mygale is a news-gathering webcrawler that specializes in (new)
World-Wide Web articles related to Python.
 http://www.awaretek.com/nowak/mygale.html 
While cosmetically similar, Mygale and the Daily Python-URL
are utterly different in their technologies and generally in
their results.

comp.lang.python.announce announces new Python software.  Be
sure to scan this newsgroup weekly.

http://groups.google.com/groups?oi=djq&as_ugroup=comp.lang.python.announce

Brett Cannon continues the marvelous tradition established by 
Andrew Kuchling and Michael Hudson of intelligently summarizing
action on the python-dev mailing list once every other week.
http://www.python.org/dev/summary/

The Python Package Index catalogues packages.
http://www.python.org/pypi/

The somewhat older Vaults of Parnassus ambitiously collects references
to all sorts of Python resources.
http://www.vex.net/~x/parnassus/   

Much of Python's real work takes place on Special-Interest Group
mailing lists
http://www.python.org/sigs/

The Python Business Forum "further[s] the interests of companies
that base their business on ... Python."
http://www.python-in-business.org

Python Success Stories--from air-traffic control to on-line
match-making--can inspire you or decision-makers to whom you're
subject with a vision of what the language makes p

ANN: rom 0.25.0 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

Big change today: rom now supports fast prefix, suffix, and pattern match
queries over your data. The method is based on the autocomplete process
described in my book, Redis in Action

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#-- 0.25.0
---
[changed] version numbers to account for bugfixes vs. feature updates.
[added] columns can now be defined to allow for prefix and/or suffix
queries.
Enabling prefix queries also enables arbitrary pattern matching over
your
data.
[fixed] in some cases, rom would allow the definition of multiple primary
keys, of which only one would ever be used (inconsistently). This will
now
result in an error.
[changed] defaulted to assume Lua is available on Redis, which has been
released for over 15 months at this point. You can disable support via
a call to rom._disable_lua_writes().
[added] the ability to cache and get the key that holds the result of a
query,
which can be used for pagination, etc. See: Query.cached_result()
[warning] using rom versions of 0.23 with 0.25.0 when prefix and suffix
indexes are enabled can result in improper results from prefix, suffix,
and/or pattern queries, and can result in orphan data living in prefix
or
suffix indexes. Upgrade all of your clients!
[changed] temporary keys for queries are now prefixed with the name of the
model over which queries are being executed on. This should effect
basically zero people, but can allow for query cleanup in the off chance
of a failure during execution.
#- 0.23 (unreleased)
-
[changed] reduced number of round trips for single-filter queries by 1,
thanks
to https://github.com/MickeyKim for the report.
#--- 0.22

[fixed] size estimation for intersection ordering when filtering has now
been
fixed, thank you to https://github.com/MickeyKim for the report and the
change (should improve performance).
[fixed] an issue with some types when trying to update attributes has now
been
fixed, thank you to https://github.com/denisvolokh for the report.
[changed] improved performance for simple numeric range queries of the form
Model.get_by(attr=value) or Model.get_by(attr=(min, max)) by roughly a
factor of 60x or better in some cases. Thank you to
https://github.com/MickeyKim for the report on poor performance.
#--- 0.21

[fixed] upload for rom 0.20 was missing new columns.py, now fixed
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: rom 0.18 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.18

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
and received an interim patch for a bug that could cause deleted
entities
to be resurrected on session.commit() or session.flush() . This has now
been fixed and a testcase has been added.
#--- 0.17

[added] LGPL 3 licensing option.
#--- 0.16

[added] Thanks to a feature request from https://github.com/jrsmith ,
Boolean
column support is now available.
[added] DateTime, Date, and Time column types because I was already adding
Boolean columns.
#--- 0.15

[fixed] Thanks to https://github.com/MickeyKim who fixed Json columns so
that
they round-trip after multiple saves.
-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: rom 0.19 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.19

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
of a bug when using unique indexes, which is now fixed and has a
testcase.
#--- 0.18

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
and received an interim patch for a bug that could cause deleted
entities
to be resurrected on session.commit() or session.flush() . This has now
been fixed and a testcase has been added.
#--- 0.17

[added] LGPL 3 licensing option.
#--- 0.16

[added] Thanks to a feature request from https://github.com/jrsmith ,
Boolean
column support is now available.
[added] DateTime, Date, and Time column types because I was already adding
Boolean columns.
-- 
http://mail.python.org/mailman/listinfo/python-list

Ann: rom 0.20 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.20

[changed] Added exception when performing .all(), .execute(), or .count() on
query objects that have had no filters or attribute ordering provided.
This addresses issue #12.
[changed] Moved column definitions to their own module, shouldn't affect any
normal uses of rom.
[added] For users of Redis 2.6 and later, there is a beta Lua-enabled
writing
option that allows for multiple unique columns on models. In some cases,
this may improve performance when writing many entities very quickly.
[added] The ability to reload an entity from Redis, optionally discarding
any
modifications to the object itself. Check out the documentation for
Model.refresh(), Session.refresh(), and Session.refresh_all()
[added] Tests for the newly changed/added features.
[changed] Tests no longer use flushdb() - all test models/indexes/etc. are
prefixed with RomTest, and we find/delete such keys before and after any
tests are run. Now anyone can reasonably run the test suite.
#--- 0.19

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
of a bug when using unique indexes, which is now fixed and has a
testcase.
#--- 0.18

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
and received an interim patch for a bug that could cause deleted
entities
to be resurrected on session.commit() or session.flush() . This has now
been fixed and a testcase has been added.
#--- 0.17

[added] LGPL 3 licensing option.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: RPQueue 0.21

[Josiah Carlson]

Hello everyone,

For those of you who didn't know, if you are interested in a Redis-backed
time and/or fifo-queue with priorities, retries, etc., to be used with
Python, one exists and is mature: it's called RPQueue, and it seeks to
simplify your life of task execution. The recent changelog entries are
below my signature.

The package has been around for two years next month, and does exactly what
you expect it to do - no more, no less.

You can find the package at:
https://github.com/josiahcarlson/rpqueue/
https://pypi.python.org/pypi/rpqueue

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.21

[changed] where available, rpqueue will now use Lua to move delayed tasks
from
the time-based priority queue to fifo queues. This should reduce
overhead
in all cases, and should drastically improve performance for those that
use large numbers of delayed tasks.
[changed] the version number is now PEP 386/440 compliant.
[added] this changelog that will document updates/changes/improvements in an
easily referenced location.
[fixed] thanks to https://github.com/dmaust , rpqueue added a missing
'redis'
requirement.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: rom 0.21 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.21

[fixed] upload for rom 0.20 was missing new columns.py, now fixed
#--- 0.20

[changed] Added exception when performing .all(), .execute(), or .count() on
query objects that have had no filters or attribute ordering provided.
This addresses issue #12.
[changed] Moved column definitions to their own module, shouldn't affect any
normal uses of rom.
[added] For users of Redis 2.6 and later, there is a beta Lua-enabled
writing
option that allows for multiple unique columns on models. In some cases,
this may improve performance when writing many entities very quickly.
[added] The ability to reload an entity from Redis, optionally discarding
any
modifications to the object itself. Check out the documentation for
Model.refresh(), Session.refresh(), and Session.refresh_all()
[added] Tests for the newly changed/added features.
[changed] Tests no longer use flushdb() - all test models/indexes/etc. are
prefixed with RomTest, and we find/delete such keys before and after any
tests are run. Now anyone can reasonably run the test suite.
#--- 0.19

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
of a bug when using unique indexes, which is now fixed and has a
testcase.
#--- 0.18

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
and received an interim patch for a bug that could cause deleted
entities
to be resurrected on session.commit() or session.flush() . This has now
been fixed and a testcase has been added.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: RPQueue 0.22

[Josiah Carlson]

Hello everyone,

For those of you who didn't know, if you are interested in a Redis-backed
time and/or fifo-queue with priorities, retries, etc., to be used with
Python, one exists and is mature: it's called RPQueue, and it seeks to
simplify your life of task execution. The recent changelog entries are
below my signature.

The package has been around for two years next month, and does exactly what
you expect it to do - no more, no less.

You can find the package at:
https://github.com/josiahcarlson/rpqueue/
https://pypi.python.org/pypi/rpqueue

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.22

[fixed] setup.py-based installations. Ran into the bug myself :/
#--- 0.21
 
[changed] where available, rpqueue will now use Lua to move delayed tasks
from
the time-based priority queue to fifo queues. This should reduce
overhead
in all cases, and should drastically improve performance for those that
use large numbers of delayed tasks.
[changed] the version number is now PEP 386/440 compliant.
[added] this changelog that will document updates/changes/improvements in an
easily referenced location.
[fixed] thanks to https://github.com/dmaust , rpqueue added a missing
'redis'
requirement.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: rom 0.22 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

As time progresses, so does my Redis object mapper.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

The changelog for recent releases can be seen below my signature.

You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

And docs can be found at:
http://pythonhosted.org/rom/

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

#--- 0.22

[fixed] size estimation for intersection ordering when filtering has now
been
fixed, thank you to https://github.com/MickeyKim for the report and the
change (should improve performance).
[fixed] an issue with some types when trying to update attributes has now
been
fixed, thank you to https://github.com/denisvolokh for the report.
[changed] improved performance for simple numeric range queries of the form
Model.get_by(attr=value) or Model.get_by(attr=(min, max)) by roughly a
factor of 60x or better in some cases. Thank you to
https://github.com/MickeyKim for the report on poor performance.
#--- 0.21

[fixed] upload for rom 0.20 was missing new columns.py, now fixed
#--- 0.20

[changed] Added exception when performing .all(), .execute(), or .count() on
query objects that have had no filters or attribute ordering provided.
This addresses issue #12.
[changed] Moved column definitions to their own module, shouldn't affect any
normal uses of rom.
[added] For users of Redis 2.6 and later, there is a beta Lua-enabled
writing
option that allows for multiple unique columns on models. In some cases,
this may improve performance when writing many entities very quickly.
[added] The ability to reload an entity from Redis, optionally discarding
any
modifications to the object itself. Check out the documentation for
Model.refresh(), Session.refresh(), and Session.refresh_all()
[added] Tests for the newly changed/added features.
[changed] Tests no longer use flushdb() - all test models/indexes/etc. are
prefixed with RomTest, and we find/delete such keys before and after any
tests are run. Now anyone can reasonably run the test suite.
#--- 0.19

[fixed] Thanks to a bug report by https://github.com/MickeyKim , was
notified
of a bug when using unique indexes, which is now fixed and has a
testcase.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: rom 0.10 - Redis object mapper for Python

[Josiah Carlson]

Hey everyone,

I know, it's been several years since I announced anything on these lists,
but I suspect that some of you may have uses for my new package, so here
you go.

The "rom" package is a Redis object mapper for Python. It sports an
interface similar to Django's ORM, SQLAlchemy + Elixir, or Appengine's
datastore.

This is the initial release, so there may be some rough edges. I've
included the basic intro for the package below. You can find the package at:
https://www.github.com/josiahcarlson/rom
https://pypi.python.org/pypi/rom

Please CC me on any replies if you have any questions or comments.

Thank you,
 - Josiah

What's new?
==

Everything

What


Rom is a package whose purpose is to offer active-record style data modeling
within Redis from Python, similar to the semantics of Django ORM,
SQLAlchemy +
Elixir, Google's Appengine datastore, and others.

Why
===

I was building a personal project, wanted to use Redis to store some of my
data, but didn't want to hack it poorly. I looked at the existing Redis
object
mappers available in Python, but didn't like the features and functionality
offered.

What is available
=

Data types:

* Strings, ints, floats, decimals
* Json columns (for nested structures)
* OneToMany and ManyToOne columns (for model references)

Indexes:

* Numeric range fetches, searches, and ordering
* Full-word text search (find me entries with col X having words A and B)

Other features:

* Per-thread entity cache (to minimize round-trips, easy saving of all
  entities)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [wxPython-users] Re:

[Josiah Carlson]

Gabriel Genellina <[EMAIL PROTECTED]> wrote:
> At Tuesday 24/10/2006 23:25, Michael S wrote:
> 
> >I extended part of my program in C, since that part
> >was too involved for Python. Now when I import the
> >module I created and call its functions, I am trying
> >to feedback some information bac to my wxPython
> >program. The function runs for a while and I wanted to
> >update a progress bar, but when I call the function of
> >my new module, wxPython doesn't respond when wx.Timer
> >is supposed to fire.
> 
> Does your extension run on a separate thread? I guess not. So the 
> wxPython code does not have a chance to run.
> Do it the other way: from your C code, call a Python function which 
> in turn updates the progress bar (and keep it simple!)

Unless the GUI thread has a chance to take passes through the wxPython
mainloop, the progress bar, etc., may not actually be updated on some
platforms.

Mr. Sherman would likely be better off creating a new thread which
handles processing, which could update a C global variable, which the
GUI thread can poll to update its progress bar, etc.  Alternatively, if
the processing can be broken down into smaller pieces, the secondary
thread can call the pieces and post events back to the GUI thread to
update the progress bar (via explicit wx.PostEvent or implicit
wx.CallAfter).


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [wxPython-users] Re:

[Josiah Carlson]

Michael S <[EMAIL PROTECTED]> wrote:
> If I create a threading.Thread derived class, and run
> the wxPython code in the main thread; using a lock,
> will I be able to suspend the Thread, even though it
> will run the C code, which is essentially a while loop
> looping through millions of lines of a text file?

If all your C code is doing is looping through lines in a text file, you
can rewrite it fairly reasonably in Python via 'for line in open(fn):'.

In terms of locks, you have to use a lock that can be acquired by either
thread (GUI or C), but that won't necessarily give you what you want;
the GUI thread in wxPython may still not like being called directly.

If you want the ability to easily restart your C code, you may want to
consider at least looking at the following page on coroutines in C:
http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html .  It would
allow you to do all of your GUI manipulation/updating in Python without
having to worry about locks or calling Python code from C.

 - Josiah


> --- Josiah Carlson <[EMAIL PROTECTED]> wrote:
> 
> > 
> > Gabriel Genellina <[EMAIL PROTECTED]> wrote:
> > > At Tuesday 24/10/2006 23:25, Michael S wrote:
> > > 
> > > >I extended part of my program in C, since that
> > part
> > > >was too involved for Python. Now when I import
> > the
> > > >module I created and call its functions, I am
> > trying
> > > >to feedback some information bac to my wxPython
> > > >program. The function runs for a while and I
> > wanted to
> > > >update a progress bar, but when I call the
> > function of
> > > >my new module, wxPython doesn't respond when
> > wx.Timer
> > > >is supposed to fire.
> > > 
> > > Does your extension run on a separate thread? I
> > guess not. So the 
> > > wxPython code does not have a chance to run.
> > > Do it the other way: from your C code, call a
> > Python function which 
> > > in turn updates the progress bar (and keep it
> > simple!)
> > 
> > Unless the GUI thread has a chance to take passes
> > through the wxPython
> > mainloop, the progress bar, etc., may not actually
> > be updated on some
> > platforms.
> > 
> > Mr. Sherman would likely be better off creating a
> > new thread which
> > handles processing, which could update a C global
> > variable, which the
> > GUI thread can poll to update its progress bar, etc.
> >  Alternatively, if
> > the processing can be broken down into smaller
> > pieces, the secondary
> > thread can call the pieces and post events back to
> > the GUI thread to
> > update the progress bar (via explicit wx.PostEvent
> > or implicit
> > wx.CallAfter).
> > 
> > 
> >  - Josiah
> > 
> > 
> >
> -
> > To unsubscribe, e-mail:
> > [EMAIL PROTECTED]
> > For additional commands, e-mail:
> > [EMAIL PROTECTED]
> > 
> > 

-- 
http://mail.python.org/mailman/listinfo/python-list

where the hell is the run command of PyPe editor?

[Josiah Carlson]

daniel <[EMAIL PROTECTED]> wrote:
> I'm trying to use PyPe, but I just got so frustrated when attempting to
> run a script, this is the very first feature I would use for an editor,
> OMG. I browsed through every single menu item and gave up...
> 
> any help would be appreciated,  (the most silly question ever, sounds
> like a new computer user. ;--()

I don't normally troll python-list, so sorry if this gets to you late,
but to answer your question; there are sample macros in the
macros/samples directory with recent PyPE distributions which include
both "run current script" and "run selected code" example macros.

If you have questions about PyPE in the future, feel free to email me
directly, including PyPE in the subject will make sure it gets past my
spam filter.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] The baby and the bathwater (Re: Scoping, augmented assignment, 'fast locals' - conclusion)

[Josiah Carlson]

Boris Borcic <[EMAIL PROTECTED]> wrote:
> [this is bytes of an oversized put-all-into-it intervention. A possibly 
> expanded 
> version will be submitted on clp with local followup before a couple days]
> 
> Josiah Carlson wrote:
> 
> [BB]
>  >> I'd say a first step in convincing me I am wrong would be to show me 
> examples of
>  >> object methods of the standard library that are recursive, and cut out for
>  >> recursion.
> 
> [JC]
>  > Actually, I don't believe that is necessary. I've shown that you would
>  > get better performance with a non-recursive function and passing two
>  > arguments, than you would passing one argument with scoping tricks to
>  > get the second.
> 
> Assuming my aim was purely performance clearly stretches the hints I gave 
> that I 
> wasn't unconcerned by it. Taking ground on this to unwarrantedly suppose that 
> my 
> methodology was deficient, rectify it, and then dictate the "necessary" 
> conclusions I find... er, preposterous seems balanced, "thought police" not 
> totally unwarranted.

In my initial and later readings of your messages, I read 2 reasons why
you were using a closure rather than some other method to perform this
particular operation:
1. Minimal code change.
2. It was faster than using an object with a particular interface.

If there were other reasons, then I obviously missed them.

However, since no one else has responded substantively to your postings,
other than Terry and Guido's 'not gonna change', I can only presume that
others have found that your desired result (augmented assignments for
names not previously existing in the current scope, but existing in a
parent lexical scope, be performed on the object bound to a parent scope
name), is undesireable for Python 2.5 .  The fact that no one has
suggested that you write a PEP, or start the discussion _after_ 2.5
comes out, is a sign to me that the feature is generally undesireable.
Without proof to the contrary (a group of core Python developers coming
out in support of the feature), I can only go by my own opinion: I don't
believe the utility of this particular feature is worthwhile enough to
change how Python works, considering the possibility for execution
errors.

[snip]

> This said, first : I deny you have or ever had real ground to refuse 
> legitimacy, 
> motivation, efficiency or methodological soundness to my approach.

In a portion of your message that I've cut out, you described how the
goal of your current programming project was to write a sudoku solver in
fewer than 60 lines and that ran in fewer than 10ms for arbitrary
problems. Great, wonderful, I wish you luck; I will not question or
"refuse legitimacy, motivation, efficiency or methodological soundness"
of your approach when using the Python programming language.

However, you are offering a series of reasons as to why the Python
programming language should be changed.  I'm questioning your reasoning.
You may find it insufficient, misinformed, prejudiced, nonsense,
mean-spirited, or even that I'm a loony who should be in your killfile.
My purpose here isn't to convince _you_ that you are wrong about this
feature, as arguing via email has never been my strong suit, but instead
to offer at least one valid reason why your change could or would hurt
some Python users, and/or the effectiveness of their code.

I will mention that the lack of _any_ public support in python-dev
likely says far more about the unliklihood of acceptance than anything
you or I have probably typed in the last few days.


[snip]
> [JC]
>  > when it was an
>  > actual error (a not uncommon case),
> 
> I guess your insistence on the rightful methodology entitles me to enquire in 
> turn on your choice of tool to establish the relative frequency of said case ?

My measurement is quite subjective.  Have I ever made that mistake, or
do I know of anyone who has made that mistake? In this case, I have made
the mistake I mention, both in personal code, as well as code I was
writing as I was tutoring.  I'm not aware of significant problems in
user code, usually because the moment the code is run, the programmer
fixes it.

Here are two questions for you:

1. Do you have any proof showing that the error I claim exists (not
uncommonly was my description), is in fact so rare, that I am
essentially the only one who has made the mistake?

2. Do lexically nested augmented assignments happen so often as to make
this problem a "must solve" for any Python revision?


> [JC]
>  > which is a bit of a no-no,
> 
> What I would have believed a /bit/ of a no-no for Python, is to turn away 
> from 
> the "consenting adults" spirit to go the way of so many other programming 
> languages.
> 
> Picture

Re: a good programming text editor (not IDE)

[josiah . carlson]

John Salerno wrote:
> I know there's a request for a good IDE at least once a week on the ng,
> but hopefully this question is a little different. I'm looking for
> suggestions for a good cross-platform text editor (which the features
> for coding, such as syntax highlighting, etc.) but not a full IDE with
> all the fancy jazz (GUI developer, UML diagrams, etc.).
>
> Ideally, it would be something I could even put on a flash drive and
> move from computer to computer, but this isn't necessary. Just something
> I can immediately use in either Windows or Linux (or Mac, if necessary).
[snip]
> Thanks for any suggestions, and again I'm sorry if this feels like the
> same question as usual (it's just that in my case, I'm not looking for
> something like SPE, Komodo, Eric3, etc. right now).

I was taking a peek at c.l.py to check for replies in another thread
and couldn't help notice your asking about editors.  Please pardon the
personal pimping, but have you looked at PyPE (pype.sf.net)?  It
doesn't do everything, isn't geared towards every language, but as far
as I know, it is fairly cross-platform (there seems to be an
outstanding bug on ubuntu 6.06, but I need to get an installation to
check it out), and can offer a fairly no-frills interface for writing
software.  The next release will include support for working from a
removable device (I plan on releasing before the end of the month), but
may not have an ubuntu fix (I may not have time to get one running).

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: a good programming text editor (not IDE)

[josiah . carlson]

Walter Dörwald wrote:
> I tried it out and the first problem I noticed is that on Windows
> opening a file from a Samba drive doesn't seem to work, as PyPE converts
> the filename to lowercase.

...Samba is tricky, and I hadn't thought of it before.  Normal Windows
is case-insensitive but case-preserving, and PyPE uses Python's
os.path.normcase(os.path.normpath(os.path.realpath(...))) to do
path/file normalization mapping for open files (keeping the
non-case-normalized filename), so that you can't really make the
mistake of opening the same file twice.  This normally works on
platforms with consistant case handling, but if you are using Samba
running on any non-OSX *nix (whose filename case handling is not like
other *nixes) from Windows, there gets to be a bit of a problem as your
*nix case handling is different from Windows.

I have an idea of what needs to be done, and the fix for it will be in
PyPE 2.4.1, which I plan on releasing by June 30th.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Python-Dev] The baby and the bathwater (Re: Scoping, augmented assignment, 'fast locals' - conclusion)

[Josiah Carlson]

Before I get started with my reply, I would just like to point out that
aside from my initial 'I see your using a closure for this problem as a
design bug', I have tried my best to be as reasonable as I can, not once
attacking your character or abilities. However, twice now you have
expressed various insults towards me in order to somehow discredit my
statements.

At this point I am no longer interested in this conversation, a
sentiment I expect you share (perhaps along with those in
comp.lang.python). If you feel the need to continue to express
discontent about me publically, have at it. Heck, I'll even start you
out: Josiah Carlson's nose is too big. Further self-deprecation I will
mark with '*'.

It is obvious to both of us that we are misunderstanding each other on
various levels [1], so perhaps cutting of communication is better for
the both of us, and if anyone else has anything to say about the topic,
they can reply to either of us.

Boris Borcic <[EMAIL PROTECTED]> wrote:
> Josiah Carlson wrote:
> > In my initial and later readings of your messages, I read 2 reasons why
> > you were using a closure rather than some other method to perform this
> > particular operation:
> 
> That wording already doesn't apply. I told you I was working *not* on a 
> program 
> but on a *space* of programs of identical function, as of manner of probing 
> recent features and modules of Python that I manage to apply to it. Besides 
> learning, my purpose is also aesthetic.

My reading of your previous message stated that while you were looking
through various algorithms and codebases, your purpose seemed quite
singular, and I'll quote your quoting of the requirement:
""universal sudoku solver in +-pure python, ~10ms/problem, and ~60 LOCS
total"."

I must be such an idiot* that I failed to understand that you were, in
fact, working on modifying "a *space* of programs of identical function". 
I'm sorry about that.


As for code aesthetics, I have little to no sympathy for relatively
minor changes.  Decorators, yeah, that was a significant change in
aesthetics (and usefulness), and when such was pointed out, I went from
a -1 to a +1. But x[0] ?= y becoming x ?= y (where ? is one of ^&|+-*/),
or as provided by Noam Raphael recently, x[()] becoming x[], I can't get
behind for purely aesthetic reasons, and see limited practical value to
them.


> > However, since no one else has responded substantively to your postings,
> > other than Terry and Guido's 'not gonna change',
> 
> That's "Terry's 'take the fairy tale'  and Guido's 'not gonna 
> change'", 
> to be exact.

I had used the single quote as a mechanism of paraphrase rather than
double quotes as exact quotation.  You obviously understood this, but I
would point out that while Terry said, "this will not change for 2.x",
Python 3.x development has started already, so it's not so much a "fairy
tale'.


> > I can only presume that
> > others have found that your desired result (augmented assignments for
> > names not previously existing in the current scope, but existing in a
> > parent lexical scope, be performed on the object bound to a parent scope
> > name), is undesireable for Python 2.5
> 
> This is again misrepresentation. I definitely did *not* come to pydev to ask 
> for 
> a change, I merely intervened in the middle on an ongoing thread at the point 
> it 
> was being said that, in effect, I did not exist. Thereupon you intervened 
> with 
> no other purpose than to show, in effect, that I did not exist, forcing me 
> into 
> an endless chain of rectifications. Now you are telling me I want it included 
> in 
> the 2.x series. That's confusing the citizen for the lobbyist.

I will point you back to Terry's posts:
http://mail.python.org/pipermail/python-dev/2006-June/066037.html
http://mail.python.org/pipermail/python-dev/2006-June/066057.html

Read them a few times.  Ok, now read my post:
http://mail.python.org/pipermail/python-dev/2006-June/065990.html

If I understood Terry correctly (and I hope I understood myself), at no
point in any of those messages did anyone state that nested augmented
assignments wouldn't be useful, or that there weren't any use-cases.
Terry stated how augmented assignments managed to work in alpha 2, why
it was a bug generally, and why that patch was being fixed/reverted.

I tried to express that I felt your use of a closure was a poor design
decision, because I had understood your code to offer an external
interface, where it actually offered an internal interface.


Over the course of the subsequent emails, I have attempted to,
essentially, make similar points as Terry, though obviously Terry was
far more conci

sys.settrace() and Python 2.3

[Josiah Carlson]

According to my reading of Python 2.3 docs, the call to goo() at the end
should exit with a KeyboardInterrupt...

import sys

def goo():
while 1:
pass

count = [100]
def foo(frame, event, arg):
count[0] -= 1
if not count[0]:
raise KeyboardInterrupt
return foo

sys.settrace(foo)

goo()

In Python 2.3, the above call to goo() doesn't seem to ever return.  It
exits with a KeyboardInterrupt in 2.4, as expected.  Does anyone have an
idea why this is the case?  I've checked the release notes for both 2.3
and 2.4 and found no discussion of trace functions in them or
sourceforge (I could be entering in the wrong terms, of course).

Any pointers as to why there is a difference would be appreciated. 
Thank you,
 - Josiah

(I'm not subscribed, but I'll be checking for replies, and ccing me on
any replies would be appreciated)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: compiling python (or ironpython) to .exe or .dll for or not for .NET

[josiah . carlson]

per9000 wrote:
> Update:
>
> I have found a holy handgrenade.
>
> I found something that seems to work fine for me - I've only tried it
> for 5 minutes but seems to work very smoothly. Open-source, Mozilla
> license, you know the drill...
>
> http://www.py2exe.org/
>
> py2exe (the name makes me fall in love) creates some dll's for me
> containing python I guess.

Your code exists as a .pyc file inside of the library.zip .  People who
still want access to it will be able to use the free or commercial
versions of decompyle to get your code back.  If you want to make it
more difficult, you can always use pyrex, but that will require a
compiler.

 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [wxPython-users] Problem in threading

[Josiah Carlson]

My response is at the end.

Sbaush <[EMAIL PROTECTED]> wrote:
> Hi all.
> I've a problem with thread in python.
> My applications has a GUI that has some button.
> I've a MVC-like architecture.
> If i click on a button there is this effect:
> 
> view.button_1_delete.Bind(EVT_BUTTON,self.OnDelete)
> view.button_1_list.Bind(EVT_BUTTON,self.OnList)
> 
> def OnDelete(self, evt):
> self.presenter.updateModel("delete")
> def OnList(self, evt):
> self.presenter.updateModel("list")
> 
> Probably you image that the boss of this program is updateModel function.
> this function has this code:
> 
> def updateModel(self,Type):
> iptType=Type
> self.readGUI(iptType)  # This function takes the request info from
> GUI
> packet=self.ipt_writeXML(iptType) # Put info in XML string
> self.sender.send(packet) #Send the XML string
> (xmlresponse,hostfrom)=self.receiver.receive() # Receive a XML
> string response from agent and the IP of agent
> self.iptResponse.run(xmlresponse,hostfrom) # view response in a
> separate view.
> 
> Now it works great, but there is a big problem: my app wait for response
> before allow user to do another request.
> I thought that with threads it could works perfectly.
> I tried to apply threads but without any good result.
> Have you idea how can i do it and how can i apply thread to it?

import wx
import wx.lib.newevent
import threading

ResponseEvent, EVT_RESPONSE = wx.lib.newevent.NewEvent()

...
def __init__(self, ...):
...
self.Bind(EVT_RESPONSE, self.GotResponse, self)
...

def _updateModel(self, Type):
iptType=Type
self.readGUI(iptType)
packet=self.ipt_writeXML(iptType)
self.sender.send(packet)
(xmlresponse,hostfrom)=self.receiver.receive()
wx.PostEvent(self,
 ResponseEvent(xml=xmlresponse,
   host=hostfrom))

def updateModel(self, Type):
threading.Thread(target=self._updateModel,
  args=(Type,)).start()

def GotResponse(self, evt):
self.iptResponse.run(evt.xml, xml.host)
...


 - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: interesting take on python OO

[Josiah Carlson]

7stud wrote:
> http://www.evolt.org/article/OO_programming_the_Python_way/18/449/
> 
> lol

I would have to agree.  I could spend time talking about how bad that 
article was, but instead I will offer useful advice for the new Python 
programmer.

For new users of Python and/or object-oriented programming, stick with 
the Python Tutorial included with the Python distribution, and/or check 
out http://www.ibiblio.org/obp/thinkCS/python/english/ .  There are some 
pieces that were obviously translated directly from Java, but it has a 
lot of good stuff.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python rocks

[Josiah Carlson]

Mark Carter wrote:
> Not that I'm particularly knowledgeable about language design issues, 
> but maybe closures and slightly different scoping rules would be nice.

Python has had closures for years.  What kind of scoping did you desire?

 > A
> pitfall of Python is knowing whether an operation is destructive or not. 

If it returns None, it probably changes the content of an object.

> I guess if it was a purely functional language, that particular question 
> wouldn't arise.

You can certainly do your programming in a functional style with Python. 
  List comprehensions and generator expressions are very good for this 
kind of thing.

 > Oh, and when I read a line, I'd like it to get rid
> of the trailing line-ending characters.

Refuse the temptation to guess.  As many people want to keep the 
newlines as not (it disambiguates the case of end of file and blank 
line), and it is a simple thing to deal with: line.rstrip('\r\n') .


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Non-blocking subprocess call

[Josiah Carlson]

Kevin Walzer wrote:
> I'm currently using os.popen in an application of mine, which calls for 
> non-blocking data, in this fashion:
> 
>  self.file = os.popen('echo %s | sudo -S /sw/bin/fink -y install %s' % 
> (self.passtext, self.packagename), 'r', os.O_NONBLOCK)
> 
> What is the equivalent/comparable call in the new subprocess module? I 
> can't find any reference in the Python docs to non-blocking streams.

If you are talking about non-blocking reading and writing from the 
subprocess, there isn't natively.  But there is a cookbook entry:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python rocks

[Josiah Carlson]

Mark Carter wrote:
> Josiah Carlson wrote:
>>  What kind of scoping did you desire?
> 
> Well, I had in mind so that if you defined a function, but wanted to 
> access a global var, that you didn't have to use the global keyword. Not 
> much of a biggie, I guess.

You already get read access to globals without using the global keyword. 
  To replace what the global name points to you need to use the global 
keyword, but if it is mutable (list, dictionary, etc.), you can append 
to it, delete items from it, etc., all without keywords.


> I guess that about wraps it up, then. Python pretty much has it all. 
> Except for macros - which is not necessarily a dealbreaker, anyway.

Yeah, the lack of macros sometimes bothers me, then I remember that if I 
*really* need them, there's always Logix and bytecodehacks.inline:
http://bytecodehacks.sourceforge.net/bch-docs/bch/module-bytecodehacks.inline.html

I don't know how much work it would take to get the bytecodehacks 
working in a modern Python, but it may be an interesting exercise.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: another thread on Python threading

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> I've recently been working on an application[1] which does quite a bit
> of searching through large data structures and string matching, and I
> was thinking that it would help to put some of this CPU-intensive work
> in another thread, but of course this won't work because of Python's
> GIL.

If you are doing string searching, implement the algorithm in C, and 
call out to the C (remembering to release the GIL).

> There's a lot of past discussion on this, and I want to bring it up
> again because with the work on Python 3000, I think it is worth trying
> to take a look at what can be done to address portions of the problem
> through language changes.

Not going to happen.  All Python 3000 PEPs had a due-date at least a 
month ago (possibly even 2), so you are too late to get *any* 
substantial change in.

> I remember reading (though I can't find it now) one person's attempt
> at true multithreaded programming involved adding a mutex to all
> object access.  The obvious question though is - why don't other true
> multithreaded languages like Java need to lock an object when making
> changes?

 From what I understand, the Java runtime uses fine-grained locking on 
all objects.  You just don't notice it because you don't need to write 
the acquire()/release() calls.  It is done for you.  (in a similar 
fashion to Python's GIL acquisition/release when switching threads) 
They also have a nice little decorator-like thingy (I'm not a Java guy, 
so I don't know the name exactly) called 'synchronize', which locks and 
unlocks the object when accessing it through a method.


  - Josiah



> == Why hasn't __slots__ been successful? ==
> 
> I very rarely see Python code use __slots__.  I think there are
> several reasons for this.  The first is that a lot of programs don't
> need to optimize on this level.  The second is that it's annoying to
> use, because it means you have to type your member variables *another*
> time (in addition to __init__ for example), which feels very un-
> Pythonic.
> 
> == Defining object attributes ==
> 
> In my Python code, one restriction I try to follow is to set all the
> attributes I use for an object in __init__.   You could do this as
> class member variables, but often I want to set them in __init__
> anyways from constructor arguments, so "defining" them in __init__
> means I only type them once, not twice.
> 
> One random idea is to for Python 3000, make the equivalent of
> __slots__ the default, *but* instead gather
> the set of attributes from all member variables set in __init__.  For
> example, if I write:
> 
> class Foo(object):
>   def __init__(self, bar=None):
> self.__baz = 20
> if bar:
>   self.__bar = bar
> else:
>   self.__bar = time.time()
> 
> f = Foo()
> f.otherattr = 40  # this would be an error!  Can't add random
> attributes not defined in __init__
> 
> I would argue that the current Python default of supporting adding
> random attributes is almost never what you really want.  If you *do*
> want to set random attributes, you almost certainly want to be using a
> dictionary or a subclass of one, not an object.  What's nice about the
> current Python is that you don't need to redundantly type things, and
> we should preserve that while still allowing more efficient
> implementation strategies.
> 
> = Limited threading =
> 
> Now, I realize there are a ton of other things the GIL protects other
> than object dictionaries; with true threading you would have to touch
> the importer, the garbage collector, verify all the C extension
> modules, etc.  Obviously non-trivial.  What if as an initial push
> towards real threading, Python had support for "restricted threads".
> Essentially, restricted threads would be limited to a subset of the
> standard library that had been verified for thread safety, would not
> be able to import new modules, etc.
> 
> Something like this:
> 
> def datasearcher(list, queue):
>   for item in list:
> if item.startswith('foo'):
>   queue.put(item)
>   queue.done()
> 
> vals = ['foo', 'bar']
> queue = queue.Queue()
> threading.start_restricted_thread(datasearcher, vals, queue)
> def print_item(item):
>   print item
> queue.set_callback(print_item)
> 
> Making up some API above I know, but the point here is "datasearcher"
> could pretty easily run in a true thread and touch very little of the
> interpreter; only support for atomic reference counting and a
> concurrent garbage collector would be needed.
> 
> Thoughts?
> 
> [1] http://submind.verbum.org/hotwire/wiki
> 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: magic names in python

[Josiah Carlson]

per9000 wrote:
> So my questions:
>  * is there a comprehensive list of magic names in python (so far i
> know of __init__ and __repr__)?
>  * are these lists complete or can magic names be added over time (to
> the python "core")?
>  * are magic names the same in different python versions?

I don't believe that there is a full list of all __magic__ methods.  The 
operator module has a fairly extensive listing of functions that call 
such methods, but I know that some have been left out.

Among those that I remember off the top of my head while typing this 
message...

__init__
__new__
__str__
__repr__
__len__
__nonzero__
__hash__
__cmp__ (__eq__, __ne__, __lt__, __gt__, __le__, __ge__)
__getattr__
__setattr__
__delattr__
__getitem__
__setitem__
__delitem__
__iter__
__neg__
__not__

There's also the not-magic named, but still somewhat magic .next() 
method on iterators/generators.

  - Josiah

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: another thread on Python threading

[Josiah Carlson]

sturlamolden wrote:
> On Jun 4, 3:10 am, Josiah Carlson <[EMAIL PROTECTED]>
> wrote:
>>  From what I understand, the Java runtime uses fine-grained locking on
>> all objects.  You just don't notice it because you don't need to write
>> the acquire()/release() calls.  It is done for you.  (in a similar
>> fashion to Python's GIL acquisition/release when switching threads)
> 
> The problem is CPython's reference counting. Access to reference
> counts must be synchronized.
> Java, IronPython and Jython uses another scheme for the garbage
> collector and do not need a GIL.

There was a discussion regarding this in the python-ideas list recently. 
  You *can* attach a lock to every object, and use fine-grained locking 
to handle refcounts.  Alternatively, you can use platform-specific 
atomic increments and decrements, or even a secondary 'owner thread' 
refcount that doesn't need to be locked by 1 thread at a time.

It turns out that atomic updates are slow, and I wasn't able to get any 
sort of productive results using 'owner threads' (seemed generally 
negative, and was certainly more work to make happen).  I don't believe 
anyone bothered to test fine-grained locking on objects.

However, locking isn't just for refcounts, it's to make sure that thread 
A isn't mangling your object while thread B is traversing it.  With 
object locking (course via the GIL, or fine via object-specific locks), 
you get the same guarantees, with the problem being that fine-grained 
locking is about a bazillion times more difficult to verify the lack of 
deadlocks than a GIL-based approach.


> Changing CPython's garbage collection from reference counting to a
> generational GC will be a major undertaking. There are also pros and
> cons to using reference counts instead of 'modern' garbage collectors.
> For example, unless there are cyclic references, one can always know
> when an object is garbage collected. One also avoids periodic delays
> when garbage are collected, and memory use can be more modest then a
> lot of small temporary objects are being used.

It was done a while ago.  The results?  On a single-processor machine, 
Python code ran like 1/4-1/3 the speed of the original runtime.  When 
using 4+ processors, there were some gains in threaded code, but not 
substantial at that point.


> There are a number of different options for exploiting multiple CPUs
> from CPython, including:

My current favorite is the processing package (available from the Python 
  cheeseshop).  You get much of the same API as threading, only you are 
using processes instead.  It works on Windows, OSX, and *nix.


> def synchronized(fun):
>from threading import RLock
>rl = RLock()
>def decorator(*args,**kwargs):
>   with rl:
>  retv = fun(*args,**kwargs)
>   return retv
>return decorator
> 
> It is not possible to define a 'synchronized' block though, as Python
> do not have Lisp macros :(

Except that you just used the precise mechanism necessary to get a 
synchronized block in Python:

 lock = threading.Lock()

 with lock:
 #synchronized block!
 pass


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python, Dutch, English, Chinese, Japanese, etc.

[Josiah Carlson]

Steve Howell wrote:
> I don't predict a huge upswing in Slavic-writing
> Python programmers after PEP 3131, even among
> children.

Are you predicting a sharp upswing in Chinese-writing (or any language) 
Python programmers after PEP 3131 among children?  If so, why certain 
groups of children and not others?

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: another thread on Python threading

[Josiah Carlson]

sturlamolden wrote:
> On Jun 4, 10:11 pm, Josiah Carlson <[EMAIL PROTECTED]>
> wrote:
> 
>>  lock = threading.Lock()
>>
>>  with lock:
>>  #synchronized block!
>>  pass
> 
> True, except that the lock has to be shared among the threads. This
> explicit initiation of an reentrant lock is avoided in a Java
> synchronized block.

You toss the lock creation in the global namespace of the module for 
which you would like to synchronize access.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: another thread on Python threading

[Josiah Carlson]

sturlamolden wrote:
> On Jun 4, 10:11 pm, Josiah Carlson <[EMAIL PROTECTED]>
> wrote:
> 
>> However, locking isn't just for refcounts, it's to make sure that thread
>> A isn't mangling your object while thread B is traversing it.
> 
> 
>> With
>> object locking (course via the GIL, or fine via object-specific locks),
>> you get the same guarantees, with the problem being that fine-grained
>> locking is about a bazillion times more difficult to verify the lack of
>> deadlocks than a GIL-based approach.
> 
> I think this is just as much a question of what the runtime should
> guarantee. One don't need a guarantee that two threads are not
> mangling the same object simultaneously. Instead, the runtime could
> leave it to the programmer to use explicit locks on the object or
> synchronized blocks to guarantee this for himself.

Why?  Right now we have a language where the only thing that doing silly 
things with threads can get you now is perhaps a deadlock, perhaps 
incorrect execution, or maybe some data corruption if you are working 
with files.  If we forced all thread users to synchronize everything 
themselves, we get an uglier language, and incorrectly written code 
could potentially cause crashes (though all of the earlier drawbacks 
still apply).  In the "safety vs. speed" or "easy vs. fast" arenas, 
Python has already chosen safe and easy rather than fast.  I doubt it is 
going to change any time soon.


>> It was done a while ago.  The results?  On a single-processor machine,
>> Python code ran like 1/4-1/3 the speed of the original runtime.  When
>> using 4+ processors, there were some gains in threaded code, but not
>> substantial at that point.
> 
> I am not surprised. Reference counts are quite efficient, contrary to
> common belief. The problem with reference counts is cyclic references
> involving objects that define a __del__ method. As these objects are
> not eligible for cyclic garbage collection, this can produce resource
> leaks.

There was a discussion about removing __del__ within the last couple 
weeks.  I didn't pay much attention to it (having learned never to use 
__del__), but I believe it involved some sort of weakref-based cleanup.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: lists - append - unique and sorted

[Josiah Carlson]

Neil Cerutti wrote:
> On 2007-06-06, rhXX <[EMAIL PROTECTED]> wrote:
>> and/or
>>
>> - SORTED - INSERT in the correct place using some criteria?
> 
> Consult the Python Docs about the heapq module.

Heaps (as produced by heapq) are not sorted.  This will not produce 
correct results unless one then pops everything and de-dupes the output.

As Diez has already said, 'use the bisect module' .

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: who know?

[Josiah Carlson]

Michel Claveau wrote:
> Hi!
> 
> This image show IronPython.
> But... what is it?
> 
> Link :
> http://msdn2.microsoft.com/en-us/vstudio/bb510103.vss_IronPython_large.jpg

What is generally referred to as "Python", available from python.org, is 
really the 'CPython' runtime and associated libraries.  It implements 
the Python runtime using C.  There is also Jython, which uses a Python 
-> Java compiler on roughly Python 2.2 compatible source.  There is also 
PyPy, which implements the Python runtime in Python with various 
backends (LLVM -> GCC, etc.).  Finally, IronPython is an implementation 
of Python using Microsoft's .NET common language runtime (the CLR), 
which allows for Python to gain access to everything available in .NET .


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How Can I Increase the Speed of a Large Number of Date Conversions

[Josiah Carlson]

Some Other Guy wrote:
> vdicarlo wrote:
>> I am a programming amateur and a Python newbie who needs to convert
>> about 100,000,000 strings of the form "1999-12-30" into ordinal dates
>> for sorting, comparison, and calculations. Though my script does a ton
>> of heavy calculational lifting (for which numpy and psyco are a
>> blessing) besides converting dates, it still seems to like to linger
>> in the datetime and time libraries.  (Maybe there's a hot module in
>> there with a cute little function and an impressive set of
>> attributes.)
> ...
>> dateTuple = time.strptime("2005-12-19", '%Y-%m-%d')
>> dateTuple = dateTuple[:3]
>> date = datetime.date(dateTuple[0], dateTuple[1],
>> dateTuple[2])
>> ratingDateOrd = date.toordinal()
> 
> There's nothing terribly wrong with that, although strptime() is overkill
> if you already know the date format.  You could get the date like this:
> 
>date = apply(datetime.date, map(int, "2005-12-19".split('-')))
> 
> But, more importantly... 100,000,000 individual dates would cover 274000
> years!  Do you really need that much??  You could just precompute a
> dictionary that maps a date string to the ordinal for the last 50 years
> or so. That's only 18250 entries, and can be computed in less than a second.
> Lookups after that will be near instantaneous:
> 
> 
>  import datetime
> 
>  days = 365*50 # about 50 years worth
>  dateToOrd = {} # dict. of date string to ordinal
...

Then there's the argument of "why bother using real dates?"  I mean, all 
that is necessary is a mapping of date -> number for sorting.  Who needs 
accuracy?

for date in inp:
 y, m, d = map(int, date.split('-'))
 ordinal = (y-1990)*372 + (m-1)*31 + d-1

Depending on the allowable range of years, one could perhaps adjust the 
1990 up, and get the range of date ordinals down to about 12 bits (if 
one packs netflix data properly, you can get everything in memory). 
With a bit of psyco, the above is pretty speedy.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python's "only one way to do it" philosophy isn't good?

[Josiah Carlson]

James Stroud wrote:
> Terry Reedy wrote:
>> In Python, you have a choice of recursion (normal or tail)
> 
> Please explain this. I remember reading on this newsgroup that an 
> advantage of ruby (wrt python) is that ruby has tail recursion, implying 
> that python does not. Does python have fully optimized tail recursion as 
> described in the tail recursion Wikipedia entry? Under what 
> circumstances can one count on the python interpreter recognizing the 
> possibility for optimized tail recursion?

Note that Terry said that you could do normal or tail recursion, he 
didn't claim that either were optimized.  As for why tail calls are not 
optimized out, it was decided that being able to have the stack traces 
(with variable information, etc.) was more useful than offering tail 
call optimization (do what I say).

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Hooking exceptions outside of call stack

[Josiah Carlson]

Warren Stringer wrote:
> Here is what I would like to do:
> 
> #
> a = Tr3()  # implements domain specific language
> a.b = 1# this works, Tr3 overrides __getattr__
> a.__dict__['b'] = 2# just so you know that b is local
> a[b] = 3   # I want to resolve locally, but get:
> 
> Traceback (most recent call last): ...
> exec cmd in globals, locals ...
> NameError: global name 'b' is not defined
> #

Note that your a[b]=3 is the same as '__ = b;a[__]=3'  You get that 
exception because b is not a bound name in the namespace you are 
currently using.  In order to get what you want, you will either need to 
use a['b'] = 3, a.b = 3, or a method that I refuse to describe to you.

> This is intended for production code.

The reason I refuse to describe to you the method that could 'solve' 
your particular problem is because it would be very difficult to 
differentiate between what you *want* to happen, and actual errors, 
which would make production code *very* difficult to get right.

As an alternative to a['b'], you could use a[Z.b], for an object Z:

 class Z:
 def __getattr__(self, a):
 return a

 Z = Z()

Which will have much less potential for destroying the maintainability 
and testability of your production code than hooking any trace function.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python's "only one way to do it" philosophy isn't good?

[Josiah Carlson]

Alexander Schmolck wrote:
> Josiah Carlson <[EMAIL PROTECTED]> writes:
> 
>> James Stroud wrote:
>>> Terry Reedy wrote:
>>>> In Python, you have a choice of recursion (normal or tail)
>>> Please explain this. I remember reading on this newsgroup that an advantage
>>> of ruby (wrt python) is that ruby has tail recursion, implying that python
>>> does not. Does python have fully optimized tail recursion as described in
>>> the tail recursion Wikipedia entry? Under what circumstances can one count
>>> on the python interpreter recognizing the possibility for optimized tail
>>> recursion?
>> Note that Terry said that you could do normal or tail recursion, he didn't
>> claim that either were optimized.  
> 
> Well yeah, but without the implication how do the two words "or tail" add to
> the information content of the sentence?

Normal and tail recursion are different, based upon whether or not one 
can technically considered to be done with the stack frame.

def normal(arg):
 if arg == 1:
 return 1
 return arg * normal(arg-1)

def tail(arg, default=1):
 if arg == 1:
 return arg * default
 return tail(arg-1, default*arg)


>> As for why tail calls are not optimized out, it was decided that being able
>> to have the stack traces (with variable information, etc.) was more useful
>> than offering tail call optimization
> 
> I don't buy this. What's more important, making code not fail arbitrarily (and
> thus making approaches to certain problems feasible that otherwise wouldn't
> be) or making it a be a bit easier to debug code that will fail arbitrarily?
> Why not only do tail-call optimization in .pyo files and get the best of both
> worlds?

I didn't make the decisions, I'm just reporting what was decided upon.

Personally, I have never found the lack of tail call optimization an 
issue for two reasons.  The first is because I typically write such 
programs in an iterative fashion.  And generally for recursion for which 
tail call optimization doesn't work (the vast majority of recursive 
algorithms I use), I typically write the algorithm recursively first, 
verify its correctness, then convert it into an iterative version with 
explicit stack.  I find it is good practice, and would be necessary 
regardless of whether Python did tail call optimization or not.


>> (do what I say).
> 
> Where did you say run out of memory and fail? More importantly how do you say
> "don't run out of memory and fail"?

By virtue of Python's underlying implementation, Python "does what I 
say", it doesn't "do what I mean".  While I may not have explicitly 
stated "run out of stack space", the underlying implementation *has* 
limited stack space.  You are stating that when you write a tail 
recursive program, you want Python to optimize it away by destroying the 
stack frames.  And that's fine.  There are tail-call optimization 
decorators available, and if you dig into sourceforge, there should even 
be a full patch to make such things available in a previous Python.

However, Python is not Lisp and is not partially defined by infinite 
recursion (see sys.setrecursionlimit() ).  Instead, it is limited by the 
C call stack (in CPython), and makes guarantees regarding what will 
always be available during debugging (the only thing that optimization 
currently does in Python at present is to discard docstrings).  If you 
want to change what is available for debugging (to make things like tail 
call optimization possible), you are free to write and submit a PEP.

In the mean time, you may need to do some source conversion.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Hooking exceptions outside of call stack

[Josiah Carlson]

Warren Stringer wrote:
> Am still trying to hook a NameError exception and continue to run. After a
> few more hours of searching the web and pouring over Martelli's book, the
> closest I've come is:
[snip]
> Is there a way of intervening as `exec cmd in globals, locals` attempts to
> translate 'c' into an object? I thought that trapping a NameError might
> work. But now, I'm not so sure.

You can always muck around with the function's globals (if the operation 
is happening inside some function...)

 >>> def foo():
... print a
...
 >>> d = {'a':1}
 >>>
 >>> foo = type(foo)(foo.func_code, d, foo.func_name, foo.func_defaults,
foo.func_closure)
 >>> a
Traceback (most recent call last):
   File "", line 1, in ?
NameError: name 'a' is not defined
 >>> foo()
1
 >>>

With a little work, you can 'merge' your namespace-like object with the 
module globals that normally exist for a function.

However, I would say again, you shouldn't be doing this kind of thing in 
production code.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiple python interpreters within the same process

[Josiah Carlson]

André wrote:
> On Jun 9, 5:00 pm, "Marcin Kalicinski" <[EMAIL PROTECTED]> wrote:
>> How do I use multiple Python interpreters within the same process?
>>
>> I know there's a function Py_NewInterpreter. However, how do I use functions
>> like Py_RunString etc. with it? They don't take any arguments that would
>> tell on which interpreter to run the string...?
>>
>> Marcin
> 
> You may want to look at the code module 
> http://docs.python.org/lib/module-code.html

That's completely unrelated.

To answer Marcin's question, from what I understand, running multiple 
Python interpreters is not supported.  There are various issues with 
object sharing and refcounts, etc., and are unlikely to be fixed soon if 
ever.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python's "only one way to do it" philosophy isn't good?

[Josiah Carlson]

Steven D'Aprano wrote:
> On Sat, 09 Jun 2007 22:52:32 +, Josiah Carlson wrote:
> 
>> the only thing that optimization 
>> currently does in Python at present is to discard docstrings
> 
> Python, or at least CPython, does more optimizations than that. Aside from
> run-time optimizations like interned strings etc., there are a small
> number of compiler-time optimizations done.
> 
> Running Python with the -O (optimize) flag tells Python to ignore
> assert statements. Using -OO additionally removes docstrings.

Oh yeah, asserts.  I never run with -O, and typically don't use asserts, 
so having or not having either isn't a big deal for me.

> Regardless of the flag, in function (and class?) definitions like the
> following:
> 
> def function(args):
> "Doc string"
> x = 1 
> s = "this is a string constant"
> "and this string is treated as a comment"
> return s*x
> 
> The string-comment is ignored by the compiler just like "real" comments.
> (The same doesn't necessarily hold for other data types.)

I would guess it is because some other data types may have side-effects. 
  On the other hand, a peephole optimizer could be written to trim out 
unnecessary LOAD_CONST/POP_TOP pairs.

> Some dead code is also optimized away:

Obviously dead code removal happens regardless of optimization level in 
current Pythons.

> Lastly, in recent versions (starting with 2.5 I believe) Python includes a
> peephole optimizer that implements simple constant folding: 

Constant folding happens regardless of optimization level in current 
Pythons.


So really, assert and docstring removals.  Eh.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Hooking exceptions outside of call stack

[Josiah Carlson]

Warren Stringer wrote:
> Josiah Carlson wrote:
>>  >>> foo = type(foo)(foo.func_code, d, foo.func_name, foo.func_defaults,
>> foo.func_closure)
> 
> Wow! I've never seen that, before. Is there documentation for `type(n)(...)`
> somewhere? I did find a very useful "Decorator for Binding Constants, by
> Raymond Hettinger", that uses this technique.

type(obj) gets you the type of the object.  To discover what it takes to 
construct a function object, I just used...

 >>> help(type(foo))
Help on class function in module __builtin__:

class function(object)
  |  function(code, globals[, name[, argdefs[, closure]]])
  |
  |  Create a function object from a code object and a dictionary.
  |  The optional name string overrides the name from the code object.
  |  The optional argdefs tuple specifies the default argument values.
  |  The optional closure tuple supplies the bindings for free variables.

...


> The calls are made from embedded classes that are constructed on the fly:
> 
>   class a(object): ...
> class b(object): ...
>class c(object): ...
> 
> for `a.b[c]=1`, a.__getattr__('b') is called but 'c' need to be resolved as
> an object before a.b.__getitem__(c) is called. Could a.b.__init__ set
> globals so that 'c' gets resolved? Is this global namespace the same as the
> `globals()` namespace? 

I don't believe there is any confusion as to *why* you are getting the 
error.  It seems plain to everyone involved.  You are getting the error 
because c is not defined in the namespace in which you are executing 
'a.b[c] = 1'.  Unless you modify the namespace in which it is running...

 >>> class a(object):
... class b(object):
... class c(object):
... pass
...
 >>> def merged_namespace(*ns):
... try:
... __builtins__
... namespaces = ns + (globals(),__builtins__.__dict__)
... except NameError:
... namespaces = ns + (globals(),__builtin__)
... ns = {}
... for i in reversed(namespaces):
... ns.update(i)
... return ns
...
 >>> class m_level_namespace(dict):
... def __init__(self, *ns):
... try:
... __builtins__
... self.namespaces = ns + (globals(),__builtins__.__dict__)
... except NameError:
... self.namespaces = ns + (globals(),__builtin__)
... def __getitem__(self, key):
... for i in self.namespaces:
... print "..."
... if key in i:
... return i[key]
... raise NameError("Name %r not found"%(key,))
... def __setitem__(self, key, value):
... self.ns[0][key] = value
... def __delitem__(self, key):
... del self.ns[0][key]
...
 >>> def foo():
... print type(globals())
... a.b[c] = 1
...
 >>> ns = m_level_namespace(a.b.__dict__)
 >>>
 >>> foo1 = type(foo)(foo.func_code, ns, foo.func_name, 
foo.func_defaults, foo.func_closure)
 >>>
 >>> foo1()
Traceback (most recent call last):
   File "", line 1, in 
   File "", line 2, in foo
NameError: global name 'type' is not defined
 >>>
 >>> ns = merged_namespace(a.b.__dict__)
 >>>
 >>> foo2 = type(foo)(foo.func_code, ns, foo.func_name, 
foo.func_defaults, foo.func_closure)
 >>>
 >>> foo2()

Traceback (most recent call last):
   File "", line 1, in 
   File "", line 3, in foo
TypeError: 'type' object does not support item assignment
 >>>

Due to the vagaries of Python's lookup mechanism, the m_level_namespace 
class doesn't work as a method of trying to build a name resolution 
order.  However, if you are OK with the limitations of needing to merge 
namespaces, then the merged_namespace function should work for you (it 
doesn't pick up changed globals, builtins, your class contents, etc.)

With all of that said, you still shouldn't be trying to do a.b[c] .  You 
should instead be doing a.b[a.b.c] .  Is it cumbersome?  Yes.  but it's 
the only mechanism that really makes sense (hacking namespaces is a 
great way of breaking your application in new and unexpected ways).


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python optimization (was Python's "only one way to do it" philosophy isn't good?)

[Josiah Carlson]

John Nagle wrote:
> Josiah Carlson wrote:
[snip]
>> Constant folding happens regardless of optimization level in current 
>> Pythons.
> 
>> So really, assert and docstring removals.  Eh.
> 
>It's hard to optimize Python code well without global analysis.
> The problem is that you have to make sure that a long list of "wierd
> things", like modifying code or variables via getattr/setattr, aren't
> happening before doing significant optimizations.  Without that,
> you're doomed to a slow implementation like CPython.
> 
>ShedSkin, which imposes some restrictions, is on the right track here.
> The __slots__ feature is useful but doesn't go far enough.
[snip]
> Python could get much, much faster.  Right now CPython is said to be 60X 
> slower
> than C.  It should be possible to get at least an order of magnitude over
> CPython.

Don't get me wrong; I'm all for adding optimizations, I was merely 
expressing that currently, 'python -OO' doesn't really do a whole lot. 
I've a long-time user of psyco, have mucked about with 
scipy.weave.inline, and have been a heavy user of Pyrex and C.  If there 
was a method of offering some simple optimization cues to the Python 
runtime to improve its speed, I would be happy to add them when I really 
care about speed (I already do more than that when writing Pyrex).  The 
real question is whether we can get a practical implementation of these 
optimizations as easy to use as psyco.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python's "only one way to do it" philosophy isn't good?

[Josiah Carlson]

Paul Rubin wrote:
> Steven D'Aprano <[EMAIL PROTECTED]> writes:
>>> Not tail calls, in general, no.
>> Sorry, how does that work? You're suggesting that there is an algorithm
>> which the compiler could follow to optimize away tail-recursion, but human
>> beings can't follow the same algorithm?
>>
>> Now I'm confused.
> 
> The usual compiler method is to translate the code into
> continuation-passing style and thereby gain tail-recursion
> optimization automagically.  Of course a human could do the same thing
> in principle, but it would result in insanely difficult-to-write,
> unreadable and unmaintainable code.  At a higher level, there are no
> Python features whatsoever, either existing or proposed, that couldn't
> be eliminated and left to the human.  Iterators?  While loops?  Who
> needs 'em?  We could all go back to programming in machine code.  But
> Python is supposed to make programming easier, not harder.  

Thanks to Richie Hindle, there exists a goto for Python implementation 
that makes such things quite trivial (assuming one doesn't like 
"abusing" break/continue/else).  http://entrian.com/goto/  (which, by 
the way, is the best April-fools joke ever)


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: memory efficient set/dictionary

[Josiah Carlson]

Rafael Darder Calvo wrote:
>> > > Please recommend a module that allows persistent set/dict storage +
>> > > fast query that best fits my problem,
>> >
>> > What is the problem you are trying to solve? How many keys do you have?
>>
>> Corpus processing. There are in the order of billions to tens of
>> billions keys (64bit integers).
>>
> I would recommend you to use a database since it meets your
> requirements (off-memory, fast, persistent). The bsdddb module
> (berkeley db) even gives you a dictionary like interface.
> http://www.python.org/doc/lib/module-bsddb.html

Standard SQL databases can work for this, but generally your 
recommendation of using bsddb works very well for int -> int mappings. 
In particular, I would suggest using a btree, if only because I have had 
troubles in the past with colliding keys in the bsddb.hash (and recno is 
just a flat file, and will attempt to create a file i*(record size) to 
write to record number i .

As an alternative, there are many search-engine known methods for 
mapping int -> [int, int, ...], which can be implemented as int -> int, 
where the second int is a pointer to an address on disk.  Looking into a 
few of the open source search implementations may be worthwhile.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: memory efficient set/dictionary

[Josiah Carlson]

koara wrote:
>>> I would recommend you to use a database since it meets your
>>> requirements (off-memory, fast, persistent). The bsdddb module
>>> (berkeley db) even gives you a dictionary like interface.
>>> http://www.python.org/doc/lib/module-bsddb.html
>> Standard SQL databases can work for this, but generally your
>> recommendation of using bsddb works very well for int -> int mappings.
>> In particular, I would suggest using a btree, if only because I have had
>> troubles in the past with colliding keys in the bsddb.hash (and recno is
>> just a flat file, and will attempt to create a file i*(record size) to
>> write to record number i .
>>
>> As an alternative, there are many search-engine known methods for
>> mapping int -> [int, int, ...], which can be implemented as int -> int,
>> where the second int is a pointer to an address on disk.  Looking into a
>> few of the open source search implementations may be worthwhile.
> 
> Thanks guys! I will look into bsddb, hopefully this doesn't keep all
> keys in memory, i couldn't find answer to that during my (very brief)
> look into the documentation.

No, bsddb does not keep all data in memory.


> And how about the extra memory used for set/dict'ing of integers? Is
> there a simple answer?

A non-long integer for a 32 bit Python uses 12 bytes.  A non-long 
integer for a 64 bit Python uses 24 bytes.

Each entry in a dictionary for a 32 bit Python uses 12 bytes; 4 for the 
hash, 4 for the key pointer, 4 for the value pointer.  Double that to 24 
bytes each in the 64 bit Python (I don't know if the hash actually has 
its range increased, but if one doesn't double the size of the pointers, 
then you have alignment issues that can slow down dictionary access).

Sets only have the hash and key pointers, so only use 8/16 bytes per entry.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save class

[Josiah Carlson]

Gabriel Genellina wrote:
> En Wed, 13 Jun 2007 23:11:22 -0300, nik <[EMAIL PROTECTED]> escribió:
>> It would seem that I want to actually save the source code for the
>> class. I know that I could of course open up an editor and just make
>> it, but my ideal would be to have the base class, Map, be able to make
>> the sub-classes. I don't want the class definition. What I want is an
>> actual class that I could later import and use somewhere else. I am
>> planning to have each one of these map objects contain a different
>> dictionary and then be able to import the map into the application,
>> but have certain methods defined in the Map super-class to draw data
>> out of the specific map's specific dictionary. I hope that makes
>> sense.
>>
>> Something like,
>> class Map:
>>  dict = {}
>>  def DoSomething(self):
>>  pass
>>
>>  def MakeNewMapSubClass(self, newclassname):
>>  """ make a new file, newclassname.py that contains a new
>> class
>>  newclassname(Map) that inherits from base-class Map.
> 
> And are you sure you actually need different subclasses? Will you 
> construct them several instances of each subclass? From the above 
> description I feel you want just different Map *instances*, each with 
> its own dict, not different *subclasses*.

What you said, and that his solution sounds like a Java approach to the 
problem (subclass an abstract base class that calls specific methods on 
the subclass to "do the right thing").

To offer the OP source he can use...

class Map:
 def __init__(self):
 self.dict = {}
 def DoSomething(self):
 #do something with self.dict

Every instance gets a new dictionary.  Now, if he actually wants to 
change the behavior of the DoSomething method, of course then it would 
make sense to subclass Map.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dynamically generated runtime methods & reflection

[Josiah Carlson]

Jay Loden wrote:
> Hi all,
> 
> First, apologies if anyone gets this twice, but it took me quite a
> while to figure out that Python.org is evidently rejecting all mail
> from my mail server because I don't have reverse DNS configured.
> Anyway:
> 
> I'm not even sure how to phrase this question properly or the right
> terminology on this so bear with me. What I'm looking to find out is
> 
> a) is there a functionality in Python where I can call a method I have
> not defined ahead of time, and based on the method name, change the
> functionality of the method at runtime?

Yes.  Implement a __getattr__ method on your class (which you mention).

> b) if not, what is the "Pythonic" approach to the problem outlined
> below? Any recommendations on how to approach the problem differently
> are welcome.
> 
> I've googled and read my Python reference pretty extensively and I've
> found some hints but nothing that really answered my questions, so
> here I am  :-)  I did figure out that you can overload __getattr__ in
> a clas to define a new method at runtime, but I got stuck when I
> couldn't figure out how to have a method know what name it was
> originally called with. That's the basic question, see below for the
> context I'm asking the question in and *why* I want to do the above
> :-)

Ahh, so you want to pass the method name to the method that you are 
returning to be called.  No problem.

 >>> import functools
 >>>
 >>> class foo:
... def __getattr__(self, name):
... return functools.partial(self.ActualMethod, name)
...
... def ActualMethod(self, name, *args, **kwargs):
... #handle *args and **kwargs based on name!
... print name, args, kwargs
...
 >>> foo().bar('hello', world=1)
bar ('hello',) {'world': 1}
 >>>


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dynamically generated runtime methods & reflection

[Josiah Carlson]

Jay Loden wrote:
> Josiah Carlson wrote:
>> Ahh, so you want to pass the method name to the method that you are 
>> returning to be called.  No problem.
>>
>>  >>> import functools
>>  >>>
>>  >>> class foo:
>> ... def __getattr__(self, name):
>> ... return functools.partial(self.ActualMethod, name)
>> ...
>> ... def ActualMethod(self, name, *args, **kwargs):
>> ... #handle *args and **kwargs based on name!
>> ... print name, args, kwargs
>> ...
>>  >>> foo().bar('hello', world=1)
>> bar ('hello',) {'world': 1}
>>  >>>
> 
> Thanks, this is exactly what I was looking for! For some reason functools 
> didn't even show up at all during Google searches...must have just had the 
> wrong search terms.

Well, the particular operation is typically called 'currying a 
function', and unless you know what to look for, it isn't very easy to 
make happen.

On the other hand, it is also relatively easy to implement by hand if 
necessary.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Method much slower than function?

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> On Jun 14, 1:10 am, Paul Rubin  wrote:
>> "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> writes:
>>> take virtually the same amount of time on my machine (2.5), and the
>>> non-join version is clearer, IMO.  I'd still use join in case I wind
>>> up running under an older Python, but it's probably not a big issue here.
>> You should not rely on using 2.5
> 
> I use generator expressions and passed-in values to generators and
> other features of 2.5.

For reference, generator expressions are a 2.4 feature.

>> or even on that optimization staying in CPython.
> 
> You also shouldn't count on dicts being O(1) on lookup, or "i in
> myDict" being faster than "i in myList".

Python dictionaries (and most decent hash table implementations) may not 
be O(1) technically, but they are expected O(1) and perform O(1) in 
practice (at least for the Python implementations).  If you have 
particular inputs that force Python dictionaries to perform poorly (or 
as slow as 'i in lst' for large dictionaries and lists), then you should 
post a bug report in the sourceforge tracker.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Method much slower than function?

[Josiah Carlson]

Francesco Guerrieri wrote:
> On 6/14/07, Peter Otten <[EMAIL PROTECTED]> wrote:
>> Gabriel Genellina wrote:
>> > ...
>> > py> print timeit.Timer("f2()", "from __main__ import 
>> f2").repeat(number=1)
>> > [0.42673663831576358, 0.42807591467630662, 0.44401481193838876]
>> > py> print timeit.Timer("f1()", "from __main__ import 
>> f1").repeat(number=1)
>> >
>> > ...after a few minutes I aborted the process...
>>
>> I can't confirm this.
> 
> [...]
> 
>> $ python2.5 -m timeit -s 'from join import f1' 'f1()'
>> 10 loops, best of 3: 212 msec per loop
>> $ python2.5 -m timeit -s 'from join import f2' 'f2()'
>> 10 loops, best of 3: 259 msec per loop
>> $ python2.5 -m timeit -s 'from join import f3' 'f3()'
>> 10 loops, best of 3: 236 msec per loop
> 
> On my machine (using python 2.5 under win xp) the results are:
 print timeit.Timer("f2()", "from __main__ import f2").repeat(number 
 = 1)
> [0.19726834822823575, 0.19324697456408974, 0.19474492594212861]
 print timeit.Timer("f1()", "from __main__ import f1").repeat(number 
 = 1)
> [21.982707133304167, 21.905312587963252, 22.843430035622767]
> 
> so it seems that there is a rather sensible difference.
> what's the reason of the apparent inconsistency with Peter's test?

It sounds like a platform memory resize difference.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: save class

[Josiah Carlson]

nik wrote:
> of having to keep track of a separate dictionary file. I am new to
> this, but I thought that this would be a regular thing to do in
> python, because people must make classes in the interactive console
> and then export them somehow for later use.

Create a file.  Put your code in it.  Run your code.  Occasionally 
copy/paste your code into the console for testing and/or learning about 
how Python works.  If you write something you want to keep in the 
console, copy it out of the console and paste it into your source file(s).

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dynamically generated runtime methods & reflection

[Josiah Carlson]

Josiah Carlson wrote:
> Well, the particular operation is typically called 'currying a 
> function', and unless you know what to look for, it isn't very easy to 
> make happen.

Replace "make happen" to "discover in the standard library".

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Priority Queue with Mutable Elements

[Josiah Carlson]

Chris Lasher wrote:
> I am working with large graphs (~150,000 to 500,000 nodes) which I
> need decompose node-by-node, in order of a node's value. A node's
> value is determined by the sum of its edge weights. When a node is
> removed from the graph, its neighbors' values must be updated to take
> into account the removed edges.
> 
> I was told to look for a priority queue in Python. I had a look at the
> heapq module. It looks like it supports ordering on insertion, but I'm
> unsure how to update the queue once a node's value changes.
> Additionally, I need the queue to be sorted on a particular attribute
> of the node objects, and don't see a way to do that other than
> override the __cmp__ method. Should I just use a list and use .sort()
> or sorted()? That seems like it could be horribly inefficient.
> 
> I found Andrew Snare's PQueue extension module [1] , which supports
> updating values in the priority queue by reassignment. It appeared to
> be broken in Python2.5 [2] but I found the offending line (a call to
> PyMem_DEL) and changed it (to PyObject_FREE) and it appears to be
> working fine. I would prefer to limit external depedencies for my
> modules, however.

See this implementation of a "pair heap":
   http://mail.python.org/pipermail/python-dev/2006-November/069845.html
...which offers the ability to update the 'priority' of an entry in the 
heap.  It requires that the 'value' in (priority, value) pairs be unique 
(to the heap) and hashable.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: smtp server simulation using Python

[Josiah Carlson]

William Gill wrote:
> I have a (web) development computer w/o an SMTP server and want to test 
> form generated e-mail using a dummy SMTP server that delivers the mail 
> message to a file, or better yet, to a text editor instead of actually 
> sending it.  Is it possible to extend the DebuggingServer class,and 
> override the process_message() method to accomplish this?  If so, any 
> suggestions are appreciated.

I would check out the smtpd module .  It isn't documented, but the 
source is fairly straightforward for subclassing.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Trivial string substitution/parser

[Josiah Carlson]

Samuel wrote:
> On Sun, 17 Jun 2007 11:00:58 +, Duncan Booth wrote:
> 
>> The elegant and lazy way would be to change your specification so that $
>> characters are escaped by $$ not by backslashes. Then you can write:
>>
> from string import Template
> ...
> 
> Thanks, however, turns out my specification of the problem was 
> incomplete: In addition, the variable names are not known at compilation 
> time.

You mean at edit-time.

 >>> t.substitute(variable1="hello", variable2="world")

Can be replaced by...

 >>> t.substitute(**vars)

...as per the standard **kwargs passing semantics.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: MS Word parser

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> I'm currently using antiword to extract content from MS Word files.
> Is there another way to do this without relying on any command prompt
> application?

There is also wvware http://wvware.sourceforge.net/, but it is also 
generally a command-line application.  Either of these programs are open 
source, so you could (with a bit of work) wrap them with Swig or Pyrex 
to access them directly from Python.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problem with Python

[Josiah Carlson]

Squzer Crawler wrote:
> On Jun 18, 11:06 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
>> On Jun 17, 8:51 pm, Squzer Crawler <[EMAIL PROTECTED]> wrote:
>>
>>> i am developing distributed environment in my college using Python. I
>>> am using therads in client for downloading wepages. Even though i am
>>> reusing the thread, memory usage get increased. I don know why.? I am
>>> using BerkelyDB for URLQueue, BeautifulShop for Parsing the webpages.
>>   Isn't the increased memory resulted from storing the already
>> processed pages?
>>
>>   Look first at all places where your code instantiates new
>> objects - and make sure you don't keep references to such objects that
>> are not needed anymore.
>>
>>   Also, reusing threads has nothing to do with saving memory - but
>> with saving on thread creation time, if I understand your problem
>> description.
> 
> what about the cyclic reference.. can i use GC in my program..
> 
> if so, please tell me how to implement.. i am calling the gc.collect()
> at the enf of the fetching.. Will it reduce my program speed. Else in
> which way i can call it..?

Garbage collection should happen automatically as long as you are 
deleting references to objects you no longer need.  If gc.garbage isn't 
empty, then you have unbreakable reference cycles.  It seems more 
likely, as [EMAIL PROTECTED] says, that you are keeping copies of the things 
you already parsed in memory.

What you can do (if you aren't able to find the bug) is have a wrapper 
program that repeatedly starts up your url fetcher via os.system(). 
Then have your url fetcher close itself down every few hours.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: cStringIO unicode weirdness

[Josiah Carlson]

Paul Rubin wrote:
> Python 2.5 (r25:51908, Oct  6 2006, 15:24:43)
> [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import StringIO, cStringIO
> >>> StringIO.StringIO('a').getvalue()
> 'a'
> >>> cStringIO.StringIO('a').getvalue()
> 'a'
> >>> StringIO.StringIO(u'a').getvalue()
> u'a'
> >>> cStringIO.StringIO(u'a').getvalue()
> 'a\x00\x00\x00'
> >>> 
> 
> I would have thought StringIO and cStringIO would return the
> same result for this ascii-encodeable string.  Worse:

You would be wrong.  The behavior of StringIO and cStringIO are 
different under certain circumstances, and those differences are 
intended.  Among them is when they are confronted with unicode, as you 
saw.  Another is when provided with an initializer...

 >>> cs = cStringIO.StringIO('a')
 >>> cs.write('b')
 Traceback (most recent call last):
   File "", line 1, in ?
 AttributeError: 'cStringIO.StringI' object has no attribute 'write'
 >>> s = StringIO.StringIO('a')
 >>> s.write('b')

There is a summer of code project that is working towards making them 
behave the same, but the results will need to wait until Python 2.6 
and/or 3.0 .  Note that there are a few "closed, won't fix" bug reports 
regarding these exact same issues in the Python bug tracker at sourceforge.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python and (n)curses

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> I need to develop a cross-platform text-mode application. I would like
> to do it in Python and I would like to use a mature text-mode library
> for the UI stuff.
> 
> The obvious choice, I thought, was ncurses. But as far as I can tell,
> it is not available for Python on Windows?
> 
> Is there a workaround? Or are there alternative libraries that might
> be used instead of (n)curses? I know I can use (n)curses on *nix and
> console on Windows etc., but that is of course suboptimal.


This link offers a series of links for console IO...
http://mail.python.org/pipermail/python-list/2005-January/303984.html

Among them is a link to the 'wcurses' module that has been relocated 
here: http://adamv.com/dev/python/curses/  It doesn't actually work in 
the console, but it seems as though you can program towards one API.

There is always cygwin + Python + ncurses.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: high performance/threaded applications in Python - your experiences?

[Josiah Carlson]

Ivan Voras wrote:
> Jay Loden wrote:
> 
>> I was hoping for some experiences that some of you on the list may have had 
>> in dealing with Python in a high performance and/or threaded environment. In 
>> essence, I'm wondering how big of a deal the GIL can be in a  real-world 
>> scenario where you need to take advantage of multiple processor machines, 
>> thread pools, etc. How much does it get in the way (or not), and how 
>> difficult have you found it to architect applications for high performance? 
>> I have read a number of articles and opinions on whether or not the GIL is a 
>> good thing, and how it affects threaded performance on multiple processor 
>> machines, but what I haven't seen is experiences of people who have actually 
>> done it and reported back "it was a nightmare" or "it's no big deal" ;)
> 
> The theory: If your threads mostly do IO, you can get decent CPU usage
> even with Python. If the threads are CPU-bound (e.g. you do a lot of
> computational work), you'll effectively only make use of one processor.
> 
> In practice, I've noticed that Python applications don't scale very much
> across CPUs even if they're doing mostly IO. I blame cache trashing or
> similar effect caused by too many global synchronization events. I
> didn't measure but the speedup may even be negative with large-ish
> number of CPUs (>=4).
> 
> OTOH, if you can get by with using forking instead of threads (given
> enough effort) you can achieve very good scaling.

Also, see the 'processing' package in the Python cheeseshop.  It allows 
you to use processes rather than threads with most of the same 
abstractions.  I hear it recently acquired the ability to pass file 
handles between processes on the same machine :)

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Chroot Jail Not Secure for Sandboxing Python?

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> On Jun 25, 1:43 am, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
>> [EMAIL PROTECTED] schrieb:
>>
>>> This wiki page suggests using a chroot jail to sandbox Python, but
>>> wouldn't running something like this in your sandboxed Python instance
>>> still break you out of the chroot jail:
>>>  os.execle ('/usr/bin/python','-c','import os; os.execlp("/bin/sh")',
>>> {})
>> Depending on how the chroot jail is set up, this command might not
>> work - in the jail, /bin/sh might not exist.
> 
> I followed up with my ISP.  Here's the answer I got:
> 
> The os.exec call prepends the chroot directory to the absolute path,
> but does NOT provide chroot for the child process.  However, as long
> as the environment is maintained, which contains an LD_PRELOAD, the
> "chroot" will also be maintained.  If  LD_PRELOAD is removed or
> ignored, then the chroot is ineffective.
> 
> Another way of saying it is that every process is responsible for
> providing and maintaining the chroot through the LD_PRELOAD variable.
> Those processes only maintain the chroot if that variable remains set.
> 
> The only solution that would bypass this problem altogether would be a
> statically linked python. (is that possible?)  It would have to be
> statically linked to a custom-modified glibc to provide the virtual
> chroot environment.

It seems to me that if a (potentially malicious) process needs to behave 
itself in order for chroot to not allow the process access to the full 
system, then chroot is broken.  But perhaps I don't understand the 
intent and scale of what chroot intends to do.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Internals and complexity of types, containers and algorithms

[Josiah Carlson]

Harald Luessen wrote:
> On Mon, 25 Jun 2007 Martin v. Löwis wrote:
> 
>> Sure, see below:
>>
>> - tuples are represented as arrays, with a single block for the
>>  entire objects (object header, tuple size, and data)
>> - list are represented as arrays, with two memory blocks:
>>  one for object header and sizes, and the other one for the
>>  "guts", i.e. the actual data. The list uses over-allocation,
>>  to avoid resizing on each addition.
>> - strings are implemented as arrays, with a single block for
>>  the entire string. In addition to header, size, and data,
>>  it also contains a cached hash and a pointer to the interned
>>  version of the string (if any).
>> - dicts are implemented as hash tables, with open addressing.
> ... and more interesting things ...
> 
> Thank you. That was the information I was looking for. 
> I just forgot to ask for sets.

Generally, sets are dictionaries without values (and some tweaks related 
to handling set operations).

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: guidance needed: best practice for script packaging

[Josiah Carlson]

Alan Isaac wrote:
> This is a simple question about actual practice.
> I just want to know how you (yes you) are
> approaching this problem. 
> The problem:
> What is the recommended packaging of
> demo scripts or test scripts for a package
> to be distributed to others to "play with".
> (I.e., without "installing".)
> 
> Example:
> Suppose I have the package structure:
> 
> package/
>__init__.py
>subpackage1/
>__init__.py
>moduleXX.py
>subpackage2/
>__init__.py
>moduleYY.py
> 
> Important detail:
> moduleXX uses a relative import to access moduleYY.
> 
> The goal:
> I have a script test.py that I want to
> distribute with the package.  This script will import
> moduleXX to illustrate or test the module's use.
> 
> Is it the case that this script cannot reasonably be
> bundled with `package`?  (I.e., within its directory
> structure.)
> 
> Note:
> If I put it in the `subpackage1` directory and
> just import moduleXX, I will get
> ValueError: Attempted relative import in non-package
> 
> Note:
> If I put it in the `package` directory and
> import subpackage1.moduleXX, I will get
> ValueError: Attempted relative import beyond toplevel package
> 
> Here is one hack, based on a suggestion of Alex Martelli
> http://mail.python.org/pipermail/python-list/2007-May/438250.html
> and others.
> - add a `scripts` subdirectory to `package`
> - use path manipulation to find the directory holding `package`
> - insert this directory in sys.path
> 
> This hack "works".
> However it has also been claimed that this approach is an
> insane for any shared code.  Is it?  If so, what is best practice?

I have used this method more than once to get around the fact that 
relative imports don't work the way I want them to in this situation. 
Alternatively, you could use...

package/
   __init__.py #for 'import package'
   test.py
   shared/
 __init__.py
 subpackage1/
   __init__.py
   moduleXX.py
 subpackage2/
   __init__.py
   moduleYY.py

And always run scripts from the package/... path or from something that 
is able to 'import package'.  It does force a layer of useless 
namespace, but it works.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: 16bit hash

[Josiah Carlson]

Robin Becker wrote:
> Is the any way to get an efficient 16bit hash in python?

hash(obj)&65535

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a Python app with Mozilla

[Josiah Carlson]

Thorsten Kampe wrote:
> Hi,
> 
> I've already sent this to the Komodo mailing list (which seemed to me 
> the more appropriate place) but unfortunately I got no response.
> 
> I'd like to build a Python GUI app. Neither Tkinter nor Wxpython nor 
> PyQT are actually what I want (because the lack of GUI builders and 
> they don't really look good on Windows and Linux).

You apparently didn't look very hard.  On the wxPython end of things 
(which I have experience with), there is wxGlade, XRCed, Boa 
Constructor, Dabo, etc.  I don't know about Tkinter (I don't use it), 
but I know that at least Qt has a very nice GUI designer and builder 
(from Trolltech if you are willing to pay for it), and if I remember 
correctly, Eric4 offers some tools to make PyQt editing nice and 
friendly.  I'm sure there are others.

As for wxPython applications not looking good on all platforms; I hate 
to break it to you, but the majority of the controls are *native* on 
whatever platform you happen to be using (except for Qt-based frontends 
in *nix, wxPython uses a Gtk binding in *nix).  Take a browse through a 
few of the applications listed in the wiki: 
http://wiki.wxpython.org/wxPythonPit_Apps , or even Andrea Gavana's 
custom widgets http://xoomer.alice.it/infinity77/eng/freeware.html .


> Komodo itself is an excellent example of a - at least Python driven - 
> application that looks superb and has superior functionality so it 
> seems natural to use the Komodo approach for me.

I could have sworn that Komodo was written in PyQt.  Unless my memory is 
failing me (doubtful, as they at least used to offer some PyQt 
development tools), I would imagine they merely embedded the mozilla 
bits and pieces with some C extensions.

In any case, you really don't want to be developing an application using 
XPI.  According to everyone I've ever talked to about XPI, while getting 
the basic stuff working isn't bad, doing anything worth mentioning is 
about as big a pain in the butt than just writing everything in 8088 
assembly.  You mileage may vary of course.


> * Is there a simple How-To how to build a very simple (Python) app 
> with the Mozilla framework? Kind of "Hello world"...?

Maybe.  Search terms you should use are 'embed XPI python', which give 
me about 189k results in google, some of which seem applicable on the 
first page (which also leads me to believe that my statements regarding 
Komodo embedding mozilla technology may be right, though maybe they 
wrote Komodo in C and Qt).


> * Is is reasonable to think that building a GUI with Mozilla is easier 
> than using Python frameworks because Mozilla does most of the GUI 
> work?

Doubtful.  Building a GUI app with XRCed and wxPython is pretty easy if 
you understand Python and object-oriented programming.  There are some 
annoying vagaries with events, but if you ask your questions on the 
wxpython-users mailing list (over at wxpython.org), one of at least a 
half-dozen people will likely answer your question.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory leak issue with complex data structure

[Josiah Carlson]

Alan Franzoni wrote:
> I have a root node which is not referenced by any other node. So, I
> created a "clear()" method which is called on all children (by calling
> their clear() method" and then clears the set with the references of the
> node itself.

Using the .clear() method on sets (or dictionaries) does not reduce the 
memory that the underlying hash tables use, it only removes references 
to the keys (and values) that the sets (and dictionaries) contain(s). 
If your sets used to be large, I could have sworn that you could force a 
resize down by adding some items to the set, but I seem to be mistaken:

 #memory use is 3440k
 s = set(xrange(100))
 #memory use is 31,864k
 s.clear()
 #memory use is 15,460k
 s.add(0)
 s.remove(0)
 #memory use is 15,460k

Then again, this may be an artifact of my operating system memory 
freeing semantics.

As is usually the case, read the C source from svn.python.org/view (or 
some other means) for more information.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CGI vs WSGI

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> I'm wondering because the only variables I ever needed were PATH_INFO,
> REQUEST_METHOD, QUERY_STRING and maybe one more, all of which should
> be available from CGI, too.

CGI starts up a new process for every request, using stdin/stdout for 
passing information.  I believe WSGI can do that, but I believe it can 
also do in-process requests (think mod_python, only with a Python web 
server), external process requests (think FastCGI or LRWP), etc.

I could certainly be mistaken, it's been a while since I looked into any 
of it.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interfacing python program to C++ program instance

[Josiah Carlson]

Jim Langston wrote:
> Windows. Situation: Using a Python program called OpenRPG.  I have a program 
> that displays form data (a character sheet) in C++.  I am able in the C++ 
> program to build a string and copy it into the clipboard, then paste it into 
> the input in the running Python program.

If I remember correctly, OpenRPG uses wxPython for it's GUI.  wxPython 
offers fairly easy access to the clipboard contents, so it could check 
the clipboard for the kind of data it wants.

> I would like to somehow automate this, that is, have the python instance 
> communicate with the C++ instance and vice versa.  I'm still trying to think 
> of a way to do this.  There seems to be a number of options, but I'm not 
> sure which one is best, or there is a better one.
> 
> 1. Write a new C++ program/library with extern C that Python could call. 
> This interface program would open up some shared memory that the C++ 
> application would also open up, and python and C++ could communicate that 
> way.  Python writing requests to the memory, C++ responding with responses.

mmap is the standard "shared memory communication" mechanism available 
on pretty much all platforms of note (Windows, Linux, OSX, BSD, Solaris, 
BeOS, etc.)


> 2. Have the C++ program interface directly into the python form, reading 
> directly from controls.  C++ could write to the input box with reponses.

If OpenRPG uses any custom controls (which are very easy to write with 
wxPython), this won't be easy.  Also, controlling an application that 
wasn't designed for such control can be a beast.


> 3. Have Python write a small file with the request to the HD.  Have the C++ 
> program intermittedly check for the presense of this file. If it exists, it 
> would open the file, read the request, write a response file, then delete 
> the file Python wrote.

This can work, but is unnecessary.  See mmap, socket, etc.


> 4. Find out if python can directly, somehow, open up shared memory and do 
> the same as 1 without the need for the extern C interface program.

import mmap


> 5. Something else I'm not thinking of but you know.

sockets.  In particular, XML-RPC for the call/return using structured 
data in XML format.  There exists XML-RPC server and client libraries in 
Python, and XML-RPC server and client libraries exist for just about 
every language worth discussing.  Alternatively, you can pass your 
queries/string directly via sockets (rather than the clipboard).


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Auto-parallelizing with decorators?

[Josiah Carlson]

Kirk Strauser wrote:
> In article <[EMAIL PROTECTED]>,
>  Kirk Strauser <[EMAIL PROTECTED]> wrote:
> 
>> I was thinking about how a lot of Lisp proponents claim that Lisp is
>> inherently parallelizable because its functions don't have (or are not
>> supposed to have) side effects, and therefore the compiler can easily tell
>> which calls may be run in parallel instead of strictly serially.  I'm not a
>> Lisp expert so I can't say whether that's true or not, but it seems like an
>> interesting idea for Python.
> 
> By the way, I uploaded a sample implementation (in Python) of what I had 
> in mind to http://www.honeypot.net/multi-processing-map-python .  Please 
> let me know what you think of it and whether it seems remotely 
> interesting or goofy.

Try the Processing package available at the Python package index. 
Create as many processes as you want, then toss the data you want 
processed into a Queue.  Watch magically as your data gets processed.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting the home directory in Python and a bug in os.path.expanduser

[Josiah Carlson]

Edward Diener wrote:
> What is the generic operating system way of getting the home directory ?
> 
> I am guessing it is os.path.expanduser("~"). Is there a better way or an 
> alternate way ?
> 
> If it is as I surmise, the aforementioned expanduser("~") of os.path 
> seems incorrect to me under Windows. The document says:
> 
> "On Windows, only "~" is supported; it is replaced by the environment 
> variable HOME or by a combination of HOMEDRIVE and HOMEPATH."
> 
> But HOME is never the home directory for Windows, only the combination 
> of HOMEDRIVE and HOMEPATH is valid, which is always set. If MSYS is 
> installed under Windows, where HOME must be set to the MSYS home 
> directory for a given user in order to emulate Linux/Unix, attempting to 
> use os.path.expanduser("~") will incorrectly return the MSYS home 
> directory for a given user rather than the Windows home directory for 
> the logged in user. So I think the os.path.expanduser("~") works 
> incorrectly in this case and needs to be fixed, else you are telling 
> users never to use MSYS under Windows.

Some people have "sane" values for HOME on Windows.  That's the only 
reason why it was included in expanduser().  The current trunk version 
of ntpath offers HOME, USERPROFILE or HOMEDRIVE+HOMEPATH, as well as the 
expansion of ~/extra/stuff .

If you would like to get rid of Python's support of HOME, please post a 
bug report or feature request on the sourceforge tracker.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: trouble controlling vim with subprocess on windows machine

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
>  I am having trouble contolling vim with subprocess on a windows
> machine.  It appears that vim comes up on the machine all right and it
> sometimes looks like it is doing the searchs what I am asking it to do
> but when I am asking it to load a file it doesn't do anything.  Is
> there something I need to do to push the data through the pipe??  Here
> is a couple different ways I am trying to do it.
[snip]

This recipe for asynchronous communication using subprocess could be 
used to write an expect-like tool:
   http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

It works on both Windows and *nix.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: trouble controlling vim with subprocess on windows machine

[Josiah Carlson]

[EMAIL PROTECTED] wrote:
> On Jul 9, 11:06 am, Josiah Carlson <[EMAIL PROTECTED]>
> wrote:
>> [EMAIL PROTECTED] wrote:
>>>  I am having trouble contolling vim with subprocess on a windows
>>> machine.  It appears that vim comes up on the machine all right and it
>>> sometimes looks like it is doing the searchs what I am asking it to do
>>> but when I am asking it to load a file it doesn't do anything.  Is
>>> there something I need to do to push the data through the pipe??  Here
>>> is a couple different ways I am trying to do it.
>> [snip]
>>
>> This recipe for asynchronous communication using subprocess could be
>> used to write an expect-like tool:
>>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
>>
>> It works on both Windows and *nix.
>>
>>   - Josiah
> 
> I had the original dir work but when I tried to trade it out with vim
> it isn't clear
> how I should call it..  vim filename and it doesn't find filename for
> some reason.
> I called it pipe and then
> 
> 
> inport pipe
> 
> def load_instrument3(instr_name, csd_name):
>   if sys.platform == 'win32':
> shell, commands, tail = ('gvim' + csd_name, (csd_name,
> csd_name), '\r\n')
>   else:
> shell, commands, tail = ('sh', ('ls', 'echo HELLO WORLD'),
> '\n')
> 
>   a = pipe.Popen(shell, stdin=pipe.PIPE, stdout=pipe.PIPE)
>   print pipe.recv_some(a),
>   for cmd in commands:
> pipe.send_all(a, csd_name)
> print pipe.recv_some(a),
>   pipe.send_all(a, csd_name)
>   print pipe.recv_some(a, e=0)
>   a.wait()

The example uses a platform-specific shell in order to use the 
environment variable PATH to discover the executable to run.  If you 
know the exact path to your binary ('gvim' for you), it should work. 
As-is, your program would require a binary with the name 'gvim'+csd_name 
in the same path as the script you are executing.

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: highly einteractive editor for python

[Josiah Carlson]

Atul Bhingarde wrote:
> Does anybody know an editor that facilitates, interactive python 
> development. Where GUI etc developed will be possible to see in real time 
> mode.

Boa Constructor?  wxGlade embedded in some other software (SPE?)  XRCed? 
  What do you mean by "real time mode"?

  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: trouble controlling vim with subprocess on windows machine

[Josiah Carlson]

agc wrote:
> Hi Josiah,
> 
 This recipe for asynchronous communication usingsubprocesscould be
 used to write an expect-like tool:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
> 
> I have played with the above recipe and it is excellent,
> but could you please go into some more detail about what is needed
> to make a cross platform [p]expect like (non-pty based) tool?

We only get ptys.  Unless you can figure out some incantation to fake 
the creation of a tty (so that ssh and other software don't conmplain), 
all you get is a pty.

To make it work like expect, one would (strictly speaking) need to do 
the pattern matching that expect does, which may require writing a new 
regular expression library.


> Specifically, could you describe how you would connect to
> *another* interactive Python process with your subclass of
> subprocess.Popen?
> i.e:
> 
> a = Popen('python', stdin=?, stdout=?, stderr=?)
> #now run an interactive session with 'a'

Some platforms have very strange buffering behavior with Python.  What I
have done in my own code (for creating a Python subprocess, though not 
using the subprocess module) is to do something like the following...

python_cmd = ''' -u -c "import sys; \
  sys.stderr=sys.__stderr__=sys.stdout;import __builtin__;\
  __builtin__.quit=__builtin__.exit=\
  'use Ctrl-Break to restart *this* interpreter';import code;\
  code.interact(readfunc=raw_input)"'''

a = Popen(python_cmd, subprocess.PIPE, subprocess.PIPE)

> I have tried several combinations of the above and I seem
> to be stuck on the fact that python is interacting with a
> 'tty', not 'std*'.  Maybe I'm missing a basic piece?

I've had buffering issues on certain platforms (Windows), error messages 
about not having a tty (when trying to control an ssh session on *nix 
and Windows), etc.  Getting this to work for arbitrary programs that 
expect a tty, and/or being able to query the terminal for other 
information (dimension, terminal emulation, etc.), will be a chore, if 
not impossible.


  - Josiah
-- 
http://mail.python.org/mailman/listinfo/python-list