Re: Database specialized in storing directed graphs?

[Chris Rebert]

On Mon, Oct 27, 2008 at 5:32 PM, Carl Banks <[EMAIL PROTECTED]> wrote:
> I was wondering if anyone had any advice on this.
>
> This is not to study graph theory; I'm using the graph to represent a
> problem domain.  The graphs could be arbitrarily large, and could
> easily have millions of nodes, and most nodes have a substantial
> amount of data associated with them.  Obviously I don't want a whole
> such graph in memory at once, so libraries the just deal with in-
> memory graphs are out.
>
> I know I could implement this with a relational DB, and I'd be fine
> with a library that was built on top of one.  But I'm hoping for
> specialzed behavior useful for graphs.
>
> For example, suppose I have a set of nodes called A.  It would be
> useful to know if any of these nodes are connected by paths that
> include no nodes in A.  I could, of course, do that by reading from
> the database and following the paths, but I clearly don't want to do
> that.  I would want instead to somehow cache the outside connectedness
> relationship between different nodes of A.  But then what happens if
> something changes elsewhere.  How is the cache for A notified, and can
> it be updated efficiently using graph theorems, rather than
> regenerated?
>
> It's very tricky; that's why I hope someone else has done it.
>
> I'm guessing no.

By sacrificing a goat at the altar of the Almighty Google, I was able
to locate a project I came upon a long while ago but couldn't remember
the name of that's vaguely like what you want, in that it's a "graph
database": Neo4j - http://neo4j.org/ (and yes, it's in Java; sigh)
Not sure it's exactly what you're looking for, but anyway

Cheers,
Chris
-- 
Follow the path of the Iguana...
http://rebertia.com

>
>
> Carl Banks
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Rhamphoryncus]

On Oct 26, 6:57 pm, "Andy O'Meara" <[EMAIL PROTECTED]> wrote:
> Grrr... I posted a ton of lengthy replies to you and other recent
> posts here using Google and none of them made it, argh. Poof. There's
> nothing that fires more up more than lost work,  so I'll have to
> revert short and simple answers for the time being.  Argh, damn.
>
> On Oct 25, 1:26 am, greg <[EMAIL PROTECTED]> wrote:
>
>
>
> > Andy O'Meara wrote:
> > > I would definitely agree if there was a context (i.e. environment)
> > > object passed around then perhaps we'd have the best of all worlds.
>
> > Moreover, I think this is probably the *only* way that
> > totally independent interpreters could be realized.
>
> > Converting the whole C API to use this strategy would be
> > a very big project. Also, on the face of it, it seems like
> > it would render all existing C extension code obsolete,
> > although it might be possible to do something clever with
> > macros to create a compatibility layer.
>
> > Another thing to consider is that passing all these extra
> > pointers around everywhere is bound to have some effect
> > on performance.
>
> I'm with you on all counts, so no disagreement there.  On the "passing
> a ptr everywhere" issue, perhaps one idea is that all objects could
> have an additionalfieldthat would point back to their parent context
> (ie. their interpreter).  So the only prototypes that would have to be
> modified to contain the context ptr would be the ones that don't
> inherently operate on objects (e.g. importing a module).

Trying to directly share objects like this is going to create
contention.  The refcounting becomes the sequential portion of
Amdahl's Law.  This is why safethread doesn't scale very well: I share
a massive amount of objects.

An alternative, actually simpler, is to create proxies to your real
object.  The proxy object has a pointer to the real object and the
context containing it.  When you call a method it serializes the
arguments, acquires the target context's GIL (while releasing yours),
and deserializes in the target context.  Once the method returns it
reverses the process.

There's two reasons why this may perform well for you: First,
operations done purely in C may cheat (if so designed).  A copy from
one memory buffer to another memory buffer may be given two proxies as
arguments, but then operate directly on the target objects (ie without
serialization).

Second, if a target context is idle you can enter it (acquiring its
GIL) without any context switch.

Of course that scenario is full of "maybes", which is why I have
little interest in it..

An even better scenario is if your memory buffer's methods are in pure
C and it's a simple object (no pointers).  You can stick the memory
buffer in shared memory and have multiple processes manipulate it from
C.  More "maybes".

An evil trick if you need pointers, but control the allocation, is to
take advantage of the fork model.  Have a master process create a
bunch of blank files (temp files if linux doesn't allow /dev/zero),
mmap them all using MAP_SHARED, then fork and utilize.  The addresses
will be inherited from the master process, so any pointers within them
will be usable across all processes.  If you ever want to return
memory to the system you can close that file, then have all processes
use MAP_SHARED|MAP_FIXED to overwrite it.  Evil, but should be
disturbingly effective, and still doesn't require modifying CPython.
--
http://mail.python.org/mailman/listinfo/python-list

Python suitable for Midi ?

[Protocol]

Hello all

Is Python suitable for building a multi-track midi sequencer (with a
gui), that would run on windows / mac ? I fail to find sufficient
information on this, being a newbie and all. Furthermore, i found
references on Python not being really able of multi-threading, that
further adds to the confusion.

Please assist.

Panos

--
http://mail.python.org/mailman/listinfo/python-list

Setting up Python on a network of machines

[John [H2O]]

I have set up the Python computing environment on a machine within a network
to include the following Python modules:
1) Scipy
2) Numpy
3) Matplotlib v.98 or greater
4) Matplotlib toolkit Basemap v.99 or greater
5) Scientific

These are standard module products available via sourceforge.net or links
directly from python.org

I would like that this implementation of python be set up on all our
machines in a network. Is there a way to do this without installing all the
python modules individually on each machine? I have heard the problem will
be that things will compile differently - for example some machines are 64
bit. Could I at least create a network instance of python for the 32 bit
machines? They are mostly the same Fedora Core [5-8].

How does one create a 'common' instance of python for a network of machines?

Thanks.
-- 
View this message in context: 
http://www.nabble.com/Setting-up-Python-on-a-network-of-machines-tp20203881p20203881.html
Sent from the Python - python-list mailing list archive at Nabble.com.

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Glenn Linderman wrote:

> so a 3rd party library might be called to decompress the stream into a
> set of independently allocated chunks, each containing one frame (each
> possibly consisting of several allocations of memory for associated
> metadata) that is independent of other frames

We use a combination of a dictionary + RGB data for this purpose. Using a
dictionary works out pretty nicely for the metadata, and obviously one
attribute holds the frame data as a binary blob.

http://www.kamaelia.org/Components/pydoc/Kamaelia.Codec.YUV4MPEG gives some
idea structure and usage. The example given there is this:

Pipeline( RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

Now all of those components are generator components.

That's useful since:
   a) we can structure the code to show what it does more clearly, and it
  still run efficiently inside a single process
   b) We can change this over to using multiple processes trivially:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

This version uses multiple processes (under the hood using Paul Boddies
pprocess library, since this support predates the multiprocessing module
support in python).

The big issue with *this* version however is that due to pprocess (and
friends) pickling data to be sent across OS pipes, the data throughput on
this would be lowsy. Specifically in this example, if we could change it
such that the high level API was this:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
use_shared_memory_IPC = True,
).run()

That would be pretty useful, for some hopefully obvious reasons. I suppose
ideally we'd just use shared_memory_IPC for everything and just go back to
this:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

But essentially for us, this is an optimisation problem, not a "how do I
even begin to use this" problem. Since it is an optimisation problem, it
also strikes me as reasonable to consider it OK to special purpose and
specialise such links until you get an approach that's reasonable for
general purpose data.

In theory, poshmodule.sourceforge.net, with a bit of TLC would be a good
candidate or good candidate starting point for that optimisation work
(since it does work in Linux, contrary to a reply in the thread - I've not
tested it under windows :).

If someone's interested in building that, then someone redoing our MiniAxon
tutorial using processes & shared memory IPC rather than generators would
be a relatively gentle/structured approach to dealing with this:

   * http://www.kamaelia.org/MiniAxon/

The reason I suggest that is because any time we think about fiddling and
creating a new optimisation approach or concurrency approach, we tend to
build a MiniAxon prototype to flesh out the various issues involved.


Michael
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python suitable for Midi ?

[Michael Sparks]

Protocol wrote:

> Is Python suitable for building a multi-track midi sequencer (with a
> gui), that would run on windows / mac ? I fail to find sufficient
> information on this, being a newbie and all. 

We had a Google Summer of Code student working on this sort of thing this
year (This clearly puts a bounds on experience btw). He got a fair way along
with this, but didn't get his code to a mergeable state (mainly in the
application files rather than in the components which are largely
mergeable).

Essentially it generated a bunch of components which he was using in his
application, which are naturally reusable.

I've pinged him since the end of GSOC, and had a response that essentially
says "sorry, too busy, maybe next semester". We'll see on that (he was
having fun, so I suspect he really does want to come back to it, but hasn't
had time). In the meantime, the answer is, yes, you can do this in Python.
His code is here, if it's useful to you:

   * http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/JT
   * ( svn co http://kamaelia.googlecode.com/svn/trunk/Sketches/JT )
  - requires Kamaelia 0.6.0 btw

Top level file/application file is this:

http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/application/trunk/jam

(The reason I don't feel it's mergeable as is, is because it could with a
code clean up IMO, but it's pretty good - to say the least - considering
the GSOC timeframe - also it shows it is doable, and if you're reusing his
components, then it should be very doable)

It is likely though that his component library:
http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/library/trunk/Kamaelia/Apps/Jam/Protocol/

will get merged before christmas though, since that code is a lot cleaner.

In case you're wondering, he was aiming to build something a bit like this,
but using Pygame as the interface, and it allows multiple users to run
their local version, connect to each other and do shared "jamming" - hence
the interesting subdirectory is "Jam"

As well as Midi it also supports OSC.

Underneath it all, he was using the python "rtmidi" bindings, and pyosc
bindings to talk Midi and Osc. (Most of his quagmire in the last bit of
GSOC was caused by audio under linux, which doesn't sound relevant to you)

> Furthermore, i found references on Python not being really able of
> multi-threading, that further adds to the confusion.

Kamaelia's component model effectively gives you concurrency for free,
since you build systems out of components that talk to each other. I
haven't attached the introspector to Jam, but I suspect it's embarrassingly
parallel. He didn't have to worry about that though :-)

If you're curious about the model, this tutorial is specifically targetted
at new developers:
http://www.kamaelia.org/MiniAxon/

It was originally written for someone (specific) who had learnt python the
previous week, had little programming experience, and we needed to get them
up to speed quickly and gently. It's been slightly generalised since, but
is a nice introduction to the ideas. (Under the hood Kamaelia is
significantly more optimised than that example, but the components you
create for that tutorial system work with the full/real world system)

Regards,


Michael
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Test, please ignore.

[Protocol]

Test, please ignore.
--
http://mail.python.org/mailman/listinfo/python-list

[newbie] Right way to access item in array?

[Gilles Ganault]

Hello

I'd like to know what the right way is to access an item in a row as
returned by a database:

=
import apsw

connection=apsw.Connection("test.sqlite")
cursor=connection.cursor()

rows=cursor.execute("SELECT isbn,price FROM books WHERE price IS
NULL")
for row in rows:

#Is this right?
for isbn in row:

if isbn:
print "Found price = " + price

connection.close(True)
=

Thank you.
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Philip Semanchuk wrote:
> On Oct 25, 2008, at 7:53 AM, Michael Sparks wrote:
>> Glenn Linderman wrote:
>>> In the module multiprocessing environment could you not use shared
>>> memory, then, for the large shared data items?
>>
>> If the poshmodule had a bit of TLC, it would be extremely useful for
>> this,... http://poshmodule.sourceforge.net/
> 
> Last time I checked that was Windows-only. Has that changed?

I've only tested it under Linux where it worked, but does clearly need a bit
of work :)

> The only IPC modules for Unix that I'm aware of are one which I
> adopted (for System V semaphores & shared memory) and one which I
> wrote (for POSIX semaphores & shared memory).
> 
> http://NikitaTheSpider.com/python/shm/
> http://semanchuk.com/philip/posix_ipc/

I'll take a look at those - poshmodule does need a bit of TLC and doesn't
appear to be maintained.

> If anyone wants to wrap POSH cleverness around them, go for it! If
> not, maybe I'll make the time someday.

I personally don't have the time do do this, but I'd be very interested in
hearing someone building an up-to-date version. (Indeed, something like
this would be extremely useful for everyone to have in the standard library
now that the multiprocessing library is in the standard library)


Michael.
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python barcode decoding

[Marco Bizzarri]

On Fri, Oct 24, 2008 at 6:05 PM, Robocop <[EMAIL PROTECTED]> wrote:
> Does anyone know of any decent (open source or commercial) python
> barcode recognition tools or libraries.  I need to read barcodes from
> pdfs or images, so it will involve some OCR algorithm.  I also only
> need to read the code 93 symbology, so it doesn't have to be very
> fancy.  The most important thing to me is that it outputs in some
> python friendly way, or ideally that it is written in python.  Any
> tips would be great!
> --
> http://mail.python.org/mailman/listinfo/python-list
>


If you don't mind using a commercial Java lib, use
http://www.tasman.co.uk/bars/readme.html. I'ts not exactly cheap, but
it works very well, and support is good.

The only suggestion I can give you is: stay away from mixed
OCR/Barcode recognition software; usually they are much slower.

Regards

-- 
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
--
http://mail.python.org/mailman/listinfo/python-list

Re: Urllib vs. FireFox

[Gilles Ganault]

On Fri, 24 Oct 2008 13:15:49 -0700 (PDT), Mike Driscoll
<[EMAIL PROTECTED]> wrote:
>On Oct 24, 2:53 pm, Rex <[EMAIL PROTECTED]> wrote:
>> By the way, if you're doing non-trivial web scraping, the mechanize
>> module might make your work much easier. You can install it with
>> easy_install.http://wwwsearch.sourceforge.net/mechanize/
>
>Or if you just need to query stuff on Amazon, then you might find this
>module helpful:
>
>http://pypi.python.org/pypi/Python-Amazon/

Thanks a bunch. I didn't know about the AWS service.
--
http://mail.python.org/mailman/listinfo/python-list

Re: [newbie] Right way to access item in array?

[Diez B. Roggisch]

Gilles Ganault wrote:

> Hello
> 
> I'd like to know what the right way is to access an item in a row as
> returned by a database:
> 
> =
> import apsw
> 
> connection=apsw.Connection("test.sqlite")
> cursor=connection.cursor()
> 
> rows=cursor.execute("SELECT isbn,price FROM books WHERE price IS
> NULL")
> for row in rows:
> 
> #Is this right?
> for isbn in row:

No. This will iterate though all columns, not just the isbn.

You can use tuple-unpacking in such cases:

for row in rows:
   isbn, price = row
   ...


Diez
--
http://mail.python.org/mailman/listinfo/python-list

Re: Database specialized in storing directed graphs?

[M.-A. Lemburg]

On 2008-10-28 01:32, Carl Banks wrote:
> I was wondering if anyone had any advice on this.
> 
> This is not to study graph theory; I'm using the graph to represent a
> problem domain.  The graphs could be arbitrarily large, and could
> easily have millions of nodes, and most nodes have a substantial
> amount of data associated with them.  Obviously I don't want a whole
> such graph in memory at once, so libraries the just deal with in-
> memory graphs are out.
> 
> I know I could implement this with a relational DB, and I'd be fine
> with a library that was built on top of one.  But I'm hoping for
> specialzed behavior useful for graphs.
> 
> For example, suppose I have a set of nodes called A.  It would be
> useful to know if any of these nodes are connected by paths that
> include no nodes in A.  I could, of course, do that by reading from
> the database and following the paths, but I clearly don't want to do
> that.  I would want instead to somehow cache the outside connectedness
> relationship between different nodes of A.  But then what happens if
> something changes elsewhere.  How is the cache for A notified, and can
> it be updated efficiently using graph theorems, rather than
> regenerated?
> 
> It's very tricky; that's why I hope someone else has done it.
> 
> I'm guessing no.

Aaron Watters is an expert on this and has implemented kjbuckets
for doing this in memory:

http://gadfly.sourceforge.net/kjbuckets.html

Gadfly uses the library to implement relational queries (and works
on disk):

http://gadfly.sourceforge.net/

The package is now maintained by Richard Jones.

You might be able to reuse some parts of Gadfly for your
purposes.

Also have a look at Pygr:

http://bioinfo.mbi.ucla.edu/pygr

which is a Python library to build a graph interface on top of
a relational database.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 28 2008)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


 Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python barcode decoding

[Wolfgang Draxinger]

Robocop wrote:

> Any tips would be great!

The open source OCR software 'gocr' does also implement some 1D
barcode recognition. In my experience it does this better then
recognizing the text:
http://jocr.sourceforge.net/

I know about only one open source library for the DataMatrix 2D
code, but this certain library is excellent:
http://www.libdmtx.org/

Wolfgang Draxinger
-- 
E-Mail address works, Jabber: [EMAIL PROTECTED], ICQ: 134682867

--
http://mail.python.org/mailman/listinfo/python-list

Python 2.5 + sqlite full text search possible?

[Guillermo]

Hi!

Is it possible to load the full-text search module for the SQLite
version bundled with  Python 2.5?

I've tested it and the stand-alone SQLite distribution doesn't seem to
include it (sqlite> .load fts2), nor does Python.

I'm on Windows XP.

Regards,

Guillermo
--
http://mail.python.org/mailman/listinfo/python-list

Re: [newbie] Right way to access item in array?

[Gerhard Häring]

Diez B. Roggisch wrote:

Gilles Ganault wrote:


Hello

I'd like to know what the right way is to access an item in a row as
returned by a database:

=
import apsw

connection=apsw.Connection("test.sqlite")
cursor=connection.cursor()

rows=cursor.execute("SELECT isbn,price FROM books WHERE price IS
NULL")
for row in rows:

#Is this right?
for isbn in row:


No. This will iterate though all columns, not just the isbn.

You can use tuple-unpacking in such cases:

for row in rows:
   isbn, price = row
   ...


You can do it even in one step with APSW (and pysqlite, and others):

for isbn, price in cur.execute("select isbn, price ..."):
 

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 2.5 + sqlite full text search possible?

[Gerhard Häring]

Guillermo wrote:

Hi!

Is it possible to load the full-text search module for the SQLite
version bundled with  Python 2.5? [...]
I'm on Windows XP.


Yes, it's possible. But not easily.

You have to replace the sqlite3.dll that comes with Python 2.5 with one 
that includes fulltext search. If you can't or don't want to build it 
yourself, you could perhaps kindly ask on the SQLite mailing list.


The DLL has to include fulltext search already, because only the very 
latest release of pysqlite wraps the method to enable loading of SQLite 
extensions. But still then, you'd need to have a custom built fts2.dll 
to load it into SQLite.


-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

mySQL problems

[Alfons Nonell-Canals]

Hello,
I've developed a program using python that have to connect to a mysql
server several times.

In a local machine (running the program in the same machine where the
mysql server is) I have no problems. I can run several instances of the
program at the same time with no problem.

Them problem arrives when I try to distribute the tasks to different
machines, for example, in a cluster. In this situation, the machine that
runs the python program is different than the machine that hosts the
database. 

If I run the python program only an other machine, only one run of the
program, there is no problem. But, If I run the program in different
machines, all of them connecting to the same database server, randomly,
it crash. Crash all running programs! at the same time.

The output is clear:

  File "/cgl/programs/ChemBang/config.py", line 67, in mysql
db = MySQLdb.connect(host=HOST, user="USER",
passwd="PASSWD",db="DATABASE")
  File "/usr/local/lib/python2.5/site-packages/PIL/__init__.py", line
74, in Connect

  File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 170,
in __init__
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL
server on 'HOST' (110)")


It is clearly related with the connection to the mySQL server. As I
said, it only happens when there are more than one programs running in
different machines (or in the same) but conecting to a mysql server that
is in an other machine. And... it is random, sometimes happens and
sometimes no...

Any idea?

Thanks!

Regards,
Alfons.

-- 

Alfons Nonell-Canals, PhD
Chemogenomics Lab
Research Group on Biomedical Informatics (GRIB) - IMIM/UPF
Barcelona Biomedical Research Park (PRBB)
C/ Doctor Aiguader, 88 - 08003 Barcelona 
[EMAIL PROTECTED] - http://cgl.imim.es
Tel. +34933160528

http://alfons.elmeuportal.cat
http://www.selenocisteina.info

--
http://mail.python.org/mailman/listinfo/python-list

Re: mySQL problems

[Tino Wildenhain]

Hi,

Alfons Nonell-Canals wrote:

Hello,
I've developed a program using python that have to connect to a mysql
server several times.

In a local machine (running the program in the same machine where the
mysql server is) I have no problems. I can run several instances of the
program at the same time with no problem.

Them problem arrives when I try to distribute the tasks to different
machines, for example, in a cluster. In this situation, the machine that
runs the python program is different than the machine that hosts the
database. 


If I run the python program only an other machine, only one run of the
program, there is no problem. But, If I run the program in different
machines, all of them connecting to the same database server, randomly,
it crash. Crash all running programs! at the same time.

The output is clear:




_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL
server on 'HOST' (110)")


It is clearly related with the connection to the mySQL server. As I
said, it only happens when there are more than one programs running in
different machines (or in the same) but conecting to a mysql server that
is in an other machine. And... it is random, sometimes happens and
sometimes no...


Maybe there is a certain connection limit which hits if your client
programms reconnect in random order? I saw this on some PHP web sites.

Probably some configuration adjustments and persistent connections
could help you in this situation.

Also, was the choice of MySQL a conscious one?

Regards
Tino


smime.p7s
Description: S/MIME Cryptographic Signature
--
http://mail.python.org/mailman/listinfo/python-list

Need advice/suggestion for small project

[AEB]

Hello, my supervisor has requested that I write a program to display
2D waves in 3D using Python.  Basically I have a time series file that
has the heights of the wave over time steps, and I want to extrude
these waves lengthwise so that they appear in a 3D manor.  I am not
very familiar with Python development, could someone point me in a
good direction, and maybe comment if this is a little to ambitious for
a first time python user?

I do have moderate programming experience in other languages (java, c/c
++, vb.net).
--
http://mail.python.org/mailman/listinfo/python-list

Re: Need advice/suggestion for small project

[Marc 'BlackJack' Rintsch]

On Tue, 28 Oct 2008 05:47:32 -0700, AEB wrote:

> Hello, my supervisor has requested that I write a program to display 2D
> waves in 3D using Python.  Basically I have a time series file that has
> the heights of the wave over time steps, and I want to extrude these
> waves lengthwise so that they appear in a 3D manor.  I am not very
> familiar with Python development, could someone point me in a good
> direction, and maybe comment if this is a little to ambitious for a
> first time python user?

This doesn't sound so much like a Python problem but more a graphics 
programming problem in general and depends on the way you want to display 
the data.  OpenGL?  Tkinter canvas?

> I do have moderate programming experience in other languages (java, c/c
> ++, vb.net).

If you are new to Python you should work through the tutorial in the 
documentation.  For people with programming experience in other languages 
`Dive into Python` is a recommended free Python text.

Ciao,
Marc 'BlackJack' Rintsch
--
http://mail.python.org/mailman/listinfo/python-list

Re: Ordering python sets

[Steven D'Aprano]

On Mon, 27 Oct 2008 17:03:58 -0700, Glenn Linderman wrote:

>>> A little harder question is how to create a key that corresponds to
>>> ascending string followed by descending string?
>>>
>>> To do that you can sort the data two times, relying on the stable
>>> nature of the Python sort.
>>> 
>>> 
> Ick. Costs twice as much.

But that's only a constant cost. For small amounts of data, it's trivial, 
and for large amounts, it's dominated by the cost of sorting N items, 
which is O(N log N) by memory. (Python's sort is a modified heap sort, I 
believe.)


[...]
> This solution seems to be the solution I was looking for.  I think "Neg"
> is a poor name, but all the good names are significantly longer... Given
> its usage, though, in a sorted call, "Rev" would be a better poor name!
> ReversedCompare might be a better long name.
> 
> How can I suggest adding it to the documentation for Python 3.0 ?  It is
> much preferable to the "obvious" solution of sorting twice, and doing
> that is a trap which many people might fall into.  I posted here
> instead, and thank you for the solution.


Why do you call it a trap? You're preferred solution is significantly 
slower for small amounts of data than the solution you call a trap:

class Neg(object):
def __init__(self, value):
self.value = value
def __cmp__(self, other):
return -cmp(self.value, other.value)

def sort_updown1(pairs):
return sorted(pairs, key=lambda (s, t): (s, Neg(t)))

import operator
def sort_updown2(pairs):
pairs = pairs[:]
pairs.sort(key=operator.itemgetter(1), reverse=True)
pairs.sort(key=operator.itemgetter(0))
return pairs

pairs = [('al', 'boo'), ('zul', 'ao'), ('al', 'den'), ('zul', 'o')]
from timeit import Timer
t1 = Timer("sort_updown1(pairs)", 
"from __main__ import sort_updown1, pairs")
t2 = Timer("sort_updown2(pairs)", 
"from __main__ import sort_updown2, pairs")


And the results:

>>> t1.repeat(number=1000)
[0.053807973861694336, 0.053076028823852539, 0.052966117858886719]
>>> t2.repeat(number=1000)
[0.022480964660644531, 0.022369861602783203, 0.02264404296875]

The difference is even more significant for larger amounts of data:

>>> pairs = pairs*100
>>> t1.repeat(number=1000)
[5.3583409786224365, 4.6985390186309814, 4.6953370571136475]
>>> t2.repeat(number=1000)
[1.1064291000366211, 1.0017910003662109, 0.48421788215637207]

And more significant still as we sort even more data:

>>> pairs = pairs*10
>>> t1.repeat(number=1000)
[53.255411863327026, 53.792617082595825, 53.549386024475098]
>>> t2.repeat(number=1000)
[5.8788919448852539, 5.0701780319213867, 4.8528299331665039]


Or, tabulating the results:

N  = 4  4004000

t1 = 0.05   4.653.3
t2 = 0.02   0.5 4.9

As you can see, the supposed "trap" of sorting twice is significantly 
faster, by an order of magnitude, than sorting it once. It would be even 
faster if I didn't bother making a copy of the list before sorting.



-- 
Steven
--
http://mail.python.org/mailman/listinfo/python-list

Sorting a list

[RC]

unsortedList = list(["XYZ","ABC"])

sortedList = unsortedList.sort()
print sortedList


Why this return None?
How do I get return as ["ABC", "XYZ"]?
--
http://mail.python.org/mailman/listinfo/python-list

Re: mySQL problems

[Alfons Nonell-Canals]

Hi,


> Maybe there is a certain connection limit which hits if your client
> programms reconnect in random order? I saw this on some PHP web sites.
> 
> Probably some configuration adjustments and persistent connections
> could help you in this situation.

I know I can check it but I am not the admin of the server where the
mySQL server is and before to check it with the sysadmin I would like to
discard other possibilities (they ask me to do this before) :-(

> Also, was the choice of MySQL a conscious one?

Yes, the choice of MySQL is conscious because I have to manage huge
amounts of data and I already have some experience in mySQL which makes
it easier.

Regards,
Alfons.


> 
> Regards
> Tino
-- 

Alfons Nonell-Canals, PhD
Chemogenomics Lab
Research Group on Biomedical Informatics (GRIB) - IMIM/UPF
Barcelona Biomedical Research Park (PRBB)
C/ Doctor Aiguader, 88 - 08003 Barcelona 
[EMAIL PROTECTED] - http://cgl.imim.es
Tel. +34933160528

http://alfons.elmeuportal.cat
http://www.selenocisteina.info

--
http://mail.python.org/mailman/listinfo/python-list

Re: [newbie] Right way to access item in array?

[Steve Holden]

Gilles Ganault wrote:
> Hello
> 
> I'd like to know what the right way is to access an item in a row as
> returned by a database:
> 
> =
> import apsw
> 
> connection=apsw.Connection("test.sqlite")
> cursor=connection.cursor()
> 
> rows=cursor.execute("SELECT isbn,price FROM books WHERE price IS
> NULL")

If you are dealing with a DB API-compliant module then the return value
from the cursor's execute method is undefined, and you need to call one
of the "fetch" methods to extract the retrieved data.

So you would want something like

cursor.execute("SELECT isbn,price FROM books WHERE price IS NULL")
rows =  cursor.fetchall()
for isbn, price in rows:
print isbn, ":", price

Once you get that working you can do your own computations with isbn and
price. Note, however, that the specific query you use guarantees that
the value of "proce" will be None, since you only retrieve the rows
where price is NULL!

> for row in rows:
> 
>   #Is this right?
>   for isbn in row:
> 
>   if isbn:
>   print "Found price = " + price
> 
> connection.close(True)
>
regards
 Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

--
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting a list

[Tino Wildenhain]

RC wrote:

unsortedList = list(["XYZ","ABC"])

sortedList = unsortedList.sort()
print sortedList


Why this return None?


Because you did not read the documentation.

Regards
Tino


smime.p7s
Description: S/MIME Cryptographic Signature
--
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting a list

[Steve Holden]

RC wrote:
> unsortedList = list(["XYZ","ABC"])
> 
> sortedList = unsortedList.sort()
> print sortedList
> 
> 
> Why this return None?
> How do I get return as ["ABC", "XYZ"]?

The list's .sort method returns None because it modifies the list
in-place, so after the call it is sorted. So you can either not assign
the list and use unsortedList (whose name now belies its sorted
condition) or you can use

sprtedList = sorted(unsortedList)

since the sorted() function returns a sorted copy of its argument.

regards
 Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

--
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting a list

[J Sisson]

To expand on Tino's response, sort() sorts in place and does not *return* a
sorted copy of the list.

In other words:

unsortedList = list(["XYZ","ABC"])
unsortedList.sort()
print sortedList

is correct.  Since sort() returns None, you lose your list if you do:

unsortedList = unsortedList.sort()

Jonathon
On Tue, Oct 28, 2008 at 8:59 AM, Tino Wildenhain <[EMAIL PROTECTED]> wrote:

> RC wrote:
>
>> unsortedList = list(["XYZ","ABC"])
>>
>> sortedList = unsortedList.sort()
>> print sortedList
>>
>>
>> Why this return None?
>>
>
> Because you did not read the documentation.
>
> Regards
> Tino
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>


-- 
Computers are like air conditioners...
They quit working when you open Windows.
--
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary

[Steve Holden]

Scott David Daniels wrote:
> Hendrik van Rooyen wrote:
>> ...
>>
>> You had Compilers!
>> You had Compiler Vendors!
>>
>> When I was lad, we had nowt but raw hardware.
>> We had to sit in cold room, ears deafened by
>> whine of fan, clicking switches to load our
>> octal in computer. We just had error light...
> 
> You had octal!  We just had oscilloscope traces shifting up and down.
> If there wasn't a metal mask over the oscilloscope with marks where
> the bits were, we'd have been in a world of hurt.
> 
You were lucky! We had to imagine where the bits would have been if we'd
had them. We only use to see real bits on Easter Sundays, if we'd been
good all year. T'rest o' t'time we 'ad ter make do wi' pieces of stone.

Bizarre though it may sound in this age of integrated circuits there
really was a storage device that used a cathode ray tube to store (IIRC)
a kilobit of information. It detected, by the use of a capacitance plate
on the front of the tube, whether the bits in each position were 1 or 0
by determining whether the beam was focused or defocused, since each had
a different effect on the charge of the plate. It was a dynamic device,
and only retained the information by writing back what was read
continuously, just like any other dynamic memory.

yorkshireman-ly y'rs  - steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

--
http://mail.python.org/mailman/listinfo/python-list

Re: Sorting a list

[Brian Blais]

On Oct 28, 2008, at 9:45 , RC wrote:


unsortedList = list(["XYZ","ABC"])

sortedList = unsortedList.sort()
print sortedList



the sort method is in-place, so it modifies the object which calls  
it, and doesn't return anything:


In [1]:unsortedList = list(["XYZ","ABC"])

In [2]:sortedList = unsortedList.sort()

In [3]:print sortedList
None

In [4]:print unsortedList
['ABC', 'XYZ']

or, better, just:

In [5]:unsortedList = list(["XYZ","ABC"])

In [6]:unsortedList.sort()

In [7]:print unsortedList
['ABC', 'XYZ']


bb

--
Brian Blais
[EMAIL PROTECTED]
http://web.bryant.edu/~bblais



--
http://mail.python.org/mailman/listinfo/python-list

Re: mySQL problems

[Tino Wildenhain]

Alfons Nonell-Canals wrote:

Hi,



Maybe there is a certain connection limit which hits if your client
programms reconnect in random order? I saw this on some PHP web sites.

Probably some configuration adjustments and persistent connections
could help you in this situation.


I know I can check it but I am not the admin of the server where the
mySQL server is and before to check it with the sysadmin I would like to
discard other possibilities (they ask me to do this before) :-(


Well in this case just open as many connections in a loop
as you can and watch for the outcome.




Also, was the choice of MySQL a conscious one?


Yes, the choice of MySQL is conscious because I have to manage huge
amounts of data and I already have some experience in mySQL which makes
it easier.


But with no other open source database I presume? Watch out! There are
lots of diamonds beside your way :-)

Regards
Tino



smime.p7s
Description: S/MIME Cryptographic Signature
--
http://mail.python.org/mailman/listinfo/python-list

Re: [newbie] Right way to access item in array?

[Gilles Ganault]

On Tue, 28 Oct 2008 12:12:26 +0100, Gerhard Häring <[EMAIL PROTECTED]>
wrote:
>You can do it even in one step with APSW (and pysqlite, and others):
>
>for isbn, price in cur.execute("select isbn, price ..."):

Thanks much guys. For those interested, here's some working code:

==
import apsw

connection=apsw.Connection("mybooks.sqlite")
cursor=connection.cursor()

for id, isbn in list(cursor.execute("SELECT id,isbn FROM books WHERE
price IS NULL")):

print isbn

connection.close(True)
==

HTH,
--
http://mail.python.org/mailman/listinfo/python-list

Re: [newbie] Right way to access item in array?

[Gilles Ganault]

On Tue, 28 Oct 2008 09:56:11 -0400, Steve Holden <[EMAIL PROTECTED]>
wrote:
>If you are dealing with a DB API-compliant module then the return value
>from the cursor's execute method is undefined, and you need to call one
>of the "fetch" methods to extract the retrieved data.

Thanks for pointing it out. I read that calling list() on the dataset
is the way to go with this module:

for id, isbn in list(cursor.execute("SELECT id,isbn FROM books WHERE
price IS NULL")):
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 26, 10:11 pm, "James Mills" <[EMAIL PROTECTED]>
wrote:
> On Mon, Oct 27, 2008 at 12:03 PM, Andy O'Meara <[EMAIL PROTECTED]> wrote:
> > I think we miscommunicated there--I'm actually agreeing with you.  I
> > was trying to make the same point you were: that intricate and/or
> > large structures are meant to be passed around by a top-level pointer,
> > not using and serialization/messaging.  This is what I've been trying
> > to explain to others here; that IPC and shared memory unfortunately
> > aren't viable options, leaving app threads (rather than child
> > processes) as the solution.
>
> Andy,
>
> Why don't you just use a temporary file
> system (ram disk) to store the data that
> your app is manipulating. All you need to
> pass around then is a file descriptor.
>
> --JamesMills

Unfortunately, it's the penalty of serialization and unserialization.
When you're talking about stuff like memory-resident images and video
(complete with their intricate and complex codecs), then the only
option is to be passing around a couple pointers rather then take the
hit of serialization (which is huge for video, for example).  I've
gone into more detail in some other posts but I could have missed
something.


Andy



--
http://mail.python.org/mailman/listinfo/python-list

parsing MS word docs -- tutorial request

[bp . tralfamadore]

All,

I am trying to write a script that will parse and extract data from a
MS Word document.  Can / would anyone refer me to a tutorial on how to
do that?  (perhaps from tables).  I am aware of, and have downloaded
the pywin32 extensions, but am unsure of how to proceed -- I'm not
familiar with the COM API for word, so help for that would also be
welcome.

Any help would be appreciated.  Thanks for your attention and
patience.

::bp::
--
http://mail.python.org/mailman/listinfo/python-list

Python 2.5 + sqlite full text search possible?

[Guillermo]

Hi!

Is it possible to use the full-text module of SQLite with the sqlite3
module? I've done a bit of investigation and it seems the stand-alone
distribution of SQLite is compiled without it, and so does the version
bundled with Python.

Regards,

Guillermo
--
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary

[Steve Holden]

D'Arcy J.M. Cain wrote:
> On Tue, 28 Oct 2008 10:06:31 -0400
> Steve Holden <[EMAIL PROTECTED]> wrote:
>> Bizarre though it may sound in this age of integrated circuits there
>> really was a storage device that used a cathode ray tube to store (IIRC)
>> a kilobit of information. It detected, by the use of a capacitance plate
> 
> A kilobit?  One tube would carry one bit.  Imagine the size of today's
> computers if we were still using them.  In those days you were lucky to
> get 4 Kbytes of core memory.
> 
> In those days they would have techs walking back and forth along
> pathways inside the memory banks with shopping carts full of tubes
> replacing them as they burned out.  Programs had to be prepared to deal
> with the fact that bits could go dead at any time and functions would
> run multiple times and hold an election to determine the correct answer.
> 
Sorry, you are thinking of bistable multivibrators. I was talking about
the Wiliams tube store:

  http://ed-thelen.org/comp-hist/SEAC-Williams-tube-desc.html

which I now see stored a half-kilobit in its SEAC implementation. They
built them to be cheaper and more compact than the single-tube bit store
(the single tube actually held two triodes in it), which would be used
for accumulators and the like - the fast stuff.

I don't ever remember programming to cope with equipment failure,
however. Did you make that bit up?

regards
 Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

--
http://mail.python.org/mailman/listinfo/python-list

How to get an object's name as a string?

[Shannon Mayne]

I would like to create objects with algorithmically determined names
based on other object names and use object names for general algorithm
input.

How would one extract the name of an object from an object instance as
a string. I would think that it is stored as an attribute of the
object but successive 'dir()' calles haven't found me the attribute
with the namestring.

My thanks!

--
http://mail.python.org/mailman/listinfo/python-list

Re: Unpacking byte strings from a file of unknown size

[Mark]

Thanks I tested your solution and that works.

One of the things that didn't work was
for chunk in myfile.read(10):
info1, info2, info3 = struct.unpack('http://mail.python.org/mailman/listinfo/python-list

parse a table in HTML page.

[antonio_wn8]

Hi all,
I have a need to read and parse a table in HTML page.

I’m using the following script:
http://trac.davidgrant.ca/browser/src/python/misc/siteuptime/TableParser.py

It works fine  aside from  link in href.

Example:

String to parse:
elognormal text

Output:
[[['elog', 'normal text']]]

as you can see it misses the info about href...
how can get this information 'vaffa.html'?

thanks,
Antonella
--
http://mail.python.org/mailman/listinfo/python-list

Unit Testing: a couple of questions

[Emanuele D'Arrigo]

Hi everybody,

I'm just having a go with Unit Testing for the first time and my
feeling about it in short is: Neat!

I'm a bit worried about the time it's taking me to develop the tests
but after only a day or so I'm already much faster than when I started
with it and the code is already much improved in terms of robustness.
A couple of "philosophical" questions have emerged in the process.

1) Granularity
Given a simple class

class myClass():
def __init__(self, data):
__data = data;

def getData(self):
return __data

def setData(self, data):
 __data = data

I've been wondering: where do I stop in terms of testing for things
that could go wrong? In this case for example, it might be reasonable
to expand the class to make sure it only receives integers and test
accordingly, i.e.:

def setData(self, data):
try:
 data = int(data)
except ValueError:
raise ValueError("Argument received cannot be converted to
integer: " + data)

But would it be reasonable to test also for the assignment operators?
After all, if, for some strange reason, there isn't enough memory,
couldn't even __data = data potentially fail?

2) Testing in isolation
I'm not entirely clear on this point. I can see how I need to test
each path of the program flow separately. But should a test -only-
rely on the object being tested and mock ones in supporting roles?
I.e. would this be wrong if SupportObject is not a mockup?

def testObjectToBeTested _forReallyBadError(self):
supportObject = SupportObject()
objectToBeTested = ObjectToBeTested()
result =  objectToBeTested.addSupportObject(supportObject)
self.failIf(result != kSuccess, "Support Object could not be
added!")

I can see how if the SupportObject class had a bug introduced in it,
this test would fail even though it has nothing to do with the
ObjectToBeTested class. However, creating mock objects can be quite an
overhead (?). I'm wondering if there is a threshold, even a fuzzy one,
under which it isn't worth doing and a test like the one above is
"good enough".

What do you guys think?

Manu
--
http://mail.python.org/mailman/listinfo/python-list

Re: How to get an object's name as a string?

[Larry Bates]

Shannon Mayne wrote:

I would like to create objects with algorithmically determined names
based on other object names and use object names for general algorithm
input.

How would one extract the name of an object from an object instance as
a string. I would think that it is stored as an attribute of the
object but successive 'dir()' calles haven't found me the attribute
with the namestring.

My thanks!



Once again (there have been many posts on this subject).  Objects can have more 
than one name in Python.  Therefore there is not a one-to-one correspondence 
between an object instance and name(s) that point to it.


Example:

a = myObject()
b = a
c = a

now a, b, c all point to the same object.  How would you define it's "name".

You are certainly free to store a name as an attribute to the object, but the 
linkage would then be purely logical.


Example:

objects = []
objects.append(myObject('a'))
#
# Find object with name == 'a'
#
obj = None
for object in objects:
if object.name == 'a':
   obj = object


-Larry
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 27, 4:05 am, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Andy O'Meara wrote:


>
> > Well, when you're talking about large, intricate data structures
> > (which include opaque OS object refs that use process-associated
> > allocators), even a shared memory region between the child process and
> > the parent can't do the job.  Otherwise, please describe in detail how
> > I'd get an opaque OS object (e.g. an OS ref that refers to memory-
> > resident video) from the child process back to the parent process.
>
> WHAT PARENT PROCESS? "In the same address space", to me, means
> "a single process only, not multiple processes, and no parent process
> anywhere". If you have just multiple threads, the notion of passing
> data from a "child process" back to the "parent process" is
> meaningless.

I know...  I was just responding to you and others here keep beating
the "fork" drum.  I just trying make it clear that a shared address
space is the only way to go.  Ok, good, so we're in agreement that
threads is the only way to deal with the "intricate and complex" data
set issue in a performance-centric application.

>
> > Again, the big picture that I'm trying to plant here is that there
> > really is a serious need for truly independent interpreters/contexts
> > in a shared address space.
>
> I understand that this is your mission in this thread. However, why
> is that your problem? Why can't you just use the existing (limited)
> multiple-interpreters machinery, and solve your problems with that?

Because then we're back into the GIL not permitting threads efficient
core use on CPU bound scripts running on other threads (when they
otherwise could).  Just so we're on the same page, "when they
otherwise could" is relevant here because that's the important given:
that each interpreter ("context") truly never has any context with
others.

An example would be python scripts that generate video programatically
using an initial set of params and use an in-house C module to
construct frame (which in turn make and modify python C objects that
wrap to intricate codec related data structures).  Suppose you wanted
to render 3 of these at the same time, one on each thread (3
threads).  With the GIL in place, these threads can't anywhere close
to their potential.  Your response thus far is that the C module
should release the GIL before it commences its heavy lifting.  Well,
the problem is that if during its heavy lifting it needs to call back
into its interpreter.  It's turns out that this isn't an exotic case
at all: there's a *ton* of utility gained by making calls back into
the interpreter. The best example is that since code more easily
maintained in python than in C, a lot of the module "utility" code is
likely to be in python.  Unsurprisingly, this is the situation myself
and many others are in: where we want to subsequently use the
interpreter within the C module (so, as I understand it, the proposal
to have the C module release the GIL unfortunately doesn't work as a
general solution).

>
> > For most
> > industry-caliber packages, the expectation and convention (unless
> > documented otherwise) is that the app can make as many contexts as its
> > wants in whatever threads it wants because the convention is that the
> > app is must (a) never use one context's objects in another context,
> > and (b) never use a context at the same time from more than one
> > thread.  That's all I'm really trying to look at here.
>
> And that's indeed the case for Python, too. The app can make as many
> subinterpreters as it wants to, and it must not pass objects from one
> subinterpreter to another one, nor should it use a single interpreter
> from more than one thread (although that is actually supported by
> Python - but it surely won't hurt if you restrict yourself to a single
> thread per interpreter).
>

I'm not following you there...  I thought we're all in agreement that
the existing C modules are FAR from being reentrant, regularly making
use of static/global objects. The point I had made before is that
other industry-caliber packages specifically don't have restrictions
in *any* way.

I appreciate your arguments these a PyC concept is a lot of work with
some careful design work, but let's not kill the discussion just
because of that.  The fact remains that the video encoding scenario
described above is a pretty reasonable situation, and as more people
are commenting in this thread, there's an increasing need to offer
apps more flexibility when it comes to multi-threaded use.


Andy




--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Greg Ewing]

Glenn Linderman wrote:

So your 50% number is just a scare tactic, it would seem, based on wild 
guesses.  Was there really any benefit to the comment?


All I was really trying to say is that it would be a
mistake to assume that the overhead will be negligible,
as that would be just as much a wild guess as 50%.

--
Greg

--
http://mail.python.org/mailman/listinfo/python-list

ignoring NAs in a dataset with numpy

[mrafi]

Hey All,
I am working with numpy 
I have a data set with a lot of nan values, and i want to calculate standard
deviation
is there a direct function to do it for example nansum(or something like
this),
to calculate standard deviation ignoring nan values??
Thanks
Rafi
-- 
View this message in context: 
http://www.nabble.com/ignoring-NAs-in-a-dataset-with-numpy-tp20202823p20202823.html
Sent from the Python - python-list mailing list archive at Nabble.com.

--
http://mail.python.org/mailman/listinfo/python-list

list versions of all installed modules

[John [H2O]]

Is there a quick way to list the version of each installed module? 
-- 
View this message in context: 
http://www.nabble.com/list-versions-of-all-installed-modules-tp20204095p20204095.html
Sent from the Python - python-list mailing list archive at Nabble.com.

--
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary

[D'Arcy J.M. Cain]

On Tue, 28 Oct 2008 10:06:31 -0400
Steve Holden <[EMAIL PROTECTED]> wrote:
> Bizarre though it may sound in this age of integrated circuits there
> really was a storage device that used a cathode ray tube to store (IIRC)
> a kilobit of information. It detected, by the use of a capacitance plate

A kilobit?  One tube would carry one bit.  Imagine the size of today's
computers if we were still using them.  In those days you were lucky to
get 4 Kbytes of core memory.

In those days they would have techs walking back and forth along
pathways inside the memory banks with shopping carts full of tubes
replacing them as they burned out.  Programs had to be prepared to deal
with the fact that bits could go dead at any time and functions would
run multiple times and hold an election to determine the correct answer.

-- 
D'Arcy J.M. Cain <[EMAIL PROTECTED]> |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP)   |  what's for dinner.
--
http://mail.python.org/mailman/listinfo/python-list

Re: How to get an object's name as a string?

[Joe Strout]

On Oct 28, 2008, at 8:41 AM, Shannon Mayne wrote:


I would like to create objects with algorithmically determined names
based on other object names and use object names for general algorithm
input.


What do you mean by the "name" of an object?  Objects don't generally  
have names, unless you explicitly define a .name property and assign  
them names.


(Variables have names, of course, but a variable isn't an object --  
it's just a reference to an object.  Many variables may refer to the  
same object, so it doesn't make any sense to ask for the name of THE  
variable which may be referring to an object at the moment.)



How would one extract the name of an object from an object instance as
a string.  I would think that it is stored as an attribute of the
object but successive 'dir()' calles haven't found me the attribute
with the namestring.


As noted above, there is no built-in name attribute.  Define one,  
perhaps like this:


class Foo():
 def __init__(name):
   self.name = name

Now your Foo objects have a name attribute, and if "x" is a reference  
to such an object, you would access that as "x.name".


It's still unclear what you intend to do with these, but if at some  
point you want to access objects by their names (from user input or  
whatever), then you'll also need a dictionary to map names to  
objects.  So to your __init__ function, you might add something like  
this:


   name_map[name] = self

where name_map was initialized to {} at the top of the file.  Then you  
can use name_map to look up any object of this class by name.   
Remember that this will keep these objects from automatically  
disappearing when there are no other references (other than the map)  
to them.  If that's a problem, explicitly remove them from the map  
when you know you're done with them, or use weak references.


Best,
- Joe

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 2.5 + sqlite full text search possible?

[Guilherme Polo]

On 10/28/08, Guillermo <[EMAIL PROTECTED]> wrote:
> Hi!
>
>  Is it possible to use the full-text module of SQLite with the sqlite3
>  module? I've done a bit of investigation and it seems the stand-alone
>  distribution of SQLite is compiled without it, and so does the version
>  bundled with Python.
>

Support for loading extensions was only added in pysqlite 2.5, so you
will have to install it.

Python 2.5 and 2.6 comes with pysqlite 2.3.2, future 3.0 will come
with pysqlite 2.4.1.


>  Regards,
>
>  Guillermo
>
> --
>  http://mail.python.org/mailman/listinfo/python-list
>


-- 
-- Guilherme H. Polo Goncalves
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 25, 9:46 am, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote:
> These discussion pop up every year or so and I think that most of them
> are not really all that necessary, since the GIL isn't all that bad.
>

Thing is, if the topic keeps coming up, then that may be an indicator
that change is truly needed.  Someone much wiser than me once shared
that a measure of the usefulness and quality of a package (or API) is
how easily it can be added to an application--of any flavors--without
the application needing to change.

So in the rising world of idle cores and worker threads, I do see an
increasing concern over the GIL.  Although I recognize that the debate
is lengthy, heated, and has strong arguments on both sides, my reading
on the issue makes me feel like there's a bias for the pro-GIL side
because of the volume of design and coding work associated with
considering various alternatives (such as Glenn's "Py*" concepts).
And I DO respect and appreciate where the pro-GIL people come from:
who the heck wants to do all that work and recoding so that a tiny
percent of developers can benefit?  And my best response is that as
unfortunate as it is, python needs to be more multi-threaded app-
friendly if we hope to attract the next generation of app developers
that want to just drop python into their app (and not have to change
their app around python).  For example, Lua has that property, as
evidenced by its rapidly growing presence in commercial software
(Blizzard uses it heavily, for example).

>
> Furthermore, there are lots of ways to tune the CPython VM to make
> it more or less responsive to thread switches via the various sys.set*()
> functions in the sys module.
>
> Most computing or I/O intense C extensions, built-in modules and object
> implementations already release the GIL for you, so it usually doesn't
> get in the way all that often.


The main issue I take there is that it's often highly useful for C
modules to make subsequent calls back into the interpreter. I suppose
the response to that is to call the GIL before reentry, but it just
seems to be more code and responsibility in scenarios where it's no
necessary.  Although that code and protocol may come easy to veteran
CPython developers, let's not forget that an important goal is to
attract new developers and companies to the scene, where they get
their thread-independent code up and running using python without any
unexpected reengineering.  Again, why are companies choosing Lua over
Python when it comes to an easy and flexible drop-in interpreter?  And
please take my points here to be exploratory, and not hostile or
accusatory, in nature.


Andy


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 2.5 + sqlite full text search possible?

[Gerhard Häring]

Guillermo wrote:

Hi!

Is it possible to use the full-text module of SQLite with the sqlite3
module? I've done a bit of investigation and it seems the stand-alone
distribution of SQLite is compiled without it, 


Yes, though compiling using the amalgamation and defining 
SQLITE_ENABLE_FTS3 helps.



and so does the version bundled with Python.


True.

I'm too lazy to build a SQLite3 DLL with FTS enabled, I'm pretty sure 
those can be found on the net.


But I've just patched pysqlite with one line:

+ext.define_macros.append(("SQLITE_ENABLE_FTS3", "1"))   # 
build with fulltext search enabled


which helped its super-crazy script mkwin32.py to build Windows binaries 
 *on Linux* that fetch the SQLite amalgamation and build Windows 
binaries for Python 2.3, 2.4 and 2.5 (no 2.6, yet).


Just go here 
http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.0/win32_fts/


download and install the binary for Python 2.5 and off you go:


from pysqlite2 import dbapi2 as sqlite3

con = sqlite3.connect(":memory:")

# example from SQLite wiki
con.execute("create virtual table recipe using fts3(name, ingredients)")
con.executescript("""
insert into recipe (name, ingredients) values ('broccoli stew', 'broccoli 
peppers cheese tomatoes');
insert into recipe (name, ingredients) values ('pumpkin stew', 'pumpkin 
onions garlic celery');
insert into recipe (name, ingredients) values ('broccoli pie', 'broccoli 
cheese onions flour');
insert into recipe (name, ingredients) values ('pumpkin pie', 'pumpkin 
sugar flour butter');
""")
for row in con.execute("select rowid, name, ingredients from recipe where name match 
'pie'"):
print row


-- Gerhard


--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about scope

[Kirk Strauser]

At 2008-10-24T01:08:12Z, Pat <[EMAIL PROTECTED]> writes:

> ---> myGlobals.py file:
>
> class myGlobals():
> remote_device_enabled = bool
>
> ---> my initialize.py file:
>
> from myGlobals import *
> def initialize():
> myGlobals.remote_device_enabled = True
>
> ---> my main.py file:
>
> import from myGlobals import *
> RDE =  myGlobals.remote_device_enabled
>
> def main():
> if RDE:# this will not give me the correct value
> process_device()

If you *really* want to organize your settings like this, may I suggest:

---> myGlobals.py file:

# This does nothing more than create a placeholder
remote_device_enabled = None

---> my initialize.py file:

import myGlobals
def initialize():
myGlobals.remote_device_enabled = True

---> my main.py file:

import myGlobals
import initialize
initialize.initialize()

def main():
if myGlobals.remote_device_enabled:
process_device()


-- 
Kirk Strauser
The Day Companies
--
http://mail.python.org/mailman/listinfo/python-list

Re: Unpacking byte strings from a file of unknown size

[Scott David Daniels]

Mark wrote:

Thanks I tested your solution and that works.

One of the things that didn't work was
for chunk in myfile.read(10):
info1, info2, info3 = struct.unpack('

this code python interprets as:

data = myfile.read(10)
for chunk in data:
.


--Scott David Daniels
[EMAIL PROTECTED]
--
http://mail.python.org/mailman/listinfo/python-list

Re: Finding the instance reference of an object

[Joe Strout]

On Oct 27, 2008, at 11:28 PM, Gabriel Genellina wrote:

En Tue, 28 Oct 2008 00:58:10 -0200, greg  
<[EMAIL PROTECTED]> escribió:



Let's look at the definitions of the terms:

(1) Call by value: The actual parameter is an expression. It is
   evaluated and the result is assigned to the formal parameter.
   Subsequent assignments to the formal parameter do not affect
   the actual parameter.

(2) Call by reference: The actual parameter is an lvalue. The
   formal parameter becomes an alias for the actual parameter,
   so that assigning to the formal parameter has the same
   effect as assigning to the actual parameter.

Seems to me that (1) describes exactly how parameter passing
works in Python. So why insist that it's *not* call by value?


Greg is right on the money here.  It really is as simple as that.

Those definitions are only applicable to unstructured, primitive  
types, where the only relevant operations are "get value" and  
"assign value".


Nonsense.  They apply to any type.  See here for an introduction to  
the topic in VB.NET:


  http://www.homeandlearn.co.uk/net/nets9p4.html

The example shown uses an Integer, but guess what -- you can pass ANY  
type either ByRef or ByVal, including object references (which are, of  
course, quite common in VB.NET).  The default is ByVal, which means  
that (as in Python) the actual parameter is an expression, and no  
assignments to it can affect the actual parameter.  It DOES NOT MATTER  
that both the formal parameter and the actual parameter may refer to  
the same object, and if that object is mutable, you can mutate that  
object.


But ByRef is another option, and if you pass an object reference  
ByRef, then the formal parameter is an alias for the actual parameter,  
and assignments to it change the actual parameter.  Python doesn't  
have this feature (nor much need for it, given its other features,  
like tuple packing/unpacking).  So, parameter passing in ByRef is  
clearly exactly the same as the default "ByVal" parameter passing in  
VB.NET... as well as Java, RB, C++ (when you remember that an object  
reference in modern languages is like a pointer in C++), and every  
other OOP language I've looked into.


Some of those languages, like Python, don't have different modes, and  
so the language designers had no call to give their one mode a name.   
So let's look at those that did have such a need, like VB.NET and  
REALbasic.  What's the default mode called?  Why, it's "ByVal".  Even  
when that value is an object reference.  This is short for "by value"  
-- it's not short for "by sharing" or "by object" or any other such  
silliness.  No such terms are needed.


There are only the two cases, which Greg quite succinctly and  
accurately described above.  One is by value, the other is by  
reference.  Python quite clearly uses by value.  Parameters are  
expressions that are evaluated, and the resulting value copied into  
the formal parameter, pure and simple.  The continued attempts to  
obfuscate this is pointless and wrong.


Best,
- Joe


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python barcode decoding

[Kirk Strauser]

At 2008-10-24T17:05:25Z, Robocop <[EMAIL PROTECTED]> writes:

> Does anyone know of any decent (open source or commercial) python
> barcode recognition tools or libraries.  I need to read barcodes from
> pdfs or images, so it will involve some OCR algorithm.  I also only
> need to read the code 93 symbology, so it doesn't have to be very
> fancy.  The most important thing to me is that it outputs in some
> python friendly way, or ideally that it is written in python.  Any
> tips would be great!

How precise are these images?  I mean, are they computer-generated, or
scanned in from other material?

If they're nice and crisp, I wrote a simple and pretty quick decoder at 
http://pypi.python.org/pypi/Daycos/1.0 .  Look in the
Daycos.Imageproc.Barcode module.
-- 
Kirk Strauser
The Day Companies
--
http://mail.python.org/mailman/listinfo/python-list

Re: Need advice/suggestion for small project

[Scott David Daniels]

AEB wrote:

Hello, my supervisor has requested that I write a program to display
2D waves in 3D using Python.  Basically I have a time series file that
has the heights of the wave over time steps, and I want to extrude
these waves lengthwise so that they appear in a 3D manor.  I am not
very familiar with Python development, could someone point me in a
good direction, and maybe comment if this is a little to ambitious for
a first time python user?

I do have moderate programming experience in other languages (java, c/c
++, vb.net).

You might want to take a look at VPython.  It is probably the easiest
way to get to 3-D in Python.
--
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary

[Hendrik van Rooyen]

 "Steve Holden" <[EMAIL PROTECTED]> wrote:

> 
> I don't ever remember programming to cope with equipment failure,
> however. Did you make that bit up?

Not at the bit level in my case - but I do remember doing silly things 
like multiplying after a divide (all integer arithmetic) to make sure the 
answer was right.

And we routinely had a checksum record at the end of all files that one
would check when reading in - it was made by adding everything in every
record as a big binary number - almost a kind of vertical parity. And it 
caught stuff out too.  Then you would just re run, and the error would 
often go away.

It has left me permanently scarred.

- Hendrik

--
http://mail.python.org/mailman/listinfo/python-list

Re: list versions of all installed modules

[Gerhard Häring]

John [H2O] wrote:
Is there a quick way to list the version of each installed module? 


$ sudo easy_install yolk
$ yolk -l

-- Gerhard

--
http://mail.python.org/mailman/listinfo/python-list

Re: Unit Testing: a couple of questions

[Antoine De Groote]

I'm wondering if don't want your class to look something like this:

class myClass():
def __init__(self, data):
self.__data = data

def getData(self):
return self.__data

def setData(self, data):
 self.__data = data

For the rest I'll let the experts argue, I don't have a lot of
experience with unit testing either... Shame on me! ;-)

Regards,
antoine

Emanuele D'Arrigo wrote:
> Hi everybody,
> 
> I'm just having a go with Unit Testing for the first time and my
> feeling about it in short is: Neat!
> 
> I'm a bit worried about the time it's taking me to develop the tests
> but after only a day or so I'm already much faster than when I started
> with it and the code is already much improved in terms of robustness.
> A couple of "philosophical" questions have emerged in the process.
> 
> 1) Granularity
> Given a simple class
> 
> class myClass():
> def __init__(self, data):
> __data = data;
> 
> def getData(self):
> return __data
> 
> def setData(self, data):
>  __data = data
> 
> I've been wondering: where do I stop in terms of testing for things
> that could go wrong? In this case for example, it might be reasonable
> to expand the class to make sure it only receives integers and test
> accordingly, i.e.:
> 
> def setData(self, data):
> try:
>  data = int(data)
> except ValueError:
> raise ValueError("Argument received cannot be converted to
> integer: " + data)
> 
> But would it be reasonable to test also for the assignment operators?
> After all, if, for some strange reason, there isn't enough memory,
> couldn't even __data = data potentially fail?
> 
> 2) Testing in isolation
> I'm not entirely clear on this point. I can see how I need to test
> each path of the program flow separately. But should a test -only-
> rely on the object being tested and mock ones in supporting roles?
> I.e. would this be wrong if SupportObject is not a mockup?
> 
> def testObjectToBeTested _forReallyBadError(self):
> supportObject = SupportObject()
> objectToBeTested = ObjectToBeTested()
> result =  objectToBeTested.addSupportObject(supportObject)
> self.failIf(result != kSuccess, "Support Object could not be
> added!")
> 
> I can see how if the SupportObject class had a bug introduced in it,
> this test would fail even though it has nothing to do with the
> ObjectToBeTested class. However, creating mock objects can be quite an
> overhead (?). I'm wondering if there is a threshold, even a fuzzy one,
> under which it isn't worth doing and a test like the one above is
> "good enough".
> 
> What do you guys think?
> 
> Manu
--
http://mail.python.org/mailman/listinfo/python-list

Re: parse a table in HTML page.

[Thomas Guettler]

Have you looked at beautiful soup?
http://www.crummy.com/software/BeautifulSoup/

antonio_wn8 schrieb:
> Hi all,
> I have a need to read and parse a table in HTML page.
> 
> I’m using the following script:
> http://trac.davidgrant.ca/browser/src/python/misc/siteuptime/TableParser.py
> 
> It works fine  aside from  link in href.
> 
> Example:
> 
> String to parse:
> elognormal text
> 
> Output:
> [[['elog', 'normal text']]]
> 
> as you can see it misses the info about href...
> how can get this information 'vaffa.html'?



-- 
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
--
http://mail.python.org/mailman/listinfo/python-list

Re: list versions of all installed modules

[Scott David Daniels]

John [H2O] wrote:
Is there a quick way to list the version of each installed module? 


import sys
for name, module in sorted(sys.modules.items()):
if hasattr(module, '__version__'):
print name, module.__version__

Of course if you add __VERSION__, VERSION, and version, you
may get more.

--Scott David Daniels
[EMAIL PROTECTED]

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 27, 10:55 pm, Glenn Linderman <[EMAIL PROTECTED]> wrote:


> And I think we still are miscommunicating!  Or maybe communicating anyway!
>
> So when you said "object", I actually don't know whether you meant
> Python object or something else.  I assumed Python object, which may not
> have been correct... but read on, I think the stuff below clears it up.
>
>
> Then when you mentioned thousands of objects, I imagined thousands of
> Python objects, and somehow transforming the blob into same... and back
> again.  

My apologies to you and others here on my use of "objects" -- I'm use
the term generically and mean it to *not* refer to python objects (for
the all the reasons discussed here).  Python only makes up a small
part of our app, hence my habit of "objects" to refer to other APIs'
allocated and opaque objects (including our own and OS APIs).  For all
the reasons we've discussed, in our world, python objects don't travel
around outside of our python C modules -- when python objects need to
be passed to other parts of the app, they're converted into their non-
python (portable) equivalents (ints, floats, buffers, etc--but most of
the time, the objects are PyCObjects, so they can enter and leave a
python context with negligible overhead). I venture to say this is
pretty standard when any industry app uses a package (such as python),
for various reasons:
   - Portability/Future (e.g.  if we do decode to drop Python and go
with Lua, the changes are limited to only one region of code).
   - Sanity (having any API's objects show up in places "far away"
goes against easy-to-follow code).
   - MT flexibility (because we always never use static/global
storage, we have all kinds of options when it comes to
multithreading).  For example, recall that by throwing python in
multiple dynamic libs, we were able to achieve the GIL-less
interpreter independence that we want (albeit ghetto and a pain).



Andy



--
http://mail.python.org/mailman/listinfo/python-list

explanation of values, references, assignment, and method calls

[Joe Strout]

I've tried to write up this topic in a clear, step-by-step manner,  
with the help of diagrams and short examples from several different  
OOP languages.  I hope it will help clear up the confusion that seems  
to be pervading the Python community (and which is far more rare in  
the other language communities, despite them having the very same  
semantics).


   

It's still a bit rough around the edges, and probably contains some  
typos, as I finished it late last night and haven't even had a chance  
to proof-read it yet.  But I may not get another chance to work on it  
for a few days, so I wanted to get it out there now.  I welcome your  
feedback.


Best,
- Joe


--
http://mail.python.org/mailman/listinfo/python-list

Re: Unit Testing: a couple of questions

[Orestis Markou]

For the first bit, a colleague has recently asked the philosophical
question, "How do you test what happens when the power goes down?" :)

In other words, only test the bits that your code does. If you want to
provide type checking, then yes, you have to test that.

It's fair to assume that everything that is *not* your code will work
as expected. If you do find 3rd-party bugs that affect you, then you
should write a unit test that exposes that bug. When they fix it, your
unittest fails, prompting you to remove any workarounds you had.

On Tue, Oct 28, 2008 at 2:56 PM, Emanuele D'Arrigo <[EMAIL PROTECTED]> wrote:
> Hi everybody,
>
> I'm just having a go with Unit Testing for the first time and my
> feeling about it in short is: Neat!
>
> I'm a bit worried about the time it's taking me to develop the tests
> but after only a day or so I'm already much faster than when I started
> with it and the code is already much improved in terms of robustness.
> A couple of "philosophical" questions have emerged in the process.
>
> 1) Granularity
> Given a simple class
>
> class myClass():
>def __init__(self, data):
>__data = data;
>
>def getData(self):
>return __data
>
>def setData(self, data):
> __data = data
>
> I've been wondering: where do I stop in terms of testing for things
> that could go wrong? In this case for example, it might be reasonable
> to expand the class to make sure it only receives integers and test
> accordingly, i.e.:
>
>def setData(self, data):
>try:
> data = int(data)
>except ValueError:
>raise ValueError("Argument received cannot be converted to
> integer: " + data)
>
> But would it be reasonable to test also for the assignment operators?
> After all, if, for some strange reason, there isn't enough memory,
> couldn't even __data = data potentially fail?
>
> 2) Testing in isolation
> I'm not entirely clear on this point. I can see how I need to test
> each path of the program flow separately. But should a test -only-
> rely on the object being tested and mock ones in supporting roles?
> I.e. would this be wrong if SupportObject is not a mockup?
>
> def testObjectToBeTested _forReallyBadError(self):
>supportObject = SupportObject()
>objectToBeTested = ObjectToBeTested()
>result =  objectToBeTested.addSupportObject(supportObject)
>self.failIf(result != kSuccess, "Support Object could not be
> added!")
>
> I can see how if the SupportObject class had a bug introduced in it,
> this test would fail even though it has nothing to do with the
> ObjectToBeTested class. However, creating mock objects can be quite an
> overhead (?). I'm wondering if there is a threshold, even a fuzzy one,
> under which it isn't worth doing and a test like the one above is
> "good enough".
>
> What do you guys think?
>
> Manu
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
[EMAIL PROTECTED]
http://orestis.gr
--
http://mail.python.org/mailman/listinfo/python-list

Extracting name strings from assigned objects

[Shannon Mayne]

I would like to ask how one might obtain the assigned name of an
assigned object as a string. I would like to use object names as an
algorithmic input.


To demonstrate... So if i have:

>>foo = {}

what can I do to the object 'foo' so as to return the string 'foo'?

Thanks!
--
http://mail.python.org/mailman/listinfo/python-list

Re: explanation of values, references, assignment, and method calls

[Bruno Desthuilliers]

Joe Strout a écrit :
I've tried to write up this topic in a clear, step-by-step manner, with 
the help of diagrams and short examples from several different OOP 
languages.  I hope it will help clear up the confusion that seems to be 
pervading the Python community


May I suggest http://effbot.org/zone/call-by-object.htm ?


--
http://mail.python.org/mailman/listinfo/python-list

Re: using modules in destructors

[MRAB]

On Oct 27, 5:40 pm, Scott David Daniels <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> > It seems to me that deleting local instances before imported modules
> > would solve the problem. Is it not possible for the interpreter to get
> > this right? Or are there cases where this would break stuff.
>
> > It seems rather unpythonic for the __del__() method to become
> > unpredictable at exit.
>
> What would you have different? How would you define the end of a
> Python program?  Would all modules and globals remain, and then
> darkness?  Would the modules get removed in alphabetical order?
> 
>
I suppose the modules could be removed in the reverse order in which
they were loaded, excepting cross-references (two objects from
different modules referring to their own modules and each other). Even
neglecting modules, if two objects have destructors and refer to each
other, which should be collected first?

> The core of the problem is that there is ((and possibly can be)
> no good definition of what happens after SystemExit leaves the
> __main__ module.
>
> --Scott David Daniels
> [EMAIL PROTECTED]

--
http://mail.python.org/mailman/listinfo/python-list

Re: Database specialized in storing directed graphs?

[George Sakkis]

On Oct 27, 8:32 pm, Carl Banks <[EMAIL PROTECTED]> wrote:

> I was wondering if anyone had any advice on this.
>
> This is not to study graph theory; I'm using the graph to represent a
> problem domain.  The graphs could be arbitrarily large, and could
> easily have millions of nodes, and most nodes have a substantial
> amount of data associated with them.  Obviously I don't want a whole
> such graph in memory at once, so libraries the just deal with in-
> memory graphs are out.
>
> I know I could implement this with a relational DB, and I'd be fine
> with a library that was built on top of one.  But I'm hoping for
> specialzed behavior useful for graphs.

If you're looking for FOSS, the Boost graph library [1] or its
parallel extension [2] is probably your best bet; it also comes with
Python bindings but they are not maintained any more. For commercial
solutions, Star-P [3] seems an interesting platform, with bindings to
Matlab and Python. Freebase [4] is apparently built on a special graph
database but unfortunately only the stored data are available, not the
DB source code.

George

[1] http://www.boost.org/doc/libs/1_36_0/libs/graph/doc/index.html
[2] http://www.osl.iu.edu/research/pbgl/
[3] http://www.interactivesupercomputing.com/success/sparsematrices.php
[4] http://www.freebase.com/help/faq#q7
--
http://mail.python.org/mailman/listinfo/python-list

Re: Database specialized in storing directed graphs?

[bearophileHUGS]

Sorry Carl Banks for the answering delay, there are problems in Google
Groups.

> This is not to study graph theory; I'm using the graph to represent a
> problem domain.  The graphs could be arbitrarily large, and could
> easily have millions of nodes, and most nodes have a substantial
> amount of data associated with them.  Obviously I don't want a whole
> such graph in memory at once, so libraries the just deal with in-
> memory graphs are out.

It doesn't sound a problem for modern PCs.
I think you can keep the whole graph topology in RAM, and the node
data on disk (for example in a file or in DB). The topology is
represented by the arcs (you have said nothing regarding arc data, so
I assume it's absent) and the nodes (in RAM you just need a 32 bit
unsigned integer to represent the index of the node that is stored on
disk. If memory becomes tight you can use just 3 bytes (2 ^ 24 = 16
millions different nodes) for the nodes, but generally the memory
required for the nodes is very little compared to the memory necessary
to store the arcs).

You haven't said how many arcs there are, total or average for node,
and if such arcs are directed or undirected.

Anyway, using my Graph class (that stores each arc twice), this takes
about 1 minute and 1.3 GB of RAM (1 million nodes, 10 arcs for node):

from graph import Graph
from random import randrange
g = Graph()
N = 100
g.addNodes(xrange(N))
for i in xrange(N * 10):
g.addArc(randrange(N), randrange(N))


You have said "could easily have millions of nodes", and every arc may
have tens of arcs or more.
("arbitrarily large" is an unsolvable problem because there are always
limits in your program, there's no program that's able to work on an
"arbitrarily large" dataset), so Python data structures become too
much costly for the RAM. With a lower level language like D/C/C++ you
can manage a bigger graph. You can use Boost Graph, but a homemade
graph structure may suffice too for your purposes.

For the problem you have explained I think a very simple graph
representation may suffice: an array of integer pairs (starting_index,
len) (starting_index can be a pointer too), where len is the number of
outbound arcs of the node n, plus an array of indexes/pointers that
list the outbound arcs. If memory gets tight you can split this second
array in two, and use an array of bytes for the lengths (if you have
more than 256 outbound arcs you may need a short). Note that if you
use indexes then a Python array.array (or Numpy) suffices.

In this situation if nnodes = 10_000_000 and narcs/node = 40 (total
nodes = 40 * 10_000_000):
nnodes * narcs * 4 + nnodes * (4 + 4) = 1_680_000_000 bytes, that is
often available on modern PCs.

On a 64 bit machine the indexes take the same memory, but pointers
twice that.

In "memory saving" mode:
nnodes * narcs * 3 + nnodes * (3 + 1) = 1_240_000_000 bytes.

A more handy compromise is:
nnodes * narcs * 3 + nnodes * (4 + 4) = 1_280_000_000 bytes.

> But then what happens if something changes elsewhere.

Regarding the data structure, if you use arrays like I have explained,
if your updates aren't much frequent then you can allocate small extra
arrays to store more arcs coming out a node (but to do this you
probably may enjoy using pointers instead of indexes). When you have a
lot of updated you can save all to disk, and then regenerate the whole
data structure.

Bye,
bearophile
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python suitable for Midi ?

[Chuckk Hubbard]

I'm writing a sequencer in Python, although it's microtonal and not
MIDI.  I'm using the Python bindings for the Csound API, all the
timing, MIDI, OSC, etc. stuff, essentially all but the GUI
capabilities, having been done by the Csound developers already.
Documentation is here and there, and Csound is another language to
learn, but it's one way to go about it.
I'm about to try to recode my app to calculate tempo and note timing
internally and send real-time notes to Csound, instead of having
Csound do it all.
The problem I've run into is that I can't set the audio to a higher
priority than the GUI (Tkinter).  If I move the mouse over the app, no
matter what, I get audio dropouts.  AFAICT this is the same for all
Python, regardless of what modules one uses: you can't assign system
priorities to different threads.  If you're planning to pipe MIDI to
another app for playback, maybe it won't be an issue for you.
Good luck!

-Chuckk

On Tue, Oct 28, 2008 at 11:26 AM, Protocol <[EMAIL PROTECTED]> wrote:
> Hello all
>
> Is Python suitable for building a multi-track midi sequencer (with a
> gui), that would run on windows / mac ? I fail to find sufficient
> information on this, being a newbie and all. Furthermore, i found
> references on Python not being really able of multi-threading, that
> further adds to the confusion.
>
> Please assist.
>
> Panos
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
http://www.badmuthahubbard.com
--
http://mail.python.org/mailman/listinfo/python-list

Re: Finding the instance reference of an object

[Dale Roberts]

On Oct 28, 2:33 am, "Gabriel Genellina" <[EMAIL PROTECTED]>
wrote:
> En Tue, 28 Oct 2008 01:16:04 -0200, Dale Roberts <[EMAIL PROTECTED]>  
> escribió:
>
>
>
> > So, then, what to tell a C++ programmer about how Python passes  
> > arguments? You say: tell them Python only passes by value. I disagree,  
> > because I think that would confuse them. Rather than try to map C++  
> > conventions onto Python, I think it is more useful to just tell them how  
> > it really works. Maybe a few statements like this:
>
> >    All values in Python are objects, from simple integers up to complex
> >    user-defined classes.
>
> >    An assignment in Python binds a variable name to an object. The
> >    internal "value" of the variable is the memory address of an object,
> >    and can be seen with id(var), but is rarely needed in practice.
>
> >    The "value" that gets passed in a Python function call is the address
> >    of an object (the id()).
>
> >    When making a function call, myfunc(var), the value of id(var) can
> >    never be changed by the function.
>
> > Not sure if these are the best. To get into much more detail, you have  
> > to start explaining mutable and immutable objects and such.
>
> I don't think the above explanation is desirable, nor needed. Objects in  
> Python don't have an "address" - it's just a CPython implementation  
> detail. The fact that id() returns that address is just an implementation  
> detail too. The calling mechanism should be explained without refering to  
> those irrelevant details.
>
> --
> Gabriel Genellina


I agree that it was a feeble and ill-advised attempt, and would like
to strike those lines from the record...

But the rest of the post is strong and accurate, I think, and coming
from a mainly C/C++ background, visualizing these object references
being passed around does help improve my understanding of Python's
*behavior* (and it apparently helps Joe too).

[May I refer to you as "Joe The Programmer" in my rhetoric? Are you a
"licensed" programmer making over $250K/year? ;-)]

If asked by a C++ programmer about Python's argument passing, I will
go with the Pass By Object explanation (which is why I called my
Python routine above ByObject()). Although there will be more
'splaining to do, at least it will give the C++ programmer pause, and
help them realize that there is something a little different that they
need to stop and try to understand.

If I just say "Python is Pass By Value, Period" without further
explanation, they will expect things to work as in my ByValue()
example above (where the caller's object contents cannot be changed),
and that is incorrect. And they may go and tell someone else, without
the detailed explanation, that "Dale says it's Pass By Value", and
I'll get blamed when their function surprisingly changes the contents
of the caller's object.

If I just say "Python is Pass By Reference", that is wrong too. As Joe
The Programmer points out, they will expect that the calling
*variable* itself can be changed, and that is wrong too.

They need to understand that Python does not map neatly, directly, and
completely to their C++ experience of Pass By Value (which involves
copying a variable), or Pass By Reference (which involves taking the
address of a variable).

The Key Concept in for a C++ programmer looking at Python is that,
unlike in C++, **Variables Cannot "Contain" Values**. All values in
Python are objects, and all variables in Python simply point to, or
are bound to, or refer to, these objects. The variables themselves do
not contain the objects - if you must (like Joe the Programmer), you
can say that all Python variables *contain* an object reference, and
it is these references that are passed around. Unlike C++, an object
reference is the ONLY thing that a Python variable can contain. A
function parameter is always passed as a reference to an object, not a
reference to a variable, or a copy of a variable or object.

And that is where the confusion arises. When people say ByRef or
ByVal, they usually mean by Reference to (address) or Value of (copy)
the *contents* of the passed variable. But, PYTHON VARIABLES DO NOT
"CONTAIN" VALUES (sorry for the shouting, but it is the most important
point here), so ByRef and ByVal lose their commonly accepted meanings.

In C++, ByValue requires a copy (and Python does not copy). In C++,
ByReference requires the address of a *variable* (an "lvalue"), and
variables do not have accessible addresses in Python.

ByObject, in contrast, requires neither (unless, like Joe The
Programmer, you consider the "value" of a variable to be the id(),
which is not what most people expect when they say "x=5" - they expect
the "value" to be 5). ByObject simply passes an object reference. Nice
and neat.

I think with that explanation (avoiding any mention of memory, or
object id's) will help them understand things better.

As you point out, this is a very old topic, but I guess each newcomer
to Python has to figure i

Re: Extracting name strings from assigned objects

[Simon Brunning]

2008/10/28 Shannon Mayne <[EMAIL PROTECTED]>:
> I would like to ask how one might obtain the assigned name of an
> assigned object as a string.

That's in the FAQ: .

-- 
Cheers,
Simon B.
[EMAIL PROTECTED]
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python suitable for Midi ?

[Derek Martin]

On Tue, Oct 28, 2008 at 06:54:57PM +0200, Chuckk Hubbard wrote:
> The problem I've run into is that I can't set the audio to a higher
> priority than the GUI (Tkinter).  If I move the mouse over the app, no
> matter what, I get audio dropouts.  AFAICT this is the same for all
> Python, regardless of what modules one uses: you can't assign system
> priorities to different threads.  If you're planning to pipe MIDI to
> another app for playback, maybe it won't be an issue for you.

FWIW...  You could take your own advice, and devide your application
in two: one process manages the GUI, and the second is a back-end
process that plays the MIDI.  Your GUI can even launch the back end,
which will inherit the priority of the GUI, after which the GUI can
reduce its own priority (the priority of the back end will not be
affected by the change)...


-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D



pgpIZwcieNKvY.pgp
Description: PGP signature
--
http://mail.python.org/mailman/listinfo/python-list

Re: Unit Testing: a couple of questions

[Mel]

Emanuele D'Arrigo wrote:
> I'm a bit worried about the time it's taking me to develop the tests
> but after only a day or so I'm already much faster than when I started
> with it and the code is already much improved in terms of robustness.
> A couple of "philosophical" questions have emerged in the process.
[ ... ]
> But would it be reasonable to test also for the assignment operators?
> After all, if, for some strange reason, there isn't enough memory,
> couldn't even __data = data potentially fail?

I would say, don't test for anything you can't fix.  If you have (what the
agile-programming people call) a "user story" describing how your program
will recover from out-of-memory, then test whether that recovery is carried
out.  Otherwise forget it.

Best think of unit tests as your sub-program spec written in a different
form: i.e. as executable programs.  You don't test for "things that can go
wrong", you test for behaviour that your program must exhibit.
> 
> 2) Testing in isolation
> I'm not entirely clear on this point. I can see how I need to test
> each path of the program flow separately. But should a test -only-
> rely on the object being tested and mock ones in supporting roles?
> I.e. would this be wrong if SupportObject is not a mockup?
> 
> def testObjectToBeTested _forReallyBadError(self):
> supportObject = SupportObject()
> objectToBeTested = ObjectToBeTested()
> result =  objectToBeTested.addSupportObject(supportObject)
> self.failIf(result != kSuccess, "Support Object could not be
> added!")
> 
> I can see how if the SupportObject class had a bug introduced in it,
> this test would fail even though it has nothing to do with the
> ObjectToBeTested class. However, creating mock objects can be quite an
> overhead (?). I'm wondering if there is a threshold, even a fuzzy one,
> under which it isn't worth doing and a test like the one above is
> "good enough".

I have heard people talk, matter-of-factly about support objects that got so
complicated that they needed unit tests of their own.  Of course, anything
can be done in a ridiculous way, but good comprehensive unit tests give you
the eternal assurance that your program is behaving properly, through all
maintenance and modification.  That's worth a lot.

--
http://mail.python.org/mailman/listinfo/python-list

Re: dictionary

[D'Arcy J.M. Cain]

On Tue, 28 Oct 2008 10:41:20 -0400
Steve Holden <[EMAIL PROTECTED]> wrote:
> Sorry, you are thinking of bistable multivibrators. I was talking about
> the Wiliams tube store:
> 
>   http://ed-thelen.org/comp-hist/SEAC-Williams-tube-desc.html

Very cool.  Yes, I was thinking of something else.  However, Williams'
first attempt at storing memory on a CRT (1946) stored only one bit.

> I don't ever remember programming to cope with equipment failure,
> however. Did you make that bit up?

No, I read it many years ago.  However, I cannot find the reference
(try searching for anything computer related and "shopping cart.") so I
cannot check the story's veracity.

-- 
D'Arcy J.M. Cain <[EMAIL PROTECTED]> |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP)   |  what's for dinner.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Web crawler on python

[Alex]

On Oct 26, 9:54 pm, sonich <[EMAIL PROTECTED]> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?

You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
 or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito/jazz-crawler/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex
--
http://mail.python.org/mailman/listinfo/python-list

Contracts for Python

[Paulo J. Matos]

Hi all,

I am wondering if there is any work on contracts for Python. I could
only find PEP316, however, I am wondering if there is any official
support for it already (tools I mean), and if it is or if it will be
officially supported in any of the next releases of Python.

Cheers,
-- 
Paulo Jorge Matos - pocmatos at gmail.com
Webpage: http://www.personal.soton.ac.uk/pocm
--
http://mail.python.org/mailman/listinfo/python-list

404 not found on for Python 2.6 Itanium

[csgrimes1]

Anyone know where else I can download 2.6 for x64 windows?

Thanks!
--
http://mail.python.org/mailman/listinfo/python-list

Specifying an IP address for outbound connections?

[erikcw]

Python seems to default to the main system IP for outbound connections
(such as urllib), but I want to bind to one of my other IPs for
outbound connections.

Any ideas?

Thanks!
--
http://mail.python.org/mailman/listinfo/python-list

zipping a directory

[Callie Bertsche]

Hey Python-ers,
I apologize if this is covered in the archives; I think I saw one
instance of it but I couldn't get the solution to work out.
I'm working on zipping an entire directory to one zip file. I can zip a
flat group of files, but when my code encounters a hierarchy of folders,
for some reason it starts to zip everything on my desktop (instead of
everything inside one folder directory on desktop).  Then it goes into
an infinite loop and barfs.
 
Here's what I have so far:
 
import zipfile
import glob, os
 
def main():
 
directory = "test\*"
(success,filename)=createZipFile(directory);
if success == 1:
print "Operation completed. All files written to zip."
else:
print "Operation failed."
 
def createZipFile(directory):
zippedHelp = zipfile.ZipFile("help.zip", "w" ) 

for entity in directory:
if os.path.isfile(entity):
 
zippedHelp.write(entity,os.path.basename(entity),zipfile.ZIP_DEFLATED)
else:
addFolderToZip(zippedHelp,entity)
 
zippedHelp.close()
return (1,zippedHelp)
 

def addFolderToZip(zippedHelp,folder):
 
for file in folder:
if os.path.isfile(file):
zippedHelp.write(file, os.path.basename(file),
zipfile.ZIP_DEFLATED)
elif os.path.isdir(file):
addFolderToZip(zippedHelp,file)
 

main()
 
Thanks for any help!! :)
Callie
--
http://mail.python.org/mailman/listinfo/python-list

Return lines in file that match string

[Travis Kirstine]

I am new to python and could use some help with a fairly easy task.  I
would like to return all lines in a file that have the string
'' to a list.

Regards,

-- 
Travis K.

Toronto, Canada

"She knows there's no success like failure
And that failure's no success at all."
-Bob Dylan-

--
http://mail.python.org/mailman/listinfo/python-list

Re: Unit Testing: a couple of questions

[Marco Bizzarri]

On Tue, Oct 28, 2008 at 3:56 PM, Emanuele D'Arrigo <[EMAIL PROTECTED]> wrote:
> Hi everybody,
>
> I'm just having a go with Unit Testing for the first time and my
> feeling about it in short is: Neat!
>
> I'm a bit worried about the time it's taking me to develop the tests
> but after only a day or so I'm already much faster than when I started
> with it and the code is already much improved in terms of robustness.
> A couple of "philosophical" questions have emerged in the process.
>
> 1) Granularity
> Given a simple class
>
> class myClass():
>def __init__(self, data):
>__data = data;
>
>def getData(self):
>return __data
>
>def setData(self, data):
> __data = data
>
> I've been wondering: where do I stop in terms of testing for things
> that could go wrong? In this case for example, it might be reasonable
> to expand the class to make sure it only receives integers and test
> accordingly, i.e.:
>
>def setData(self, data):
>try:
> data = int(data)
>except ValueError:
>raise ValueError("Argument received cannot be converted to
> integer: " + data)
>
> But would it be reasonable to test also for the assignment operators?
> After all, if, for some strange reason, there isn't enough memory,
> couldn't even __data = data potentially fail?
>
> 2) Testing in isolation
> I'm not entirely clear on this point. I can see how I need to test
> each path of the program flow separately. But should a test -only-
> rely on the object being tested and mock ones in supporting roles?


IMHO, don't do that. Mocking everything and anything can become
incredibly complex. Wait to do it. As usual, experience plays a big
role on this; start mocking on what will make your test *simpler* and
easier to write and to understand, for you today and for you tomorrow
(and remember that you tomorrow is a different one from you today).

Before writing a mock, ask yourself: can I use a real object? The both
of you know the behaviour of the real object; you hadn't to think
about what some you yesterday wrote inside it.


> I.e. would this be wrong if SupportObject is not a mockup?
>
> def testObjectToBeTested _forReallyBadError(self):
>supportObject = SupportObject()
>objectToBeTested = ObjectToBeTested()
>result =  objectToBeTested.addSupportObject(supportObject)
>self.failIf(result != kSuccess, "Support Object could not be
> added!")

> I can see how if the SupportObject class had a bug introduced in it,
> this test would fail even though it has nothing to do with the
> ObjectToBeTested class.

> However, creating mock objects can be quite an
> overhead (?). I'm wondering if there is a threshold, even a fuzzy one,
> under which it isn't worth doing and a test like the one above is
> "good enough".



> What do you guys think?
>
> Manu
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Creating mock objects in python can be quite easy; even more if you're
on a greenfield project, and not with some thousands or ten of
thousands of legacy code around. So, the burden is not that much. If
you end using more than a couple of mocks in one test, most probably
there is something wrong, and it would be better if you reconsider
your design.

Also, think if you really need mock object in your tests. Can't it be
easier to use a "real" object? Unless creating it is really complex,
and making it show the behaviour you want it to show is hard or
unclear, you can use it directly.

You wont' use a mock for a datetime object, am I right?

Moreover, don't fight your problem (integrating your objects) at the
wrong level (unit testing); you'll need integration tests, where you
use your *REAL* object end-to-end, maybe even talking with other
subsystems, in order to check that you are working for real. And
there,maybe, you could discover that your discipline failed and you
didn't changed the signature of the method of a mock object after you
changed it in the real one. It happens, so, put your tests in place to
catch this problem.

Regards
Marco

-- 
Marco Bizzarri
http://notenotturne.blogspot.com/
http://iliveinpisa.blogspot.com/
--
http://mail.python.org/mailman/listinfo/python-list

Re: 404 not found on for Python 2.6 Itanium

[Christian Heimes]

[EMAIL PROTECTED] wrote:

Anyone know where else I can download 2.6 for x64 windows?


x64 is AMD64 aka X64_86 and not the Itanium version. Itanium is IA64. We 
don't build Python for IA64 anymore.


Christian

--
http://mail.python.org/mailman/listinfo/python-list

Re: Contracts for Python

[Chris Rebert]

On Tue, Oct 28, 2008 at 10:47 AM, Paulo J. Matos <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I am wondering if there is any work on contracts for Python. I could
> only find PEP316, however, I am wondering if there is any official
> support for it already (tools I mean), and if it is or if it will be
> officially supported in any of the next releases of Python.

No, it's not officially supported and there are currently no plans to
support it, though as PEP 316 is Deferred and not Rejected, it's
theoretically possible that it could be Accepted and added to the
language some day.
At any rate, there are several third-party libraries available that
allow you to use Design by Contract in Python, albeit without
language-level support.
See http://pypi.python.org/pypi?%3Aaction=search&term=contract&submit=search
for a list

Cheers,
Chris

>
> Cheers,
> --
> Paulo Jorge Matos - pocmatos at gmail.com
> Webpage: http://www.personal.soton.ac.uk/pocm
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list

Re: Return lines in file that match string

[Chris Rebert]

On Tue, Oct 28, 2008 at 11:37 AM, Travis Kirstine
<[EMAIL PROTECTED]> wrote:
> I am new to python and could use some help with a fairly easy task.  I
> would like to return all lines in a file that have the string
> '' to a list.
>

from __future__ import with_statement

with open('path/to/file') as f:
desired_lines = [line.strip() for line in f if "" in line]

For your reference, that uses the "with statement" and "list comprehensions".
It would also be advisable for you to read through the fine tutorial
at: http://docs.python.org/tutorial/index.html

Cheers,
Chris

> Regards,
>
> --
> Travis K.
>
> Toronto, Canada
> 
> "She knows there's no success like failure
> And that failure's no success at all."
> -Bob Dylan-
> 
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
Follow the path of the Iguana...
http://rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list

Re: Database specialized in storing directed graphs?

[flyingfrog]

Don't really know if this will be useful but i'd try pytables:
http://www.pytables.org/moin
it deals very well with every kind of hierarchical data sets, doesn't
matter the size.
It will let you load only significant data, and you'll be able to
query your data.
It's built on top of HDF5 libraries but exposes a very friendly
pythonic interface.
Surely you'll still have to implement all of the graph logic yourself
but this could be a good starting point.
Hope it helps

Marco
--
http://mail.python.org/mailman/listinfo/python-list

import foo vs. python -m foo

[Simon Bierbaum]

Hi all,

what is the difference between saying "import foo" in an interactive  
prompt and starting one using "python -m foo"? The -m switch is not  
covered in the man page, is it even officially supported? I'm asking  
because one of my modules fails on import in the second version but  
succeeds in the first.


Thanks, Simon

--
http://mail.python.org/mailman/listinfo/python-list

Re: big objects and avoiding deepcopy?

[Aaron Brady]

On Oct 27, 1:10 pm, Terry Reedy <[EMAIL PROTECTED]> wrote:
> [EMAIL PROTECTED] wrote:
> > Robert Kern:
> >> This is similar to implementing "Undo" functionality in applications.<
>
> > In a quite-high-level language (like Python, but not necessarily in
> > Python itself) it may become eventually advantageous to add some (even
> > limited) built-in form of undo.
>
> Right now, I believe, one could develop an undo module with undolist and
> undodict classes that wrap instance mutations with an append to an
> undoinfo list.

The trick would be forking instances, so that you could operate on
separate histories/branches independently.  Not necessary for most
undo purposes, of course.
--
http://mail.python.org/mailman/listinfo/python-list

Re: ignoring NAs in a dataset with numpy

[Robert Kern]

mrafi wrote:

Hey All,
I am working with numpy 
I have a data set with a lot of nan values, and i want to calculate standard

deviation
is there a direct function to do it for example nansum(or something like
this),
to calculate standard deviation ignoring nan values??


You will want to ask numpy questions on the numpy-discussion mailing list.

  http://www.scipy.org/Mailing_Lists

scipy.stats.nanstd() implements this. Additionally, data[~isnan(data)].std() 
works unless if you need to apply the std() along just one axis.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list

Re: question about the textwrap module

[TP]

Gabriel Genellina wrote:

> You may pre-process your text (stripping redundant whitespace) before
> using textwrap:

Thanks Gabriel for your answers!
I finally have subclassed textwrap.TextWrapper.

Julien


-- 
python -c "print ''.join([chr(154 - ord(c)) for c in '*9(9&(18%.9&1+,\'Z
(55l4('])"

"When a distinguished but elderly scientist states that something is
possible, he is almost certainly right. When he states that something is
impossible, he is very probably wrong." (first law of AC Clarke)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Finding the instance reference of an object

[Dale Roberts]

On Oct 28, 11:59 am, Joe Strout <[EMAIL PROTECTED]> wrote:
> ...
>
> There are only the two cases, which Greg quite succinctly and  
> accurately described above.  One is by value, the other is by  
> reference.  Python quite clearly uses by value.  Parameters are  
> expressions that are evaluated, and the resulting value copied into  
> the formal parameter, pure and simple.  The continued attempts to  
> obfuscate this is pointless and wrong.
>
> Best,
> - Joe

5 + 3

What is the "value" of that expression in Python? Can you tell me?

99.99% of programmers (who do not have this thread as context) will
say that the value is 8. But you say the value is the memory address
of the resulting object created when the + operator is applied to the
5 object and the 3 object. That is the "value" that is copied.

Okay, you can have it that way, but every time you explain to someone
that Python passes "By Value", you will have to add the additional
baggage that, oh, by the way, there is a completely different meaning
for "value" in Python than what you are used to.

Then the questions and puzzled looks will start...

And when they tell their friend that Joe The Programmer said it's Pass
By Value, your additional context may not be present any longer, and
the friend will be very confused.

In my opinion, best just to head it off and call it something
different so as not to confuse.

dale
--
http://mail.python.org/mailman/listinfo/python-list

HTML File Parsing

[Felipe De Bene]

I'm having problems parsing an HTML file with the following syntax :


User ID
NameDate
and so on

whenever I feed the parser with such file I get the error :

Traceback (most recent call last):
  File "C:\Documents and Settings\Administrator\My Documents\workspace
\thread\src\parser.py", line 91, in 
p.parse(thechange)
  File "C:\Documents and Settings\Administrator\My Documents\workspace
\thread\src\parser.py", line 16, in parse
self.feed(s)
  File "C:\Python25\lib\HTMLParser.py", line 110, in feed
self.goahead(0)
  File "C:\Python25\lib\HTMLParser.py", line 152, in goahead
k = self.parse_endtag(i)
  File "C:\Python25\lib\HTMLParser.py", line 316, in parse_endtag
self.error("bad end tag: %r" % (rawdata[i:j],))
  File "C:\Python25\lib\HTMLParser.py", line 117, in error
raise HTMLParseError(message, self.getpos())
HTMLParser.HTMLParseError: bad end tag: "", at
line 515, column 45

Googling around I've found a solution to a similar situation, over and
over again :
http://64.233.169.104/search?q=cache:zOmjwM_sGBcJ:coding.derkeiler.com/pdf/Archive/Python/comp.lang.python/2006-02/msg00026.pdf+CDATA_CONTENT_ELEMENTS&hl=pt-BR&ct=clnk&cd=5&gl=br&client=firefox-a

but coding :

you can disable proper parsing by setting the CDATA_CONTENT_ELEMENTS
attribute on the parser instance, before you start parsing. by
default, it is
set to
CDATA_CONTENT_ELEMENTS = ("script", "style")
setting it to an empty tuple disables HTML-compliant handling for
these
elements:
p = HTMLParser()
p.CDATA_CONTENT_ELEMENTS = ()
p.feed(f.read())

didn't solve my problem. I've made a little modification then to
HTMLParser.py instead that solved the problem, as follows:
original: endtagfind = re.compile('')
my version : endtagfind = re.compile('')

it worked ok for all the files I needed and also for a different file
I also parse using the same library. I know it might sound stupid but
I was just wondering if there's a better way of solving that problem
than just modifying the standard library. Any clue ?

thx in advance,
Felipe.
--
http://mail.python.org/mailman/listinfo/python-list

Re: big objects and avoiding deepcopy?

[Daniel da Silva]

http://groups.google.com/group/perl.perl6.language/msg/b0cfa757f0ce1cfd?pli=1

: )


On Mon, Oct 27, 2008 at 1:12 PM, <[EMAIL PROTECTED]> wrote:

> Robert Kern:
> > This is similar to implementing "Undo" functionality in applications.<
>
> In a quite-high-level language (like Python, but not necessarily in
> Python itself) it may become eventually advantageous to add some (even
> limited) built-in form of undo. Both to give a simpler way to
> implement a undo functionality into user-level programs (that is to
> implement the Undo command in a program with GUI), but more
> importantly to help the programmer too in some more general
> programming tasks.
>
> So it's probably a feature we'll see in the next generation of high-
> level languages, among few other features that today are missing in
> Python (only sometimes missing for performance reasons).
>
> Bye,
> bearophile
> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
Daniel da Silva
(240) 678 - 4686
GSFC, GES-DISC 610.2
University of Maryland
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Rhamphoryncus]

On Oct 28, 9:30 am, "Andy O'Meara" <[EMAIL PROTECTED]> wrote:
> On Oct 25, 9:46 am, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote:
>
> > These discussion pop up every year or so and I think that most of them
> > are not really all that necessary, since the GIL isn't all that bad.
>
> Thing is, if the topic keeps coming up, then that may be an indicator
> that change is truly needed.  Someone much wiser than me once shared
> that a measure of the usefulness and quality of a package (or API) is
> how easily it can be added to an application--of any flavors--without
> the application needing to change.
>
> So in the rising world of idle cores and worker threads, I do see an
> increasing concern over the GIL.  Although I recognize that the debate
> is lengthy, heated, and has strong arguments on both sides, my reading
> on the issue makes me feel like there's a bias for the pro-GIL side
> because of the volume of design and coding work associated with
> considering various alternatives (such as Glenn's "Py*" concepts).
> And I DO respect and appreciate where the pro-GIL people come from:
> who the heck wants to do all that work and recoding so that a tiny
> percent of developers can benefit?  And my best response is that as
> unfortunate as it is, python needs to be more multi-threaded app-
> friendly if we hope to attract the next generation of app developers
> that want to just drop python into their app (and not have to change
> their app around python).  For example, Lua has that property, as
> evidenced by its rapidly growing presence in commercial software
> (Blizzard uses it heavily, for example).
>
>
>
> > Furthermore, there are lots of ways to tune the CPython VM to make
> > it more or less responsive to thread switches via the various sys.set*()
> > functions in the sys module.
>
> > Most computing or I/O intense C extensions, built-in modules and object
> > implementations already release the GIL for you, so it usually doesn't
> > get in the way all that often.
>
> The main issue I take there is that it's often highly useful for C
> modules to make subsequent calls back into the interpreter. I suppose
> the response to that is to call the GIL before reentry, but it just
> seems to be more code and responsibility in scenarios where it's no
> necessary.  Although that code and protocol may come easy to veteran
> CPython developers, let's not forget that an important goal is to
> attract new developers and companies to the scene, where they get
> their thread-independent code up and running using python without any
> unexpected reengineering.  Again, why are companies choosing Lua over
> Python when it comes to an easy and flexible drop-in interpreter?  And
> please take my points here to be exploratory, and not hostile or
> accusatory, in nature.
>
> Andy

Okay, here's the bottom line:
* This is not about the GIL.  This is about *completely* isolated
interpreters; most of the time when we want to remove the GIL we want
a single interpreter with lots of shared data.
* Your use case, although not common, is not extraordinarily rare
either.  It'd be nice to support.
* If CPython had supported it all along we would continue to maintain
it.
* However, since it's not supported today, it's not worth the time
invested, API incompatibility, and general breakage it would imply.
* Although it's far more work than just solving your problem, if I
were to remove the GIL I'd go all the way and allow shared objects.

So there's really only two options here:
* get a short-term bodge that works, like hacking the 3rd party
library to use your shared-memory allocator.  Should be far less work
than hacking all of CPython.
* invest yourself in solving the *entire* problem (GIL removal with
shared python objects).
--
http://mail.python.org/mailman/listinfo/python-list

Re: parse a table in HTML page.

[Stefan Behnel]

antonio_wn8 wrote:
> I have a need to read and parse a table in HTML page.
> 
> I’m using the following script:
> http://trac.davidgrant.ca/browser/src/python/misc/siteuptime/TableParser.py
> 
> It works fine  aside from  link in href.
> 
> Example:
> 
> String to parse:
> elognormal text
> 
> Output:
> [[['elog', 'normal text']]]

You should try lxml.html. It gives you various tools like XPath to look for
specific elements and helper functions to find the links in an HTML document.

http://codespeak.net/lxml/

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: HTML File Parsing

[Stefan Behnel]

Felipe De Bene wrote:
> I'm having problems parsing an HTML file with the following syntax :
> 
> 
> User ID
> Name BGCOLOR='#c0c0c0'>Date
> and so on
> 
> whenever I feed the parser with such file I get the error :
> 
> HTMLParser.HTMLParseError: bad end tag: "", at
> line 515, column 45

Your HTML page is not HTML, i.e. it is broken. Python's HTMLParser is not made
for parsing broken HTML. However, you can use the parse of lxml.html to fix up
your HTML for you.

http://codespeak.net/lxml/

Stefan
--
http://mail.python.org/mailman/listinfo/python-list

Re: Do I need a lock here?

[jasiu85]

On Oct 27, 10:12 am, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
> jasiu85 schrieb:
>
>
>
> > Hey,
>
> > Please take a look at the code of the two threads below:
>
> > COMMON_DICT = {}
>
> > def thread_1():
> >     global COMMON_DICT
> >     local_dict = prepare_dict()
> >     COMMON_DICT = local_dict
>
> > def thread_2():
> >     global COMMON_DICT
> >     local_dict = COMMON_DICT
> >     use_dict(local_dict)
>
> > Do I need a lock to protect the COMMON_DICT dictionary? AFAIK bytecode
> > operations are atomic and in each thread there's only one crucial
> > bytecode op: STORE_NAME in the first thread and LOAD_NAME in the
> > second one. So I suspect that everything will work just fine. Am I
> > right?
>
> Depending on what you mean by "right".
>
> The above is not enough to judge what is really happening. But depending
> on the execution order, thread_1 overwrites the reference in COMMON_DICT
> *after* thread_2 has altered it.
>
> But it won't crash or anything. If that's right for you.
>
> Diez

The second thread doesn't alter the dictionary, it only reads it. Even
if the first thread puts a reference to a new dictionary into the
COMMON_DICT variable while the second thread is reading the old
dictionary, such sematics is fine for me. I'm seeking for a kind of
Producer/Consumer pattern. So the first thread produces some data in a
form of a dictionary. The second thread reads it from time to time.
It's sufficient for me if a dictionary is "lost" because the first
thread overwrites the COMMON_DICT variable before the second thread
tries to read it. It's also sufficient for me if the first thread
updates COMMON_DICT variable while the second thread is reading the
previous copy of the dictionary. So from my point of view the most
critical parts are these two lines:

COMMON_DICT = local_dict # in the first thread
local_dict = COMMON_DICT # in the second thread

Can these two instructions by any chance interfere with each other in
a way that will crash either of the threads?

I hope I made myself a little bit more clear :).

Thanks!!

Mike
--
http://mail.python.org/mailman/listinfo/python-list

Re: import foo vs. python -m foo

[Hrvoje Niksic]

Simon Bierbaum <[EMAIL PROTECTED]> writes:

> Hi all,
>
> what is the difference between saying "import foo" in an interactive
> prompt and starting one using "python -m foo"? The -m switch is not
> covered in the man page, is it even officially supported?

My copy of the man page states:

-m module-name
Searches sys.path for the named module and runs the corresponding
.py file as a script.

> I'm asking because one of my modules fails on import in the second
> version but succeeds in the first.

It probably contains buggy code inside "if __name__ == '__main__': ...".
--
http://mail.python.org/mailman/listinfo/python-list

Tkinter: How to get Label wraplength functionality in Text Box

[Mudcat]

I've tried quite a few things to get this correct but have hit a
couple of sticking points that I can't figure out. I need to ge the
Text box to function like the 'wraplength' option in a Label.

I've been able to adjust the height of the text by calculating the
number of lines needed to display the text. That's fairly simple. I
know the number of characters in the text and width of the box (which
is static). From that point I can calculate how many lines of display
is needed and resize it.

The problem arises when I use the 'wrap' option of the Text Box so the
words aren't chopped off. Once the wrapping is done there are dead
spaces left at the end of the lines which are ignored when the char
count is done. As a result sometimes the last line is not shown. I can
always just add +1 to the number, but then sometimes I get an empty
line. Space is at a premium in this app, so I have to cut everything
down to use only what's necessary.

So does anyone know how add in those extra spaces to get this to
adjust correctly? (Or if there is another way to get the Text Box to
automatically adjust it's size that I'm not aware of?)

Thanks
--
http://mail.python.org/mailman/listinfo/python-list

Re: Database specialized in storing directed graphs?

[Aaron Brady]

On Oct 27, 7:32 pm, Carl Banks <[EMAIL PROTECTED]> wrote:
> I was wondering if anyone had any advice on this.
>
> This is not to study graph theory; I'm using the graph to represent a
> problem domain.  The graphs could be arbitrarily large, and could
> easily have millions of nodes, and most nodes have a substantial
> amount of data associated with them.  Obviously I don't want a whole
> such graph in memory at once, so libraries the just deal with in-
> memory graphs are out.
>
> I know I could implement this with a relational DB, and I'd be fine
> with a library that was built on top of one.  But I'm hoping for
> specialzed behavior useful for graphs.
>
> For example, suppose I have a set of nodes called A.  It would be
> useful to know if any of these nodes are connected by paths that
> include no nodes in A.  I could, of course, do that by reading from
> the database and following the paths, but I clearly don't want to do
> that.  I would want instead to somehow cache the outside connectedness
> relationship between different nodes of A.  But then what happens if
> something changes elsewhere.  How is the cache for A notified, and can
> it be updated efficiently using graph theorems, rather than
> regenerated?
>
> It's very tricky; that's why I hope someone else has done it.
>
> I'm guessing no.
>
> Carl Banks

Are you just looking for a persistent graph?  The usual options are
'shelve', 'sqllite', 'mmap', and 'ctypes.Structure'.  Or did you need
a special structure for the 'outside connectedness' properties?

Storing that is the usual time-space tradeoff.  (Actually, I think
that term refers to something slightly distinct.  This would be more
properly termed read-time/write-time trade off, roughly.)

Assuming you pick an RDB, you'd have a table of vertices and a table
of edges.  Did you want 'set A' to be a table and persist between runs
of the program?  If not, just keep a set of 2-tuples of nodes that
satisfy that property.  I think the set S you're after is: all x,y
such that x is a vertex in A, y is a vertex in A, and there exists a P
such that P is a path, P starts at x, P ends at y, and vertex v in P
implies that v is x, v is y, or v is not in A.  I don't know if you
can cache any information about A that gives you any shortcuts in
calculating S, or what the running time or running space of the
fastest/smallest algorithm is.  At worst, it's O( |V| * |E| ) on each
change to V or E.  Unverified.

You might be able to store the shortest path in a mapping from 2-
tuples to paths.  Or better yet, map each node in A to the set of all
nodes reachable from it only entering A once.  Then, you might save
time on either adding a node/vertex to A or G, or removing one from A
or G, or both.  Maybe mapping each node in G to that relative set.
You didn't say whether the graph was directed and/or cyclic.

A good place to start would be, if you add a node to G, can S
decrease?  No, because the original path P still exists.  If you
remove one from A, still in G, S can stay the same size, might
decrease.  If you remove a node from G, not in A, same, etc.
--
http://mail.python.org/mailman/listinfo/python-list