Question about 'iterable cursors'

[Frank Millman]

Hi all

I am using a few DB_API adaptors - ceODBC for Sql Server, psycopg2 for 
PostgreSQL, and sqlite3 for sqlite3.


They all offer the feature that if a cursor executes a SELECT, the cursor 
returns an iterator which can be used to fetch one row at a time. I have 
been using this feature for a while and it seems like a 'good thing'.


Now I am not so sure. I am using a connection pool to maintain connections 
to the database. A principle I am following is that a connection must be 
returned quickly, so that it is available for reuse.


I have been happily returning the connection, but keeping the cursor open 
while processing the rows selected. I now realise that this is dangerous. 
Therefore I have changed my system to execute fetchall() on the cursor 
before returning the connection. This obviously loses the benefit of the 
iterator.


I would appreciate confirmation that my thinking is correct on this issue. 
Or is there any way that I can have my cake and eat it?


Thanks

Frank Millman


--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about 'iterable cursors'

[Alain Ketterlin]

"Frank Millman"  writes:

> I am using a few DB_API adaptors - ceODBC for Sql Server, psycopg2 for
> PostgreSQL, and sqlite3 for sqlite3.
>
> They all offer the feature that if a cursor executes a SELECT, the
> cursor returns an iterator which can be used to fetch one row at a
> time. I have been using this feature for a while and it seems like a
> good thing'.
>
> Now I am not so sure. I am using a connection pool to maintain
> connections to the database. A principle I am following is that a
> connection must be returned quickly, so that it is available for
> reuse.
>
> I have been happily returning the connection, but keeping the cursor
> open while processing the rows selected. I now realise that this is
> dangerous. Therefore I have changed my system to execute fetchall() on
> the cursor before returning the connection. This obviously loses the
> benefit of the iterator.
>
> I would appreciate confirmation that my thinking is correct on this
> issue. Or is there any way that I can have my cake and eat it?

Your thinking is correct: you need to keep the connection while
processing the cursor. Databases are made to scale, you may well be
processing the first lines of the result before the DBMS has even
finished scanning tables. View this as a pipe, the cursor being one end
of the pipe. The usual setting, fetching one line at a time, lets you
overlap your processing with the network transfers.

Fetching all data, returning the connection, and then start processing
only makes sense if the processing take a lot of time (I mean: a lot
more than fetching results), which is a rare case. Unless you are in
such an extreme situation, I would suggest leaving the optimization to
the connection pool, which is here to solve what you are trying to
solve.

-- Alain.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about 'iterable cursors'

[Frank Millman]

"Alain Ketterlin"  wrote

"Frank Millman"  writes:


I am using a few DB_API adaptors - ceODBC for Sql Server, psycopg2 for
PostgreSQL, and sqlite3 for sqlite3.

They all offer the feature that if a cursor executes a SELECT, the
cursor returns an iterator which can be used to fetch one row at a
time. I have been using this feature for a while and it seems like a
good thing'.

Now I am not so sure. I am using a connection pool to maintain
connections to the database. A principle I am following is that a
connection must be returned quickly, so that it is available for
reuse.

I have been happily returning the connection, but keeping the cursor
open while processing the rows selected. I now realise that this is
dangerous. Therefore I have changed my system to execute fetchall() on
the cursor before returning the connection. This obviously loses the
benefit of the iterator.

I would appreciate confirmation that my thinking is correct on this
issue. Or is there any way that I can have my cake and eat it?


Your thinking is correct: you need to keep the connection while
processing the cursor. Databases are made to scale, you may well be
processing the first lines of the result before the DBMS has even
finished scanning tables. View this as a pipe, the cursor being one end
of the pipe. The usual setting, fetching one line at a time, lets you
overlap your processing with the network transfers.

Fetching all data, returning the connection, and then start processing
only makes sense if the processing take a lot of time (I mean: a lot
more than fetching results), which is a rare case. Unless you are in
such an extreme situation, I would suggest leaving the optimization to
the connection pool, which is here to solve what you are trying to
solve.



Thank you, Alain. That is very clear.

So my analysis of the problem is correct, but my solution is wrong.

Instead of executing fetchall() and returning the connection, I should 
retain the connection until I have exhausted the cursor.


That makes a lot of sense.

Frank


--
http://mail.python.org/mailman/listinfo/python-list

SQLAlchemy DB Migration

[Merwin]

Hi everybody,

I'm looking for a way to migrate SQLAlchemy based database easily. I 
found Migrate, but finally it's almost the same as doing the migration 
by hand.


If this doesn't exists, do you think that it would be hard to do a 
simple migration tool which would :


- Add/Delete columns from DB if they are not anymore in the models 
(With a confirmation for deletion)
- Add/Delete constraints/sequences on table if they are not on the 
model too


We could also imagine a way to :

- Handle column rename, with data copy from one to another
- Handle column type change, with proper data conversion

Of course, such a system can't be 100% automated, that's not the goal. 
Human interventation will be needed, but it might be more simpler for 
simple cases than Migrate.


--
Thibaut
--
http://mail.python.org/mailman/listinfo/python-list

How to undo a Python setuptools --prefix path blunder

[Kev]

When I installed Python's setuptools I absent mindedly tacked on a --
prefix path I had been using on another machine:

sh setuptools-0.6c11-py2.7.egg --prefix=/opt/python2.7.2

Now after this blunder when I try to install pip I get the following
error:

[root@kkdev src]# easy_install pip
Searching for pip
Best match: pip 1.0.2
Processing pip-1.0.2-py2.7.egg
pip 1.0.2 is already the active version in easy-install.pth
Installing pip script to /usr/bin
error: /usr/bin/pip: No such file or directory

What's happening is that a symbolic link is being created that points
to the folder I specified in the --prefix path which is obviously
wrong:

[root@kkdev src]# ls -al /usr/bin/pip
lrwxrwxrwx 1 root root 24 Nov  5 17:01 /usr/bin/pip -> /opt/
python2.7.2/bin/pip

I deleted this link and then re-ran the setuptools installer and
specified the correct prefix (my Python install lives in /usr/lib/
python2.7):

sh setuptools-0.6c11-py2.7.egg --prefix=/usr

I then re-ran easy_install pip and it looked like I'd fixed my finger
trouble. However when I went to install virtualenv I encountered the
same problem:

[root@kkdev src]# pip install virtualenv

[uninteresting installer output snipped]

Installing virtualenv script to /usr/bin

error: /usr/bin/virtualenv: No such file or directory

Again the wrong path is being used to create the symbolic link to
where virtualenv is installed:

[root@kkdev src]# ls -al /usr/bin/virtualenv
lrwxrwxrwx 1 root root 31 Nov  5 17:01 /usr/bin/virtualenv -> /opt/
python2.7.2/bin/virtualenv

(I'm running Fedora 15 32bit which has Python 2.7.1 installed out of
the box)

How do I fix/repair this permanently?

Thanks
Kevin



-- 
http://mail.python.org/mailman/listinfo/python-list

A Python script to put CTAN into git (from DVDs)

[Jonathan Fine]

Hi

This it to let you know that I'm writing (in Python) a script that 
places the content of CTAN into a git repository.

https://bitbucket.org/jfine/python-ctantools

I'm working from the TeX Collection DVDs that are published each year by 
the TeX user groups, which contain a snapshot of CTAN (about 100,000 
files occupying 4Gb), which means I have to unzip folders and do a few 
other things.


CTAN is the Comprehensive TeX Archive Network.  CTAN keeps only the 
latest version of each file, but old CTAN snapshots will provide many 
earlier versions.


I'm working on putting old CTAN files into modern version control. 
Martin Scharrer is working in the other direction.  He's putting new 
files added to CTAN into Mercurial.

http://ctanhg.scharrer-online.de/

My script works already as a proof of concept, but needs more work (and 
documentation) before it becomes useful.  I've requested that follow up 
goes to comp.text.tex.


Longer terms goals are git as
* http://en.wikipedia.org/wiki/Content-addressable_storage
* a resource editing and linking system

If you didn't know, a git tree is much like an immutable JSON object, 
except that it does not have arrays or numbers.


If my project interests you, reply to this message or contact me 
directly (or both).


--
Jonathan
--
http://mail.python.org/mailman/listinfo/python-list

Re: A Python script to put CTAN into git (from DVDs)

[Jakub Narebski]

Jonathan Fine  writes:

> Hi
> 
> This it to let you know that I'm writing (in Python) a script that
> places the content of CTAN into a git repository.
>  https://bitbucket.org/jfine/python-ctantools

I hope that you meant "repositories" (plural) here, one per tool,
rather than putting all of CTAN into single Git repository.
 
> I'm working from the TeX Collection DVDs that are published each year
> by the TeX user groups, which contain a snapshot of CTAN (about
> 100,000 files occupying 4Gb), which means I have to unzip folders and
> do a few other things.

There is 'contrib/fast-import/import-zips.py' in git.git repository.
If you are not using it, or its equivalent, it might be worth checking
out.
 
> CTAN is the Comprehensive TeX Archive Network.  CTAN keeps only the
> latest version of each file, but old CTAN snapshots will provide many
> earlier versions.

There was similar effort done in putting CPAN (Comprehensive _Perl_
Archive Network) in Git, hosting repositories on GitHub[1], by the name
of gitPAN, see e.g.:

  "The gitPAN Import is Complete"
  http://perlisalive.com/articles/36
 
[1]: https://github.com/gitpan

> I'm working on putting old CTAN files into modern version
> control. Martin Scharrer is working in the other direction.  He's
> putting new files added to CTAN into Mercurial.
>  http://ctanhg.scharrer-online.de/

Nb. thanks to tools such as git-hg and fast-import / fast-export
we have quite good interoperability and convertability between
Git and Mercurial.

P.S. I'd point to reposurgeon tool, which can be used to do fixups
after import, but it would probably won't work on such large (set of)
repositories.

P.P.S. Can you forward it to comp.text.tex?
-- 
Jakub Narębski
-- 
http://mail.python.org/mailman/listinfo/python-list

Python lesson please

[gene heskett]

Greetings experts:

I just dl'd the duqu driver finder script from a link to NSS on /., and 
fixed enough of the tabs in it to make it run error-free.  At least python 
isn't having a litter of cows over the indentation now.

But it also runs instantly on linux.

This line looks suspect to me:
 rootdir = sys.argv[1]

And I have a suspicion it is null on a linux box.

How can I fix that best?

Thanks.

Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: 
Most of the fear that spoils our life comes from attacking difficulties
before we get to them.
-- Dr. Frank Crane
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A Python script to put CTAN into git (from DVDs)

[Jonathan Fine]

On 06/11/11 16:42, Jakub Narebski wrote:

Jonathan Fine  writes:


Hi

This it to let you know that I'm writing (in Python) a script that
places the content of CTAN into a git repository.
  https://bitbucket.org/jfine/python-ctantools


I hope that you meant "repositories" (plural) here, one per tool,
rather than putting all of CTAN into single Git repository.


There are complex dependencies among LaTeX macro packages, and TeX is 
often distributed and installed from a DVD.  So it makes sense here to 
put *all* the content of a DVD into a repository.


Once you've done that, it is then possible and sensible to select 
suitable interesting subsets, such as releases of a particular package. 
Users could even define their own subsets, such as "all resources needed 
to process this file, exactly as it processes on my machine".


In addition, many TeX users have a TeX DVD.  If they import it into a 
git repository (using for example my script) then the update from 2011 
to 2012 would require much less bandwidth.


Finally, I'd rather be working within git that modified copy of the ISO 
when doing the subsetting.  I'm pretty sure that I can manage to pull 
the small repositories from the big git-CTAN repository.


But as I proceed, perhaps I'll change my mind (smile).


I'm working from the TeX Collection DVDs that are published each year
by the TeX user groups, which contain a snapshot of CTAN (about
100,000 files occupying 4Gb), which means I have to unzip folders and
do a few other things.


There is 'contrib/fast-import/import-zips.py' in git.git repository.
If you are not using it, or its equivalent, it might be worth checking
out.


Well, I didn't know about that.  I took a look, and it doesn't do what I 
want.  I need to walk the tree (on a mounted ISO) and unpack some (but 
not all) zip files as I come across them.  For details see:


https://bitbucket.org/jfine/python-ctantools/src/tip/ctantools/filetools.py

In addition, I don't want to make a commit.  I just want to make a ref 
at the end of building the tree.  This is because I want the import of a 
TeX DVD to give effectively identical results for all users, and so any 
commit information would be effectively constant.



CTAN is the Comprehensive TeX Archive Network.  CTAN keeps only the
latest version of each file, but old CTAN snapshots will provide many
earlier versions.


There was similar effort done in putting CPAN (Comprehensive _Perl_
Archive Network) in Git, hosting repositories on GitHub[1], by the name
of gitPAN, see e.g.:

   "The gitPAN Import is Complete"
   http://perlisalive.com/articles/36

[1]: https://github.com/gitpan


This is really good to know!!!  Not only has this been done already, for 
similar reasons, but github is hosting it.  Life is easier when there is 
a good example to follow.



I'm working on putting old CTAN files into modern version
control. Martin Scharrer is working in the other direction.  He's
putting new files added to CTAN into Mercurial.
  http://ctanhg.scharrer-online.de/


Nb. thanks to tools such as git-hg and fast-import / fast-export
we have quite good interoperability and convertability between
Git and Mercurial.

P.S. I'd point to reposurgeon tool, which can be used to do fixups
after import, but it would probably won't work on such large (set of)
repositories.


Thank you for the pointer to reposurgeon.  My approach is a bit 
different.  First, get all the files into git, and then 'edit the tree' 
to create new trees.  And then commit worthwhile new trees.


As I recall the first 'commit' to the git repository for the Linux 
kernel was just a tree, with a reference to that tree as a tag.  But no 
commit.



P.P.S. Can you forward it to comp.text.tex?


Done.

--
Jonathan

--
http://mail.python.org/mailman/listinfo/python-list

RSS feed creation?

[Dan Stromberg]

Is there an opensource Python tool for creating RSS feeds, that doesn't
require large dependencies?

I found feedformatter.py on pypi, but it seems a little old, and its sole
automated test gives a traceback.

Is there a better starting point?

(I'd of course prefer something that'll run on 3.x and 2.x, but will settle
for either)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[Dave Angel]

On 11/06/2011 01:14 PM, gene heskett wrote:

Greetings experts:

I just dl'd the duqu driver finder script from a link to NSS on /., and
fixed enough of the tabs in it to make it run error-free.  At least python
isn't having a litter of cows over the indentation now.

But it also runs instantly on linux.

This line looks suspect to me:
  rootdir = sys.argv[1]

And I have a suspicion it is null on a linux box.

How can I fix that best?

Thanks.

Cheers, Gene


Nothing wrong with that line, assuming the user of the script happened 
to provide an argument for rootpath.  Probably it should be preceded by 
a a conditional:


if len(argv) <2:
   tell.user.that.she's.missing.the.rootdir.parameter
 else:
   rootdir = sys.argv[1]
   etc.

What does the help text show when you run the script with the --help 
argument?


--

DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: A Python script to put CTAN into git (from DVDs)

[Jakub Narebski]

The following message is a courtesy copy of an article
that has been posted to comp.lang.python,comp.text.tex as well.

Jonathan Fine  writes:
> On 06/11/11 16:42, Jakub Narebski wrote:
>> Jonathan Fine  writes:
>>
>>> This it to let you know that I'm writing (in Python) a script that
>>> places the content of CTAN into a git repository.
>>>   https://bitbucket.org/jfine/python-ctantools
>>
>> I hope that you meant "repositories" (plural) here, one per tool,
>> rather than putting all of CTAN into single Git repository.

[moved]
>> There was similar effort done in putting CPAN (Comprehensive _Perl_
>> Archive Network) in Git, hosting repositories on GitHub[1], by the name
>> of gitPAN, see e.g.:
>>
>>"The gitPAN Import is Complete"
>>http://perlisalive.com/articles/36
>>
>> [1]: https://github.com/gitpan
[/moved]
 
> There are complex dependencies among LaTeX macro packages, and TeX is
> often distributed and installed from a DVD.  So it makes sense here to
> put *all* the content of a DVD into a repository.

Note that for gitPAN each "distribution" (usually but not always
corresponding to single Perl module) is in separate repository.
The dependencies are handled by CPAN / CPANPLUS / cpanm client
(i.e. during install).
 
Putting all DVD (is it "TeX Live" DVD by the way?) into single
repository would put quite a bit of stress to git; it was created for
software development (although admittedly of large project like Linux
kernel), not 4GB+ trees.

> Once you've done that, it is then possible and sensible to select
> suitable interesting subsets, such as releases of a particular
> package. Users could even define their own subsets, such as "all
> resources needed to process this file, exactly as it processes on my
> machine".

This could be handled using submodules, by having superrepository that
consist solely of references to other repositories by the way of
submodules... plus perhaps some administrativa files (like README for
whole CTAN, or search tool, or DVD install, etc.)

This could be the used to get for example contents of DVD from 2010.


But even though submodules (c.f. Subversion svn:external, Mecurial
forest extension, etc.) are in Git for quite a bit of time, it doesn't
have best user interface.
 
> In addition, many TeX users have a TeX DVD.  If they import it into a
> git repository (using for example my script) then the update from 2011
> to 2012 would require much less bandwidth.

???

> Finally, I'd rather be working within git that modified copy of the
> ISO when doing the subsetting.  I'm pretty sure that I can manage to
> pull the small repositories from the big git-CTAN repository.

No you cannot.  It is all or nothing; there is no support for partial
_clone_ (yet), and it looks like it is a hard problem.

Nb. there is support for partial _checkout_, but this is something
different.

> But as I proceed, perhaps I'll change my mind (smile).
> 
>>> I'm working from the TeX Collection DVDs that are published each year
>>> by the TeX user groups, which contain a snapshot of CTAN (about
>>> 100,000 files occupying 4Gb), which means I have to unzip folders and
>>> do a few other things.
>>
>> There is 'contrib/fast-import/import-zips.py' in git.git repository.
>> If you are not using it, or its equivalent, it might be worth checking
>> out.
> 
> Well, I didn't know about that.  I took a look, and it doesn't do what
> I want.  I need to walk the tree (on a mounted ISO) and unpack some
> (but not all) zip files as I come across them.  For details see:
>  https://bitbucket.org/jfine/python-ctantools/src/tip/ctantools/filetools.py
> 
> In addition, I don't want to make a commit.  I just want to make a ref
> at the end of building the tree.  This is because I want the import of
> a TeX DVD to give effectively identical results for all users, and so
> any commit information would be effectively constant.

Commit = tree + parent + metadata.

I think you would very much want to have linear sequence of trees,
ordered via DAG of commits.  "Naked" trees are rather bad idea, I think.
 
> As I recall the first 'commit' to the git repository for the Linux
> kernel was just a tree, with a reference to that tree as a tag.  But
> no commit.

That was a bad accident that there is a tag that points directly to a
tree of _initial import_, not something to copy.

-- 
Jakub Narębski
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[Cameron Simpson]

On 06Nov2011 13:14, gene heskett  wrote:
| Greetings experts:
| 
| I just dl'd the duqu driver finder script from a link to NSS on /., and 
| fixed enough of the tabs in it to make it run error-free.  At least python 
| isn't having a litter of cows over the indentation now.
| 
| But it also runs instantly on linux.
| 
| This line looks suspect to me:
|  rootdir = sys.argv[1]
| 
| And I have a suspicion it is null on a linux box.

That line collects the first command line argument. sys.argv is the
command line; argv[0] is the command name, argv[1] the first argument
and so forth.

Also, if there is no first argument then trying to access argv[1] will
raise an IndexError exception, not return a null value.

If you're this new to python, note that easy debugging statements can be
written:

  print >>sys.stderr, "the value of foo is", foo

and so forth.

Perhaps more relevantly:

If you have unmangled a lot of tabs, remember that control flow is
indentation based in python, and you may have broken some logic.
(For this reason a number of us set our editors to work only in spaces).

Anyway, while changing:

  statement1
  statement2

into:

  statement1
statement2

will usually make python complain, changing:

  if some test here:
statement1
statement2

into:

  if some test here:
statement1
  statement2

just means that statement2 is no longer inside the if-statement, and
elicit no compaints. But will elicit bugs!

Cheers,
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

The ZZR-1100 is not the bike for me, but the day they invent "nerf" roads
and ban radars I'll be the first in line..AMCN
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[gene heskett]

On Sunday, November 06, 2011 06:07:36 PM Dave Angel did opine:

> On 11/06/2011 01:14 PM, gene heskett wrote:
> > Greetings experts:
> > 
> > I just dl'd the duqu driver finder script from a link to NSS on /.,
> > and fixed enough of the tabs in it to make it run error-free.  At
> > least python isn't having a litter of cows over the indentation now.
> > 
> > But it also runs instantly on linux.
> > 
> > This line looks suspect to me:
> >   rootdir = sys.argv[1]
> > 
> > And I have a suspicion it is null on a linux box.
> > 
> > How can I fix that best?
> > 
> > Thanks.
> > 
> > Cheers, Gene
> 
> Nothing wrong with that line, assuming the user of the script happened
> to provide an argument for rootpath.  Probably it should be preceded by
> a a conditional:
> 
>  if len(argv) <2:
> tell.user.that.she's.missing.the.rootdir.parameter
>   else:
> rootdir = sys.argv[1]
> etc.
> 
> What does the help text show when you run the script with the --help
> argument?

It has no --help:
[root@coyote gene]# ./duqu-drivers-detect.py --help
[root@coyote gene]#

Sorry.

Thanks & Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
My web page: 
Expedience is the best teacher.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Usefulness of the "not in" operator

[DevPlayer]

On Oct 16, 12:05 am, Steven D'Aprano  wrote:
> On Sat, 15 Oct 2011 15:04:24 -0700, DevPlayer wrote:
> > I thought "x not in y" was later added as syntax sugar for "not x in y"
> > meaning they used the same set of tokens. (Too lazy to check the actual
> > tokens)
Stated in response to OP wanting a seperate token for "not in" verse
"is not".

> Whether the compiler has a special token for "not in" is irrelevant.
I don't know.

> Perhaps it uses one token, or two, or none at all because a
> pre-processor changes "x not in y" to "not x in y". That's
> an implementation detail.
I agree.

> What's important is whether it is valid syntax or not, and how it is
> implemented.
I agree.

> As it turns out, the Python compiler does not distinguish the two forms:
>
> >>> from dis import dis
> >>> dis(compile('x not in y', '', 'single'))
>
>   1           0 LOAD_NAME                0 (x)
>               3 LOAD_NAME                1 (y)
>               6 COMPARE_OP               7 (not in)
>               9 PRINT_EXPR
>              10 LOAD_CONST               0 (None)
>              13 RETURN_VALUE        >>> dis(compile('not x in y', '', 
> 'single'))
>
>   1           0 LOAD_NAME                0 (x)
>               3 LOAD_NAME                1 (y)
>               6 COMPARE_OP               7 (not in)
>               9 PRINT_EXPR
>              10 LOAD_CONST               0 (None)
>              13 RETURN_VALUE

So cool! Thanks for showing how to do that.

I tried to say implementing a seperate method was not efficient.

> Also for what it is worth, "x not in y" goes back to at least Python 1.5,
> and possibly even older. (I don't have any older versions available to
> test.)
So "not in" was added as an alternative (just a long time ago).
I too am glad they added it.

> (2) Instead of writing "True if blah else False", write "bool(blah)".
Good tip! I like.

>
> > class Y(object):
> >     def __contains__(self, x):
> >         for item in y:
> >         if x == y:
> >             return True
> >         return False
>
> You don't have to define a __contains__ method if you just want to test
> each item sequentially. All you need is to obey the sequence protocol and
> define a __getitem__ that works in the conventional way:

Didn't intend to show how to implement __contains__ using "==" and
__not_contains__ "<>" in python but to show that python didn't benefit
from the not_in loop as much as for example assembly language does
it's loop (x86 LOOPE/LOOPZ vs LOOPNZ/LOOPNE).

>
> >>> class Test:
>
> ...     def __init__(self, args):
> ...             self._data = list(args)
> ...     def __getitem__(self, i):
> ...             return self._data[i]
> ...>>> t = Test("abcde")
> >>> "c" in t
> True
> >>> "x" in t
> False
Another new thing for me.

>
> Defining a specialist __contains__ method is only necessary for non-
> sequences, or if you have some fast method for testing whether an item is
> in the object quickly. If all you do is test each element one at a time,
> in numeric order, don't bother writing __contains__.
>
> > And if you wanted "x not in y" to be a different token you'd have to ADD
>
> Tokens are irrelevant. "x not in y" is defined to be the same as "not x
> in y" no matter what.
> You can't define "not in" to do something completely different.
I agree they are not implemented differently.
I agree that they shouldn't be implemented differently.
I disagree they can not be implemented differently. I think they can.
But I see no reason too.

> > class Y(object):
> >     def __not_contained__(self, x):
> >         for item in self:
> >             if x == y:
> >                 return False
> >         return True
>
> > AND with __not_contained__() you'd always have to iterate the entire
> > sequence to make sure even the last item doesn't match.
> > SO with one token "x not in y" you DON'T have to itterate through the
> > entire sequence thus it is more effiecient.
> That's not correct.
> Steven
I tried to prove my point and failded and instead proved (to myself)
you are correct. It is not more efficient. Also I should have used if
<> y: continue to have better tried to make the point but it wouldn't
have mattered. I still would have been wrong.

But I did walk away from this topic with some goodie tips. Thanks
Steven.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[Dave Angel]

On 11/06/2011 06:10 PM, gene heskett wrote:

On Sunday, November 06, 2011 06:07:36 PM Dave Angel did opine:


On 11/06/2011 01:14 PM, gene heskett wrote:

Greetings experts:

I just dl'd the duqu driver finder script from a link to NSS on /.,
and fixed enough of the tabs in it to make it run error-free.  At
least python isn't having a litter of cows over the indentation now.

But it also runs instantly on linux.

This line looks suspect to me:
   rootdir = sys.argv[1]

And I have a suspicion it is null on a linux box.

How can I fix that best?

Thanks.

Cheers, Gene


Nothing wrong with that line, assuming the user of the script happened
to provide an argument for rootpath.  Probably it should be preceded by
a a conditional:

  if len(argv)<2:
 tell.user.that.she's.missing.the.rootdir.parameter
   else:
 rootdir = sys.argv[1]
 etc.

What does the help text show when you run the script with the --help
argument?


It has no --help:
[root@coyote gene]# ./duqu-drivers-detect.py --help
[root@coyote gene]#

Sorry.

Thanks&  Cheers, Gene



So is there any documentation?  If not, your first job is to guess what 
it's supposed to do, figure out if you care, and then document it.  In 
the process you'll probably find a few bugs.


First bug is it silently quits when you run it with no arguments. Is 
that true?  If so, figure out where it quits, since it clearly never 
reaches the line you quote.


Actually I'd probably start with the assumption that you might have 
messed up the tab expansion somewhere.  How did you expand the tabs? 
Perhaps by using the Linux expands utility?  Or did you do it manually?


--

DaveA
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[David Riley]

On Nov 6, 2011, at 3:34 PM, Cameron Simpson wrote:

> Perhaps more relevantly:
> 
> If you have unmangled a lot of tabs, remember that control flow is
> indentation based in python, and you may have broken some logic.
> (For this reason a number of us set our editors to work only in spaces).

I would absolutely check this first.  I can't count the number of programs I've 
screwed up in difficult-to-detect ways because I've converted from tabs to 
spaces incorrectly.  It's how I've finally learned to let a program/script do 
it for me, because Python is the first language I've used with significant 
leading whitespace.

- Dave

-- 
http://mail.python.org/mailman/listinfo/python-list

question about Tkinter delete

[Kristen Aw]

Hi all

I don't understand why I get this error. I'm trying to delete the existing 
points, then redraw them after this bit of code to 'animate' my simulation.

def update(self, point1, point2):
        # Deletes existing points
        if self.point1:
            self.w.delete(point1)
            self.master.update_idletasks()
        if self.point2:
            self.w.delete(point2)
            self.master.update_idletasks()
#draw new point
# . . .

The error message that I get is:
. . . in update
    self.w.delete(point1)
  File "C:\PYTHON26\LIB\LIB-TK\Tkinter.py", line 2181, in delete
    self.tk.call((self._w, 'delete') + args)
TclError: invalid command name ".44593760"-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python lesson please

[Steven D'Aprano]

On Sun, 06 Nov 2011 13:14:43 -0500, gene heskett wrote:

> I just dl'd the duqu driver finder script from a link to NSS on /., and 
> fixed enough of the tabs in it to make it run error-free.  At least
> python isn't having a litter of cows over the indentation now.



Presumably the script should be working. If it is known to be not 
working, then you probably need to fix a lot more than just indentation.

If this is "pre-alpha, doesn't even compile" software, then good luck, 
you'll need to read the source and understand it to make it work. It's 
not sufficient to just get it to a point where you can say "Cool, it 
compiles!" by fixing some indentation.

If this is supposed to be working, there's a reason why it's not working 
for you. Perhaps you are trying to use it with the wrong version of 
Python. Perhaps the download is corrupted. Fix those before mangling 
indentation in random ways: start with a fresh, uncorrupted download. 
Check that the md5 sum matches that provided by the download site 
(assuming they provide one). Find out what version of Python is 
supported, and use that.


> This line looks suspect to me:
>  rootdir = sys.argv[1]

What makes you think that line is suspect? It looks fine to me.


> And I have a suspicion it is null on a linux box.

This is Python, not bash. Sys arguments are not filled in with "null", 
whatever you think "null" is. (The None built-in? The empty string?)

Have you tried giving the script an argument when you call it? If in 
doubt, call it with "." (the current directory) as argument.

My wild guess is that you actually need to supply a meaningful directory 
path before the script will do anything, and until then, it will silently 
fail. Which is terrible design.


> How can I fix that best?

If it's not broken, it doesn't need to be fixed. Don't make assumptions 
about what code needs to be fixed based on nothing more than gut feeling.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

[ANN] Oktest.py 0.10.0 released - a new-style testing library

[Makoto Kuwata]

Hi,

I released Oktest.py 0.10.0.
http://packages.python.org/Oktest/
http://www.kuwata-lab.com/oktest/

Oktest.py is a new-style testing library for Python.
::

from oktest import ok, NG
ok (x) > 0 # same as assertTrue(x > 0)
ok (s) == 'foo'# same as assertEqual(s, 'foo')
ok (s) != 'foo'# same as assertNotEqual(s, 'foo')
ok (f).raises(ValueError)  # same as assertRaises(ValueError, f)
ok (u'foo').is_a(unicode)  # same as assertTrue(isinstance(u'foo', unicode))
NG (u'foo').is_a(int)  # same as assertTrue(not isinstance(u'foo', int))
ok ('A.txt').is_file() # same as assertTrue(os.path.isfile('A.txt'))
NG ('A.txt').is_dir()  # same as assertTrue(not os.path.isdir('A.txt'))

See
  http://www.kuwata-lab.com/oktest/oktest-py_users-guide.html
for details.


Changes and Enhancements


* [change] 'oktest.spec()' is obsoleted completely.
  It will print warning message if you use it.

* [change] 'oktest.helper' module is renamed to 'oktest.util'.
  ('oktest.helper' is still available for backward compabibility.)

* [enhance] Add 'oktest.main()' which is a replacement of 'oktest.run()'.
  Using 'oktest.main()' instead of 'oktest.run()', command options are
available.
  ex::

## for example:
$ python test/foobar_test.py -sp -f test='*keyword*'
## is almost same as:
$ python -m oktest test/foobar_test.py -sp -f test='*keyword*'

* [enhance] Add 'oktest.fail(message)' which is same as
'unittest.fail(message)'.
  ex::

from oktest import fail
fail("not impelmented yet")# will raise AssertionError

* [enhance] (Experimental) Add '@todo' decorator which is equivarent to
  '@unittest.expectedFailure'.
  ex::

from oktest import ok, test, todo

def add(x, y):
return 0  # not implemented yet!

class AddTest(unittest.TestCase):
@test("returns sum of arguments.")
@todo # equivarent to @unittest.expectedFailure
def _(self):
ok (10, 20) == 30   ## will be failed expectedly
## (because not implemented yet)

  Expected failure of assertion is reported as '[TODO]', not '[Failed]'.

* [enhance] (Experimental) Test context supported.
  It helps you to describe specification in structured style.
  ex::

from oktest import ok, test
from oktest.context import subject, situation

class SampleTestCase(unittest.TestCase):
  SUBJECT = "class 'Sample'"

  with subject("method1()"):

with situation("when condition:"):

  @test("spec1")
def _(self):
...

  @test("spec2")
  def _(self):
...

with situation("else:"):

  @test("spec3")
  def _(self):
...

  Output exmple::

$ python test/example_test.py
* class 'Sample'
  + method1()
+ when condition:
  - [ok] spec1
  - [ok] spec2
+ else:
  - [ok] spec3
## total:3, passed:3, failed:0, error:0, skipped:0   (elapsed 0.000)


--
regards,
makoto kuwata
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about 'iterable cursors'

[John Nagle]

On 11/6/2011 12:04 PM, Dennis Lee Bieber wrote:

On Sun, 6 Nov 2011 11:39:56 +0200, "Frank Millman"
declaimed the following in gmane.comp.python.general:



So my analysis of the problem is correct, but my solution is wrong.

Instead of executing fetchall() and returning the connection, I should
retain the connection until I have exhausted the cursor.

That makes a lot of sense.


Especially if all you are processing are read-only activities.

If you have a connection/cursor doing write operations, you may not
be able to commit those writes until all reading cursors have closed.
(Read the documentation on the SQLite3 locking system -- though the
newest version has added a second type of locking which may complicate
the matter. The original/normal scheme has potential readers "outside"
SQLite3, active readers "inside" SQLite3 -- when an active reader cursor
advances to a pending write, it blocks all the potential readers from
entering, but is itself blocked until all other active readers have
exited)


   Right.  The scarce resource is database locks, not connections.
Especially with SQLite, which has, by necessity, a rather brutal
locking strategy.

   Realize that SQLite is not a high-performance multi-user database.
You use SQLite to store your browser preferences, not your customer
database.

   If you're doing enough transactions from multiple processes that
performance is an issue, you need to move up to MySQL or Postgres.
If almost all transactions are SELECTs, performance may not be
too bad, but if there are INSERT and UPDATE transactions on the
same table, performance will be awful.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

Re: How to undo a Python setuptools --prefix path blunder

[Matej Cepl]

Dne 6.11.2011 14:18, Kev napsal(a):

Again the wrong path is being used to create the symbolic link to
where virtualenv is installed:


http://packages.python.org/distribute/easy_install.html#administrator-installation 
for list of additional configuration files which might go wrong (namely 
*.pth and distutils.cfg).


Is that it?

Matěj
--
http://mail.python.org/mailman/listinfo/python-list

Re: RSS feed creation?

[Stefan Behnel]

Dan Stromberg, 06.11.2011 21:00:

Is there an opensource Python tool for creating RSS feeds, that doesn't
require large dependencies?

I found feedformatter.py on pypi, but it seems a little old, and its sole
automated test gives a traceback.

Is there a better starting point?

(I'd of course prefer something that'll run on 3.x and 2.x, but will settle
for either)


I'd just go with ElementTree and builder.py.

http://effbot.org/zone/element-builder.htm

http://effbot.python-hosting.com/file/stuff/sandbox/elementlib/builder.py

Building an RSS-API on top of that is so trivial that it's not worth any 
further dependencies anyway. Not sure if this version of builder.py runs in 
Py3, but it certainly won't be hard to fix even if it doesn't currently.


Stefan

--
http://mail.python.org/mailman/listinfo/python-list