Most direct way to strip unoprintable characters out of a string?

[Steve Bergman]

When sanitizing data coming in from HTML forms, I'm doing this (lifted 
from the Python Cookbook):

from string import maketrans, translate, printable
allchars = maketrans('','')
delchars = translate(allchars, allchars, printable)
input_string = translate(input_string, allchars, delchars)

Which is OK.  But it seems like there should be more straightforward way 
that I just haven't figured out.  Is there?

Thanks,
Steve Bergman
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Most direct way to strip unoprintable characters out of a string?

[Steve Bergman]

George Sakkis wrote:

>
>
>If by straightforward you mean one-liner, there is:
>''.join(c for c in input_string if c not in string.printable)
>
>If you care about performance though, string.translate is faster; as always, 
>the best way to decide
>on a performance issue is to profile the alternatives on your data and see if 
>it's worth going for
>the fastest one at the expense of readability.
>
>  
>
Thank you for the reply.  I was really thinking of some function in the 
standard library like:

s = stripUnprintable(s)

When I learned php, I more or less took the route of using whatever I 
found that 'worked'.  In learning Python, I'm trying to take my time and 
learn the 'right' (that's pronounced 'beautiful') way of doing things.

As it stands, I've stashed the string.translate code in a short function 
with a comment explaining what it does and how.  I mainly didn't want to 
use that if there was some trivial built-in that everyone else uses.

Thanks Again,
Steve


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Most direct way to strip unoprintable characters out of a string?

[Steve Bergman]

Fredrik Lundh wrote:

>("sanitizing" HTML data by running filters over encoded 8-bit data is hardly
>ever the right thing to do...)
>
>
>  
>
I'm very much open to suggestions as to the right way to do this.  I'm 
working on this primarily as a learning project and security is my 
motivation for wanting to strip  the unprintables.

Is there a better way? (This is a mod_python app , just for reference.)

Thanks,
Steve
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3! Finally!

[Steve Bergman]

Duncan Booth wrote:

>Did you download the file using Firefox? It seems to have a 'feature' that 
>when you download a file and there is already a file of the same name it 
>finds the first number in the filename and increments it.
>
>  
>
Yes, isn't that fun?  This has been a known problem with a bug filed for 
well over a year.  Perhaps closer to a year and a half.  Not that the 
bug has been simply ignored.  It has been hashed and rehashed over and 
over.  Someone proposes a solution.  Firefox devs come back and say that 
the solution will not work in some corner case on Windows.  Someone 
proposes a solution.  Firefox devs come back and say that it will 
adversely affect some Linux users.  Someone proposes a solution.  
Firefox devs come back and say that certain MacOS users could experience 
a problem with that.  Someone proposes that in the case that there is a 
filename collision, the user might simply be prompted to decide on a 
filename. Firefox devs totally reject that idea (and rather 
vehemently).  You see, that would totally freak out the end user, and 
that is *certainly* not a good solution.

So the interim solution, for over a year, has been to leave it broken 
for everyone...


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Distributing programs

[Steve Bergman]

Leif K-Brooks wrote:

>But remember that Python bytecode can be easily decompiled with a
>publicly-available program.
>  
>
I hope it is not considered too antisocial to bring it up here, but 
there is always PyObfuscate:

http://www.lysator.liu.se/~astrand/projects/pyobfuscate/

-Steve Bergman
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Distributing programs

[Steve Bergman]

Wouter van Ooijen (www.voti.nl) wrote:

>Yes, and you must also include a blank sheet, signed by you in blood.
>  
>
I thought you only had to do that if you were submitting a patch to 
MySQL, Qt, OpenOffice, or OpenSolaris.  ;-)

-Steve Bergman
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Which SQL module to use?

[Steve Bergman]

mrstephengross wrote:

>I'd like to do some basic SQL stuff in Python. It seems like there are
>a heck of a lot of SQL modules for Python. What's the simplest and
>easiest one to use?
>  
>
The suggestion to use sqllite is probably a good one to get started.

I'm a PostgreSQL fan and use pygresql.  It has 2 modes:  "Classic" and 
"DBAPI 2.0". 

The DPAPI 2.0 interface makes your SQL portable between RDBMs.  But you 
can't beat the classic interface for simplicity, elegance, and convenience.

With classic, just pass a dictionary and say insert this or update this 
and its done.  With selects, its easy to get your results as a dictionary.

-Steve Bergman
-- 
http://mail.python.org/mailman/listinfo/python-list

Pygresql classic vs DB-API interface

[Steve Bergman]

I am relatively new to python and am developing an application using 
mod_python/pygresql/postgresql.

Being attracted to the idea of database portability, I started out using 
the DB-API 2.0 compliant pgdb module.  However, I am finding it to be 
pretty clunky compared to the classic pg interface.
It just doesn't seem "pythonic" to me.  Chief among my concerns is the 
way it won't return results as a dictionary, so I have to address the 
data using cryptic numeric indeces.  To be fair, the 3rd party dtuple 
module lets me use dictionaries, but it still seems a bit clunky.  Also, 
insertion into a table seems unnecessarily ugly.  With the classic 
interface, I just pass a tablename and a dictionary and let the 
interface deal with it.  With pgdb, if I want to do something like that 
I've got to write my own code to do it.  Or does DB-API and pyformat 
allow me to do this in a more elegant way of which I am unaware?

I still like the portability of pgdb though.  So, does anyone have any 
recommendations?  Perhaps a module that does for inserts and updates 
kind of what dtuple does for selects?

Thank you for any guidance.

Steve Bergman
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ide RAD for linux?

[Steve Bergman]

LaGuna wrote:

>Suse 9.3
>
>Ide RAD for linux?
>
>  
>
'Idle' is shipped with current versions of python.  There is also 
PyDev,which is a plugin for Eclipse. I found Eclipse to be rather 
top-heavy and went back to Idle.

-Steve
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: functional or object-oriented?

[Steve Bergman]

beza1e1 wrote:

>I see myself shifting more and more over to the functional kind of
>coding. Could be related to the Haskell, we had to learn in CS. Now i
>was wondering, how other people use Python?
>
>With functional i mean my files mostly consist of functions and only
>rarely i use "class". The library modules seem to be mostly written the
>object-way on the other hand.
>
>If you use both paradigms. What are your criterias to choose the right
>method for a project?
>
>  
>
Here is a quote by Alex Martelli from the "Python Cookbook":

"If the packaging is in terms of objects that typically comprise state 
and behavior, you're using OOP.  Some object-oriented languages force 
you to use OOP for everything, so you end up with many object which lack 
either state or behavior.  Python, however, supports multiple 
paradigms.  While everything in Python is an object, you package things 
as OOP objects only when you want to.  Other languages try to force your 
programming style into a predefined mold for your own good, while Python 
empowers you to make and express your own design choices.

With OOP, once you have specified how an object is composed, you can 
instantiate as many objects of that kind as you need. When you don't 
want to create multiple objects, consider using other Python constructs 
such as modules."


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Intermediate to expert book

[Steve Bergman]

I'm waiting for the release of the next edition of "Programming Python" 
from O'Reilly.  It's due out in December.  The current edition is rather 
oldish.  From 2001, I believe.

Steve
-- 
http://mail.python.org/mailman/listinfo/python-list

Remarkable results with psyco and sieve of Eratosthenes

[Steve Bergman]

Just wanted to report a delightful little surprise while experimenting
with psyco.
The program below performs astonoshingly well with psyco.

It finds all the prime numbers < 10,000,000

Processor is AMD64 4000+ running 32 bit.

Non psyco'd python version takes 94 seconds.

psyco'd version takes 9.6 seconds.

But here is the kicker.  The very same algorithm written up in C and
compiled with gcc -O3, takes 4.5 seconds.  Python is runng half as fast
as optimized C in this test!

Made my day, and I wanted to share my discovery.

BTW, can this code be made any more efficient?




#!/usr/bin/python -OO
import math
import sys
import psyco

psyco.full()

def primes():
primes=[3]
for x in xrange(5,1000,2):
maxfact = int(math.sqrt(x))
flag=True
for y in primes:
if y > maxfact:
break
if x%y == 0:
flag=False
break
if flag == True:
primes.append(x)
primes()

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Remarkable results with psyco and sieve of Eratosthenes

[Steve Bergman]

Will McGugan wrote:

> Some trivial optimizations. Give this a whirl.


I retimed and got 9.7 average for 3 runs on my version.

Yours got it down to 9.2.

5% improvement. Not bad.

(Inserting '2' at the beginning doesn't seem to impact performance
much.;-) )

BTW, strictly speaking, shouldn't I be adding something to the floating
point sqrt result, before converting to int, to allow for rounding
error?  If it is supposed to be 367 but comes in at 366., don't
I potentially classify a composite as a prime?

How much needs to be added?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Remarkable results with psyco and sieve of Eratosthenes

[Steve Bergman]

[EMAIL PROTECTED] wrote:
> > BTW, can this code be made any more efficient?
>
> I'm not sure, but the following code takes around 6 seconds on my
> 1.2Ghz iBook. How does it run on your machine?
>
>

Hmm.  Come to think of it, my algorithm isn't the sieve.

Anyway, this is indeed fast as long as you have enough memory to handle
it for the range supplied.

It comes in at 1.185 seconds average on this box.

Come to think of it, there is a supposedly highly optimized version of
the sieve in The Python Cookbook that I've never bothered to actually
try out.  Hmmm...

-- 
http://mail.python.org/mailman/listinfo/python-list

Why are slice indices the way they are in python?

[Steve Bergman]

A couple of off the wall questions.

It seems to me that there is usually a solid *reason* for things in
Python and I'm wondering about the rationale for the way slicing works:

my_string[2:5]

gets you the 3rd through the 3rd through the 5th character of the
string because indexing starts at 0 and you get everything up to, but
not including the second index.

Why?  It doesn't seem intuitive to me.  To me, it makes it harder, not
easier, to work with slices than if indexing started at 1 and  the
above expression got you the 2nd throught the 5th character.

Another thing that I've been puzzling over is the pow() function.

pow(x,y) gives x**y.  Fine.

But pow(x,y,z) gives (x**y) % c

I'm curious to know what the pressing reason for such a feature was.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: why would anyone use python when java is there?

[Steve Bergman]

Bruce Eckel states the case pretty well in this interview:

http://www.artima.com/intv/aboutme.html

Bruce is the author of "Thinking In Java" and other excellent books,
but has migrated from the Java camp. (I'm excited to see him getting at
least a bit involved in TurboGears.  He has a lot to offer any project
in which he takes an interest.)

I find his opinions particularly relevant.

I've tried doing the Java thing, mainly because of the hype surrounding
it... and the marketability.  But I never got very far because I just
disliked the language. (Personal opinion.  But hey, this *is*
comp.lang.python!)

-- 
http://mail.python.org/mailman/listinfo/python-list

About the 79 character line recommendation

[Steve Bergman]

As I study Python, I am trying to develop good, Pythonic, habits.  For
one thing, I am trying to keep Guido's the style guide in mind.

And I know that it starts out saying that it should not be applied in
an absolute fashion.

However, I am finding that the 79 character line prescription is not
optimal for readability.

Certainly, cutting back from the length of lines that I used to use has
*helped* readability.  But if I triy very hard to apply 79, I think
readability suffers.  If this were just something that was an issue
occasionally, I would just put it off to "know when to break the
rules".  However, find myself going to 90 to 100 characters very
frequently.  Now, if it were just me, I'd shoot for < 100.  However,
the Python philosophy includes making code easier for others to read,
as well.

So, I was wondering what more accomplished Python programmers thought
about this.

While I'm on this general topic, the guide mentions a pet peeve about
inserting more than one space to line up the "=" in assignment
statements.  To me, lining them up, even if it requires quite a few
extra spaces, helps readability quite a bit.  Comments?

Thanks,
Steve Bergman

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: About the 79 character line recommendation

[Steve Bergman]

Thanks for the responses.

The point about 132 columns is good.  Pretty much any printer will
handle that today, though I reserve the right to change my mind about
the utility of 17cpi print after I'm 50. Hopefully, all printers will
be at least 1200dpi by then.  ;-)

---

Yes, I dislike "\" for continuation, and use the implicit continuation
between parentheses, braces, etc. wherever possible.  I don't mind
breaking up comma separated values like function method arguments, but
I very much dislike breaking at an "=" sign.

(BigLong VariableName
 = BigLongMethodCallWithALotOfArguments)

Come to think of it, though, I'm working mostly in TurboGears (web
devel framework) which may not represent python code in general all
that well wrt line length.  Lot's of lines that are harder to break up
than in some other environments, perhaps.

---

I'm finding 100 to be a nice balance.  It forces me not to be lazy and
allow really long lines, but allows me to format so as to make the
meaning most clear.

-- 
http://mail.python.org/mailman/listinfo/python-list

Fuzzy string comparison

[Steve Bergman]

I'm looking for a module to do fuzzy comparison of strings.  I have 2
item master files which are supposed to be identical, but they have
thousands of records where the item numbers don't match in various
ways.  One might include a '-' or have leading zeros, or have a single
character missing, or a zero that is typed as a letter 'O'.  That kind
of thing.  These tables currently reside in a mysql database.  I was
wondering if there is a good package to let me compare strings and
return a value that is a measure of their similarity.  Kind of like
soundex but for strings that aren't words.

Thanks,
Steve Bergman

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fuzzy string comparison

[Steve Bergman]

Thanks, all. Yes, Levenshtein seems to be the magic word I was looking
for.  (It's blazingly fast, too.)

I suspect that if I strip out all the punctuation, etc. from both the
itemnumber and description columns, as suggested, and concatenate them,
pairing the record with its closest match in the other file, it ought
to be pretty accurate.  Obviously, the final decision will be up to a
human being, but this should help them quite a bit.

BTW, excluding all the items that match exactly, I only have 8000 items
in one file to compare to 2600 in the other.  As fast as
python-levenshtein seems to be, this should finish in well under a
minute.

Thanks again.

-Steve

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: piping question

[Steve Bergman]

Have you tried running python with '-u'?  That turns off most buffering
within python at least.  I'm not familiar with newspost, so I've no
idea what to do about any output buffering it might be doing.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: scanning for numerals / letters

[Steve Bergman]

Something like this should work:

==
for c in form.get('date'):
  if c in string.letters:
print "ERROR: You have to enter a date with numbers."
==

You have to import the string module.  'letters' is one of the
attributes defined in that module.

Other attributes defined there include 'digits'. (Hint, hint. ;-) )

Also note that if you are not sure if a dictionary has a particular
key, you can use:

form.get('date')

and you will get the value if the key 'date' exists.  If not, you get
None.

It's handier than checking has_key.

You can also say:

form.get('date', '01/01/70')

If the dictionary has a key 'date', you will get the value associated
with it.  Othewise, you will get '01/01/70'.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Starting value with raw_input

[Steve Bergman]

Kinda ugly, and I lifted it from an old post, but this should work:

import readline
readline.set_startup_hook(lambda:
readline.insert_text('supercalifragilisticexpialidocious'))
try:
  new_value = raw_input()
finally:
  readline.set_startup_hook(None)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Building a web questionnaire, can it be done in Python ?

[Steve Bergman]

The most popular choice for web apps, and the one I use myself, would
be Django.  You might post your question in the Django group:

http://groups-beta.google.com/group/django-users

The only thing that I see that could be a problem would be the legacy
database.  But that depends very much upon your exact situation.
--
http://mail.python.org/mailman/listinfo/python-list

Python module for reading FilePro files?

[Steve Bergman]

Does anyone know of a Python package or module to read data files from
the venerable old Filepro crossplatform database/IDE?
-- 
http://mail.python.org/mailman/listinfo/python-list

How to make this unpickling/sorting demo faster?

[Steve Bergman]

I'm involved in a discussion thread in which it has been stated that:

"""
Anything written in a language that is > 20x slower (Perl, Python,
PHP) than C/C++ should be instantly rejected by users on those grounds
alone.
"""

I've challenged someone to beat  the snippet of code below in C, C++,
or assembler, for reading in one million pairs of random floats and
sorting them by the second member of the pair.  I'm not a master
Python programmer.  Is there anything I could do to make this even
faster than it is?

Also, if I try to write the resulting list of tuples back out to a
gdbm file, it takes a good 14 seconds, which is far longer than the
reading and sorting takes.  The problem seems to be that the 'f' flag
to gdbm.open() is being ignored and writes are being sync'd to disk
either on each write, or on close.  I'd really prefer to let the OS
decide when to actually write to disk.

I'm using python 2.5.2, libgdm 1.8.3, and python-gdbm 2.5.2 under
Ubuntu 8.4 beta and an  x86_64 architechture.

Thanks for any tips.

=
import cPickle, gdbm, operator
dbmIn = gdbm.open('float_pairs_in.pickel')
print "Reading pairs..."
pairs = cPickle.loads(dbmIn['pairs'])
print "Sorting pairs..."
pairs.sort(key=operator.itemgetter(1))
print "Done!"
=


The input file was created with this:

=
import random, gdbm, cPickle
print "Creating pairs file..."
pairs = [(random.random(), random.random(),) for pair in
range(0,100)]
dbmOut = gdbm.open('float_pairs_in.pickel', 'nf')
dbmOut['pairs'] = cPickle.dumps(pairs, 2)
dbmOut.close()
print "Done!"
=
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python module for reading FilePro files?

[Steve Bergman]

Thanks.  Yes, there is a php-filepro extension that I could hook up
with using the command line interpreter.  That may well be what I end
up doing.  Or maybe port the php extension to python.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to make this unpickling/sorting demo faster?

[Steve Bergman]

> THe above is applied slavishly by those who value machine time over
> peoples time. Do you want to work with them?

I understand where you are coming from.  But in writing the code
snippet I learned something about pickling/unpickling, which I knew
*about* but had never actually used before.  And also learned the
quick and easy way of DSU sorting, which got easier and faster in
2.4.  So, personally, I'm ahead on this one.  Otherwise, I agree that
arguing the matter in long flame threads is a waste of time.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to make this unpickling/sorting demo faster?

[Steve Bergman]

Thanks for the help.  Yes, I see now that gdbm is really superfluous.
Not having used pickel before, I started from the example in "Python
in a Nutshell" which uses anydbm.  Using a regular file saves some
time.  By far, the biggest improvement has come from using marshall
rather than cPickel. Especially for writing. gc.disable() shaved off
another 20%.  I can't use psyco because I'm on x86_64. And my goal is
to demonstrate how Python can combine simplicity, readability, *and*
speed when the standard library is leveraged properly.  So, in this
case, calling C or assembler code would defeat the purpose, as would
partitioning into smaller arrays.  And my processor is single core.

I started at about 7 seconds to just read in and sort, and at 20
seconds to read, sort, and write.  I can now read in, sort, and write
back out in almost exactly 5 seconds, thanks to marshal and your
suggestions.

I'll look into struct and ctypes.  Although I know *of* them, I'm not
very familiar *with* them.

Thank you, again, for your help.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to make this unpickling/sorting demo faster?

[Steve Bergman]

On Apr 17, 7:37 pm, Paul Rubin  wrote:

> Therefore the likelihood of a C or asm program
> being 20x faster including disk i/o is dim.  But realistically,
> counting just CPU time, you might get a 20x speedup with assembler if
> you're really determined, using x86 SSE (128-bit vector) instructions,
> cache prefetching, etc.

I think that the attitude that is prevalent is that although Python
and other "interpreted" languages are slow, there are places where
that slowness is not important and using them is OK.  The point that I
am trying to make with my example is that often, by leveraging the
care and optimization work that others have put into the Python
standard library over the years, a rather casual programmer can match
or better the performance of 'average' C code.  i.e. C code that was
written by a good C programmer while no one was looking over his
shoulder, and no pressures motivated him to spend a lot of time
optimizing the C to the hilt, or switching to asm.

With the reading and writing of the data (which actually works out to
about 23MB, marshalled) now down to 1 second each, I'm content.  In
the beginning, the io time was overshadowing the sort time by a good
bit.  And it was the sort time that I wanted to highlight.

BTW, no one has responded to my challenge to best the original sample
Python code with C, C++, or asm.

-- 
http://mail.python.org/mailman/listinfo/python-list

Sorting a list of objects by multiple attributes

[Steve Bergman]

Hi,

I am trying to come up with a clean and simple way to sort a list of
objects (in this case SQLObject instances) by multiple attributes.

e.g. a Person object may have an age, a lastName, and a firstName.

I'd like to be able to arbitrarily sort by any combination of those in
any order.

I would especially like it to be clear and simple, since I am
converting a web app from using SQL to using an ORM only and so the
sorting will actually be controlled by variables coming from SELECTs in
a web form.  In SQL this is easy enough to accomplish, and I can do it
in python with L.sort(key=lambda i: i.whatever).  But for the multiple
attribute case, I'm at a bit of a loss.

Thanks for any help.

-Steve

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help for a complete newbie

[Steve Bergman]

The indentation is wrong.  Python cares about indentation.


print " "
print "This \"autotp\" program will create raw bitmap test pattern
images."
print " "
print "Please read the information below thoroughly:"
print " "
print "1. Graphic files MUST be TIFF images."
print "2. Images MUST have been ripped though a Nuvera system as a
Print
and Save job"
print "3. Images should already be named with the desired file name."
print "4. The Lead Edge and file name should be identified in the
image."
print "5. The name includes the purpose for, resolution of, side, and
paper size"
print "6. Images should be rotated to print correctly from Service
Diagnostics."
print " "
print "EXAMPLE: Bypass_BFM_Damage_1200x1200_Letter.tif"
print " "

# Get the decision if operator is ready
ready = raw_input("Ready to proceed ? TYPE (y)es or (n)o: ")

if ready  == "y":
print "You are Ready"
else:
print "Try again"

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: HELP PLEASE: What is wrong with this?

[Steve Bergman]

Welcome to Python! :-)

You may find this mailing list useful:

http://mail.python.org/mailman/listinfo/tutor

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help for a complete newbie

[Steve Bergman]

You can also access this group through Google Groups:

http://groups.google.com/group/comp.lang.python

which has nice search features.

I found O'Reilly's "Learning Python" to be helpful in combination with
O'Reilly's "Python in a Nutshell".  Learning python is a nice
introduction to Python but is (intentionally) *far* from complete.
Python In a Nutshell is a reference work.  The two complement each
other nicely.

Both are a bit out of date.  The current python version is 2.4.
Learning Python covers 2.3.  Nutshell covers up to 2.2.

One of these days Alex will come out with a 2nd Edition of Nutshell and
there will be much rejoicing. ;-)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Should I learn Python instead?

[Steve Bergman]

There is also TurboGears, which (IMO) is greatness in the making.
Though unfortunately, the documentation is still catching up to the
greatness.  I'm using it and loving it, though.

http://www.turbogears.org

DJango is great for content management.  TurboGears is (IMO) a more
generalized framework.  It all depends on what kind of site you are
wanting to create.

-- 
http://mail.python.org/mailman/listinfo/python-list