Re: OT: Entitlements [was Re: Python usage numbers]

[Tim Wintle]

(Sorry for top-posting this bit, but I think it's required before the
rest of my response)

At the risk of wading into this from a UK citizen's perspective:

You're imagining a public healthcare system as if it were private.

Imagine you go to a doctor and say "I've got the flu, can you give me
antibiotics".

In a Private healthcare system:

 * The doctor gets paid for retaining a client.
 * He is incentivised to do what you request.
... so he gives you the antibiotics.

In a Public healthcare system:
 * The doctor is paid no matter what.
 * His job is to stop the population becoming ill.
 * By reducing illnesses he reduces his workload, without reducing his
wage

... so he'll only give you antibiotics if he feels you are at serious
risk, and giving you antibiotics carries less risk for the population
than the risk of the population getting immunities.

Same goes for surgery etc.

On Mon, 2012-02-13 at 08:01 -0800, Rick Johnson wrote:
> And just how much healthcare dollars are you entitled to exactly? Can
> you put your entitlement into some form of monetary value?
> 
> And how can we ever make a system like this fair? If someone works for
> 30 years and pays a 30% tax rate and another works for 2 years and
> pays 15%, then how do we delegate the fair share?

If your children are educated privately then should you still be paying
taxes for education?

If you work for/bank with a company that doesn't need to be bailed out,
then should you still pay tax for that?

If you never need benefits (welfare) then should your taxes be paying
for that?

you can use that same argument for everything that taxes pay for - the
only logical conclusion of that argument is anarchy (i.e. no taxes, and
no government).

If you are an anarchist then that's a different argument all together
(not saying it doesn't have intellectual validity).



> Healthcare is expensive. Do you want a minimum wage doctor curing your
> ills? And the frivolous lawsuits are not bringing the costs down
> either.

It's so expensive because of the marketing, and because of all the
middle-men.

A public health system doesn't need to do that marketing.

They also don't need to put up with people who aren't seriously ill - I
don't know how long your private appointments are, but here in the UK a
standard doctor's appointment is 5-10 minutes. If they decide you're
actually ill they may extend that.

> > - bosses win, because they have reduced absenteeism, lower training costs
> > to replace workers who die, and fewer epidemics that threaten their own
> > families
> 
> BS! With free healthcare, those who would have allowed their immune
> system fight off the flu, now take off from work, visit a local
> clinic, and get pumped full of antibiotics so they can create a new
> strain of antibiotic resistant flu virus! Thanks free healthcare!

See my comments at the top.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python math is off by .000000000000045

[Tim Wintle]

On Sat, 2012-02-25 at 09:56 -0800, Tobiah wrote: 
> > For every floating point
>  > number there is a corresponding real number, but 0% of real numbers
>  > can be represented exactly by floating point numbers.
> 
> It seems to me that there  are a great many real numbers that can be
> represented exactly by floating point numbers.  The number 1 is an
> example.
> 
> I suppose that if you divide that count by the infinite count of all
> real numbers, you could argue that the result is 0%.

It's not just an argument - it's mathematically correct.

The same can be said for ints representing the natural numbers, or
positive integers.

However, ints can represent 100% of integers within a specific range,
where floats can't represent all real numbers for any range (except for
the empty set) - because there's an infinate number of real numbers
within any non-trivial range.


Tim



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Tim Wintle]

On Thu, 2012-04-05 at 12:00 +, Steven D'Aprano wrote:
> The reason this is a Gotcha rather than a bug is because the JSON 
> standard specifies the behaviour (probably in order to be compatible with 
> Javascript).

It's not to be compatible with javascript (you can use either in
javascript)

I believe the choice is to make the parser as simple as possible. Agreed
it's a gotcha, but json is almost always generated automatically.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Half-baked idea: list comprehensions with "while"

[Tim Wintle]

On Fri, 2012-04-27 at 19:57 +1000, Chris Angelico wrote:
> On Fri, Apr 27, 2012 at 7:49 PM, Miles Rout  wrote:
> > We have if inside list comprehensions? I didn't know that, could you provide
> > an example?
> 
> You mean like:
> 
> [x*2+1 for x in range(10) if x%3]

Or like:

>>> print [ 0 if b%2==1 else 1 for b in range(10)]
[1, 0, 1, 0, 1, 0, 1, 0, 1, 0]


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: John Carmack glorifying functional programing in 3k words

[Tim Wintle]

On Wed, 2012-05-02 at 17:31 +0200, Tomasz Rola wrote: 
> positive aura drives more people and more permamently towards you. Perhaps he 
> should 
> develop an alter ego that could stand side by side with Dalai Lama and see 
> which one gets more attention.

Really?



If all you want is attention then "being nice" doesn't seem to be the
best option.

(of course if you want any respect...)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open Source: you're doing it wrong - the Pyjamas hijack

[Tim Wintle]

On Tue, 2012-05-08 at 15:20 +1000, Chris Angelico wrote:
> I hope that pyjamas can be restored at some point to a single live
> project. Whether that's headed by Luke Leighton or C Anthony Risinger
> (neither of whom I know at all and thus I can't speak to either's
> merits) or someone else, I don't particularly care

I have met Luke (At Europython), and honestly it was his enthusiasm that
got me to look at pyjamas in the first place. To be fair I still haven't
used it in anger, but I've poked around a lot, it's been under
consideration for several bits of work.

Although I don't think I've met C Anthony Risinger, his behaviour has
seriously put me off the project - and if I consider using it in the
future I'm going to be "pricing in" the cost of maintaining a complete
local fork as part of the decision.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Open Source: you're doing it wrong - the Pyjamas hijack

[Tim Wintle]

On Tue, 2012-05-15 at 12:39 +0200, Pascal Chambon wrote:
> believe me all this fuss is pitiful compared to the real harm that was
> done numerous time to willing newcomers, on pyjs' old ML, when they
> weren't aware about the heavy dogmas lying around.
> 
> A demo sample  (I quote it each time the suvject arises, sorry for
> duplicates)
>  
> | Please get this absolutely clear in your head: that
> | 
> | you do not "understand" my reasoning is completely and utterly
> | 
> | irrelevant.  i understand *your* reasoning; i'm the one making the
> | 
> | decisions, that's my role to understand the pros and cons.  i make a
> | 
> | decision: that's the end of it.
> | 
> | You present reasoning to me: i weight it up, against the other
> | 
> | reasoning, and i make a decision.  you don't have to understand that
> | 
> | decision, you do not have to like that decision, you do not have to
> | 
> | accept that decision.
> | 
>  

The above seems perfectly reasonable to me.

You're working with Python anyway - a language organised by a team that
gives full control to the BDFL...

Imagine instead that you were talking about a bug in a proprietary piece
of software (Oracle / Internet Explorer / etc) - do you think they'd let
*you* make the decision, or keep the option under discussion until *you*
fully understood the reasoning of the company that owned the code? No -
they'd listen to your argument, weigh up the two sides, and make a
decision on their own.

The idea of having two sides able to make their cases and one person
rule on them is incredibly common - it's how courts across the world
work, and it's how management of any team (software related or not)
goes.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: English version for Mémento Python 3 (draft, readers needed)

[Tim Wintle]

On Wed, 2012-06-06 at 12:03 +0200, Ulrich Eckhardt wrote:
> "block else for other cases" - this sounds as if it was blocking the
> else. Maybe "else-block for other cases",

I would say "else block". "else-block" is grammatically correct too, but
I don't think I've seen it used regularly.

RE: the order - "else" is being used as an adjective to clarify the noun
"block" - in English the adjective comes before the noun (unlike a lot
of European languages)

e.g. we say "the red book", not "the book red", where the French would
say "livre rouge" (I believe).

If you want to put an adjective after the noun (for poetical reasons
etc) then there needs to be another clause. e.g. "the book, which was
red"

> but English hyphenation is complicated and I'm not sure.

You're German and you say English hyphenation is complicated! ;-)

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: English version for Mémento Python 3 (draft, readers needed)

[Tim Wintle]

On Wed, 2012-06-06 at 16:56 -0400, Jerry Hill wrote:
> For what it's worth, I've never seen either of those constructs ("see
> overleaf" and "see over").  Are they perhaps more common in a
> particular academic context, or possibly more common in places that
> use "British English" spellings rather than "American English"?

Perhaps - "overleaf" is relatively common in documents here - while
filling out forms, in exam papers, etc.

However a quick search suggests the usage is in British and American
dictionaries with the same meaning.

Tim


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Unit test failing please help

[Tim Wintle]

On Fri, 2011-08-26 at 08:35 -0700, lblake wrote:
> Hi I am new to python I am at bit lost as to why my unit test is
> failing below is the code and the unit test:
> 
> class Centipede(object):
> legs, stomach

This doesn't do what you think it does.

"legs, stomach" is a statement and is not defining any variables at all.

Presumably you've also got variables named legs and stomach in the
module's scope - as I'd expect to see a NameError : name 'legs' is not
defined.

(I'd also expect a SyntaxError from having an empty __init__ function
body)

You probably want do write something like this:

class Centipede(object):
def __init__(self):
self.legs = []
self.stomach = []


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: why memoizing is faster

[Tim Wintle]

On Fri, 2011-03-25 at 09:49 +0100, Andrea Crotti wrote:
> Terry Reedy  writes:
> >
> > For the reason Stefan explained and hinted above. Try the following instead:
> >
> > def fib_iter(n, _cache = [1,1]):
> >   k = len(_cache)
> >   if n >= k:
> > for i in range(k, n+1):
> >_cache.append(_cache[i-2] + _cache[i-1])
> >   return _cache[n]
> >
> I don't get what's the difference between that and:
> 
> def fib_iter(n):
> ls = [0, 1]
> for i in range(2, n+1):
> ls.append(ls[i-2] + ls[i-1])
> 
> return ls[n]

> How can passing a default "_cache" argument can make such a difference?

Default values are initialised at definition time.

Since lists are mutable, the _cache variable really is a cache - the
following time you call the function it will store all the previous
calculations.

e.g.

>>> def default_value(_cache=[]):
...   _cache.append(len(_cache))
...   print _cache
... 
>>> default_value()
[0]
>>> default_value()
[0, 1]
>>> default_value()
[0, 1, 2]
>>> default_value()
[0, 1, 2, 3]



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Non-deterministic output

[Tim Wintle]

On Mon, 2011-03-28 at 12:42 +0200, Esben Nielsen wrote:
> We are making a prototype program in Python. I discovered the output was
> non-deterministic, i.e. I rerun the program on the same input files and
> get different output files. We do not use any random calls, nor
> threading.
> 
> One of us thought it could be set and dictionaries not always yielding
> the same results. I, however, would think that given the exact same
> operations, a set/dictionary would always yield the same results. Am I
> correct? Or could different runs of the same program yield different
> results due to, say, different memory locations?

If you're using id() at any point (e.g. in __hash__ ) then that would
lead to non-deterministic behaviour.

obviously so would time.time() etc.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fun python 3.2 one-liner

[Tim Wintle]

On Tue, 2011-04-05 at 15:38 +0200, Daniel Fetchinson wrote:
> Yeah, but we don't live in the 80's or 90's anymore and our screens
> can support xterms (or let alone IDE widows) much wider than 80
> characters. I'm using 140 for python these days. Seriously, who would
> want to limit him/herself to 80 characters in 2011?

I'd rather have two files open with 80 columns in them than a single
file with 160 columns and have to switch between files.

Tim Wintle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Questions about GIL and web services from a n00b

[Tim Wintle]

On Fri, 2011-04-15 at 12:33 -0400, Chris H wrote:
> 
> 1. Are you sure you want to use python because threading is not good
> due to the Global Lock (GIL)?  Is this really an issue for
> multi-threaded web services as seems to be indicated by the articles
> from a Google search?  If not, how do you avoid this issue in a
> multi-threaded process to take advantage of all the CPU cores
> available?

Is the limiting factor CPU?

If it isn't (i.e. you're blocking on IO to/from a web service) then the
GIL won't get in your way.

If it is, then run as many parallel *processes* as you have cores/CPUs
(assuming you're designing an application that can have multiple
instances running in parallel so that you can run over multiple servers
anyway).

Tim Wintle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The python implementation of the "relationships between classes".

[Tim Wintle]

On Thu, 2011-11-10 at 22:25 +0800, Jerry Zhang wrote:
> 
> 
> 2011/11/10 Chris Angelico 
> On Fri, Nov 11, 2011 at 12:58 AM, Jerry Zhang
>  wrote:
> > Cls_a:
> > def __init__(self):
> > self.at1 = 1
> > Cls_b:
> > def __init__(self):
> > self.com1 = Cls_a()
> > def __del__(self):
> > del self.com1
> > Is it a right implementation for composition?
> 
> 
> Yes, except that you don't need to explicitly del it. Python
> (at
> least, CPython) is reference-counted; your Cls_b object owns a
> reference to the Cls_a object, so (assuming nothing else has a
> reference) that Cls_a will be happily cleaned up when the
> Cls_b is.
> 
> Python doesn't really talk about "composition" etc. It's much
> simpler:
> everything's an object, and you have references to that
> object. A
> named variable is a reference to some object. A member on an
> object
> is, too. Whenever an object is expired, all objects that it
> references
> lose one reference, and if that was the sole reference, those
> objects
> get expired too. It's a change of thinking, perhaps, but not a
> difficult one in my opinion; and it's so easy to work with
> when you
> grok it.
> 
> 
> Unfortunately there is a difference between composition and
> aggregation in my real word, and my application really care this since
> it is trying to simulate this real world model, so my system should
> track this difference accurately, otherwise the system may not work
> well. 

You might want to look into weak references:
http://docs.python.org/library/weakref.html

Although I agree with Chris that it sounds like your code might be
easier with a change of perspective (I've not done much UML, but the way
you're describing things sounds like a java-ish way of looking at it to
me)

> For example, 
> a. the Cls_arm and Cls_body may be composition, but not aggregation.
> My app must ensure that " one arm instance only live with one body
> instance, if the body instance die, the arm instance must die.
>  b. the Cls_auto and the Cls_tyre may be aggregation. "One tyre still
> can live even the auto is dead."

That behaviour isn't really hard-coded as part of the language in
python, as it's not required for memory management in the same way it
would be in C++ or langauges without GCs.

As long as no objects other than body objects hold a reference to arm
objects then the arm object will be deleted.

For The tyre object to be deleted when the auto object is deleted, there
would have to be no references left to the tyre object. If there aren't
any references then you can't know if it exists or not anyway, so the
distinction isn't useful.

> Meanwhile, I have a ZODB running, which stores all the living
> objects. 

The ZODB is append only and stores all objects. Deleting references to
an object (which would causes deletion of standard python objects) won't
delete it from the zodb, it'll just delete the references.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Use and usefulness of the as syntax

[Tim Wintle]

On Sat, 2011-11-12 at 12:56 +0100, candide wrote:

> So what is the pragmatics of the as syntax ?

Another case:

try:
import json
except:
import simplejson as json


(same goes for several modules where the C implementation may or may not
be available)

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can't decompress data; zlib not available

[Tim Wintle]

On Sun, 2011-11-13 at 11:17 -0700, Steve Edlefsen wrote:
> 
> which appears to install zlib when python is reinstalled.  Except I
> can't run make without errors and there is no configuration file.
> 
> How do I reinstall python to include zlib?

Which OS are you on? Linux? BSD?

How did you install Plone?

First I'd check if there's a module shadowing the builtin zlib module -
i.e. if you've got a local file called "zlib.py" which is getting
imported by mistake.


Fairly much all *nix systems will have a python installation out of the
box - it looks like you need python2.6

I've never had a missing zlib module - but it's possible that it might
be missing if you don't have the zlib/deflate headers installed - if
they're not available then I'd try installing them and then reinstalling
the package you started with.

Tim Wintle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can't decompress data; zlib not available

[Tim Wintle]

On Mon, 2011-11-14 at 12:30 -0700, Steve Edlefsen wrote:
> I did a search on files named "python" on my machine.
> There are 23 not including the ones in the Plone
> buildout-cache in my account.  Seems like a lot of
> applications install their own copy of python.
> 
> There are also
> 
> ./usr/lib/x86_64-linux-gnu/libz.so
> ./usr/lib/libz.so
> 
> which, I believe, are the zlib libraries.

but do you have the headers?

On my ubuntu it's

/usr/include/zlib.h

As Christian pointed out, Ubuntu 11.04 introduced some changes which may
have broken the installation - there is a patch on the python bug
tracker which will provide a work-around, but it is not going to be
applied to 2.6 or lower as they are not actively maintained branches any
more.

You'll need to run through the steps that the plone installer makes and
patch the extracted python directory before it makes it's local python.

> I've read that you can reinstall python with configure
> using the "--with-zlib" option, but configure isn't in
> 
> /usr/local/Plone/Python-2.6/lib/python2.6/config
> 
> I think the python interpreter for the command line is
> the one in /usr/bin/python.  Would this be the one I
> reconfigure for zlib?  Should I simply install python from
> 
> Python-3.2.2.tgz?

no - you need to use python2.6 - python3 is effectively a different
language.

python is a core part of *nix these days, so playing around recompiling
the system python can cause a lot of pain.


Tim


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Make a small function thread safe

[Tim Wintle]

On Fri, 2011-12-16 at 05:21 -0800, Brad Tilley wrote:
> 107 void increment_counter( unsigned int& counter )
> 108 {
> 109 boost::mutex::scoped_lock lock( counter_lock );
> 110 ++counter;
> 111 }


with counter_lock:
counter += 1


... where counter_lock is a threading.Lock instance.

(see docs for the threading module)

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Make a small function thread safe

[Tim Wintle]

On Fri, 2011-12-16 at 09:24 -0500, Brad Tilley wrote:
> So something like this then:
> 
> import threading
> 
> shared_container = []
> lock = threading.Lock()
> 
> class thread_example( threading.Thread ):
> 
> def __init__( self ):
> threading.Thread.__init__ (self)
> 
> def run(t):
> lock
> shared_container.append(t.name)

should be:
  def run(t):
  with lock:
  shared_container.append(t.name)

(or lock.acquire() and lock.release() as you mentioned)

> # main
> 
> threads = []
> for i in xrange(10):
> thread = thread_example()
> threads.append(thread)
> 
> for thread in threads:
> thread.start()

you'll either need to lock again here, or join each thread:

for thread in threads:
  thread.join()

> for item in shared_container:
> print item 

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python philosophical question - strong vs duck typing

[Tim Wintle]

On Wed, 2012-01-04 at 11:30 -0300, Sean Wolfe wrote:
> On Tue, Jan 3, 2012 at 7:28 PM, Ben Finney  wrote:
> > Sean Wolfe  writes:
> >
> >> Hello everybody, I'm a happy pythonista newly subscribed to the group.
> >
> > Welcome!
> 
> Thanks! and thanks to all, hjaha.
> 
> >
> >> I have a theoretical / philosophical question regarding strong vs duck
> >> typing in Python. Let's say we wanted to type strongly in Python
> >
> > There may be an unstated assumption there, and your wording confuses me.
> >
> 
> yep, probably. I am throwing around terminology a bit. Here's another
> attempt -->
> 
> If I am willing to create some python code, when needed, where when I
> create a variable, let's say an integer, that it will be for all time
> an integer, and also that a method that returns say a Sprite custom
> object, and will for all time return only a Sprite object ... , does
> this get me significantly closer to being able to compile to C++?

I'd really recommend looking at Cython - which has optional static
typing and does compile into C / C++ (as a python extension)

More generally, a compiler can perform static analysis on code which
will re-order AST nodes into single constant assignments. I've forgotten
the name but it's something like single static assignment form. When the
return type of functions is known it can lead to known types for
variables.

It's being used heavily in the newest generation of javascript JITs to
speed up generated native code.

However, when a function has multiple return types (e.g. {}.get returns
None if there is no result) then you can't imply the type of the
variable even in this form.

A JIT (such as pypy) can generate the native code for all seen return
types - which is why JITs can in general be more useful to dynamically
typed languages such as Python than compilers.

Another issue is where types can be modified (e.g. in python you can
modify the class of an object at runtime) - dynamic language features
such as this make what counts as a "type" fairly flexible. JITs are
getting around this using "hidden classes" (there are lots of other
names for the same thing) - again it would be very difficult to
statically compile this kind of thing to native code.

> I am just thinking in my brain about the differences between cpp and
> python, and if there is a way to compromise a bit on the python-ness
> to get closer to cpp, but still be able to keep a lot of the goodness,
> then put in a translator or converter to cpp and gain performance by
> using cpp code. Sounds like Rpython, cython, shedskin are doing a lot
> or all of this, so lots to study up on.

Yup

Tim Wintle

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to support a non-standard encoding?

[Tim Wintle]

On Fri, 2012-01-06 at 10:03 +, Ivan wrote:
> Dear All
> 
> I'm developing a python application for which I need to support a 
> non-standard character encoding (specifically ISO 6937/2-1983, Addendum 
> 1-1989).

If your system version of iconv contains that encoding (mine does) then
you could use a wrapped iconv library to avoid re-inventing the wheel.

I've got a forked version of the "iconv" package from pypi available
here:



.. it should work on python2.5-2.7

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to support a non-standard encoding?

[Tim Wintle]

On Fri, 2012-01-06 at 12:00 -0800, jmfauth wrote:
> The distibution of such a codec may be a problem.

There is a register_codec method (or similar) in the codecs module.

Tim


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: replacing __dict__ with an OrderedDict

[Tim Wintle]

On Tue, 2012-01-10 at 09:05 -0500, Roy Smith wrote:
> 
> I guess MongoDB is not a serious database? 

That's opening up a can of worms ;)


... anyway, cassandra is far better.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

thread. question

[Tim Wintle]

Hi, 
This is my first post here - google hasn't helped much but sorry if this
has been asked before.

I've been wondering about some of the technicalities of locks in python
(2.4 and 2.5 mainly).

I'm using the "old" thread module as (1) I prefer the methods and (2) It
should be a tiny bit faster.

As far as I can see, the thread module is fairly much a direct wrapper
around "PyThread_allocate_lock" etc. - defined in
"thread_pthread.h" (took me ages to find the definitions in a header
file!), so as far as I can tell the implementation is very closely
wrapped around the system's threading implementation. 

Does that mean that every python thread is actually a real separate
thread (for some reason I thought they were implemented in the
interpreter)?

Does that also mean that there is very little extra interpreter-overhead
to using locks (assuming they very infrequently block)? Since if you use
the thread module direct the code doesn't seem to escape from C during
the acquire() etc. or am I missing something important?

Thanks,

Tim Wintle




--
http://mail.python.org/mailman/listinfo/python-list

Re: thread. question

[Tim Wintle]

Thanks for both replies,

On Mon, 2009-02-09 at 15:59 +0100, Christian Heimes wrote:
> You shouldn't use the thread module directly. It's not meant to be used
> by a user. Please stick to the threading module. You won't notice a
> slowdown, trust me :)
I'm aware that thread is being renamed to _thread in python 3.0, but is
it being depricated or anything like that?

This is for an app that has been running for quite a long time and it's
now time for fairly heavy optimisations as load is increasing (Believe
me, I wouldn't have been looking at the C otherwise) - so I'll see if I
do notice any effect with threading.


> Yes, Python uses native threads, not green threads. However Python pure
> code can't utilize more than one CPU per process. On order to use
> multiple CPUs you have to use multiple processes or use C code that
> doesn't use any Python API. Google for "GIL" if you are interested in
> more information.

Thanks for the info - I'm very aware of the GIL, but had never looked at
the implementation before and for some reason thought that python's
threads were in userspace.

I'm already using a multiple-process architecture overall, but using
threads to avoid waiting when handling SQL and TCP.

Tim


--
http://mail.python.org/mailman/listinfo/python-list

Re: thread. question

[Tim Wintle]

On Mon, 2009-02-09 at 21:02 +0100, Christian Heimes wrote:
> The module was renamed to _thread to stop people from using it directly.
> The extension module is the interface to some low level types and
> functions. Especially the usage of thread.start_new_thread is
> problematic, since it bypasses Python's high level threading API.

Ok, I'll take that in - I'll have a look around the threading module to
see what it is that it bypasses so I can persuade myself that _thread is
bad ...

> For the rest I have to agree with Jean-Paul. If you need performance
> don't use threads! Threads and performance are orthogonal -- sometimes
> they are even contradictorily.

I've taken that in too - "performance" here is over a non-uniform
workload (essentially a web server), but I'm probably best offloading
just about all the work to a secondary process (already passed most of
it) and dealing with requests in a completely synchronous process.

Tim

--
http://mail.python.org/mailman/listinfo/python-list

Re: Module to read/write MIDI ?

[Tim Wintle]

> I had my first look around pypi.python.org/pypi yesterday and didn't
> see anything.  Is there a MIDI-module for Python ?  If not, I'll
> email Sean to ask if he'd mind me translating his module into Python3...



This is the only project I have seen that aims to do that - I haven't
actually used it, and there might be others out there (I believe pygame
 supports midi-output).



> 
> Regards,  Peter
> 

--
http://mail.python.org/mailman/listinfo/python-list

Re: Speedup Bitmap Parser

[Tim Wintle]

On Mon, 2009-02-16 at 16:47 -0800, bearophileh...@lycos.com wrote:
> Jayson Santos:
> > Do I need use some patters or others progamming conventions ?
> 
> That's a strong question...
> Knowing OOP well is important, and it's often a good way to program,
> etc. But... You may try to rewrite your code with functions only and
> no classes (well, you may need to use classes for the TK gui).
> Feel free to not follow this suggestion of mine. Even if you follow
> it, it will not damage your brain :-)

I'd strongly agree with the above - object creation is overkill in
inner-loops.

Invoking functions and any kind of variable allocation is also going to
slow it down. That's what bearophile's previous comment about accessing
line.colors without the "." lookup by enumerating them was about.

At the end of the day, image manipulation is often better left to C (or
compiled extensions to python in Cython etc.) than done in Python - it's
just too tough to do pointer-magic in Python. You may find that so much
time is being spent by your application in the ".create_line" method
that there's no other option - I'd highly recommend PIL though.


Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: memory recycling/garbage collecting problem

[Tim Wintle]

On Tue, 2009-02-17 at 00:40 -0800, Chris Rebert wrote: 
> >
> > 'gc.collect()' -- I believe, but I'm not the specialist in it.
> 
> If I understand correctly, that only effects objects that are part of
> a reference cycle and doesn't necessarily force the freed memory to be
> released to the OS.

I believe that's correct.

If the OP is worrying about memory usage then they should also be aware
that there are lots of very clever things done pre-assigning and keeping
hold of memory with python's in-built types to let them scale well that
can be confusing when you're looking at memory usage.

Basically malloc() and free() are computationally expensive, so Python
tries to call them as little as possible - but it's quite clever at
knowing what to do - e.g. if a list has already grown large then python
assumes it might grow large again and keeps hold of a percentage of the
memory.

The outcome is that trying to reduce memory usage can change what data
structures you should use - tupples use less space than lists, etc.

Tim W

--
http://mail.python.org/mailman/listinfo/python-list

Re: memory recycling/garbage collecting problem

[Tim Wintle]

On Tue, 2009-02-17 at 17:04 +0100, Christian Heimes wrote:
> Tim Wintle wrote:
> > Basically malloc() and free() are computationally expensive, so Python
> > tries to call them as little as possible - but it's quite clever at
> > knowing what to do - e.g. if a list has already grown large then python
> > assumes it might grow large again and keeps hold of a percentage of the
> > memory.
> 
> You are almost right. Python's mutable container types like have a
> non-linear growth rate.
> 
> >From the file listobject.c
> 
> /*
> This over-allocates proportional to the list size, making room
> for additional growth.  The over-allocation is mild, but is
> enough to give linear-time amortized behavior over a long
> sequence of appends() in the presence of a poorly-performing
> system realloc().
> The growth pattern is:  0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...
> */
> 
> new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6);

Sorry, I think I didn't phrase myself very well - I was trying to
explain that de-allocation of memory follows different scaling behaviour
to allocation - so a large list that's shrunk is likely to take more
memory than a small list that's grown i.e. the part just above your
quote:


/* 
Bypass realloc() when a previous overallocation is large enough
to accommodate the newsize.  If the newsize falls lower than half
the allocated size, then proceed with the realloc() to shrink the list.
*/
if (allocated >= newsize && newsize >= (allocated >> 1)) {
assert(self->ob_item != NULL || newsize == 0);
Py_SIZE(self) = newsize;
return 0;
}


it's all very clever stuff btw.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Searching Google?

[Tim Wintle]

On Wed, 2009-02-18 at 13:36 +0100, Johannes Bauer wrote:
> Curt Hash schrieb:
> 
> > You just need to change your User-Agent so that Google doesn't know a
> > Python script is making the request:
> 
> Why would Google not send a response if the User-Agent header field is
> not a recognized browser? I doubt that's what's happening and the OP
> would just like the convenience of a search API instead of parsing the
> web page (like other applications do). Ugly, but it works.

I suspect that it's the other way around - Google has black-listed the
standard python user-agent rather than whitelisting useragents.

Think about how much power it takes to do a query on Google - if they
provided a search API they would lose out on advertising on the results
- which at the end of the day is their income-source.

It's a pain not to have a search API, but you've got to understand it!


--
http://mail.python.org/mailman/listinfo/python-list

Re: Revision Control

[Tim Wintle]

On Wed, 2009-02-18 at 09:08 -0800, Sam Clark wrote:
> Any suggestions for a beginer on what to use for version control?
> It's just me, the lone person programming.  I've already nailed one
> "version" of my code accidentaly.  MS VSS is too expensive for the
> stuff I'm doing, plus I really don't like MS much...  Any free open
> source stuff out there?

I use svn (subversion) at work and it works fine, but I'm using bazaar
 on my own code, and even alongside svn to help
with merging branches, porting code up-stream etc.

bazaar is really great in my experience, although relatively new so you
might prefer to stick to svn. I haven't tried using bazaar on windows.


If you do go with svn, try the latest version, since merging support is
much better in this.


Tim


--
http://mail.python.org/mailman/listinfo/python-list

os.fdopen giving "Invalid Argument" from pipes

[Tim Wintle]

Was wondering if someone could point out what the stupid thing I'm doing
wrong is:


{{{
import os, time
def run_multi_proc():
server_send, logger_recieve = os.pipe()
pid = os.fork()
if pid == 0:
# we are the logger
#os.close(server_send)
logger_recieve = os.fdopen(logger_recieve)
time.sleep(20)
else:
# we are the server
#os.close(logger_recieve)
server_send = os.fdopen(server_send,"w")
time.sleep(3)
os.kill(pid,signal.SIGTERM)

if __name__ == "__main__":
run_multi_proc()
}}}


> python test.py

Traceback (most recent call last):
  File "test.py", line 20, in 
run_multi_proc()
  File "test.py", line 9, in run_multi_proc
logger_recieve = os.fdopen(logger_recieve)
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "test.py", line 20, in 
run_multi_proc()
  File "test.py", line 15, in run_multi_proc
server_send = os.fdopen(server_send,"w")
OSError: [Errno 22] Invalid argument


really confused about why this isn't working - running on Linux with
python 2.5.2

Tim W

--
http://mail.python.org/mailman/listinfo/python-list

Re: os.fdopen giving "Invalid Argument" from pipes

[Tim Wintle]

On Thu, 2009-02-19 at 11:50 -0500, Jean-Paul Calderone wrote:
> You got server_send and logger_receive backwards 

Doh!

> (also, i before e *except* after c et cetera).  Flip 'em around and
> all is well.

Thanks - never was great at speling :-)

Tim

--
http://mail.python.org/mailman/listinfo/python-list

Re: Will multithreading make python less popular?

[Tim Wintle]

On Thu, 2009-02-19 at 12:48 -0800, Falcolas wrote:
> That's fair, but by using a high level language in the first place,
> you've already made the conscious decision to sacrifice speed for ease
> of programming. Otherwise, you would probably be programming in C.
My parents would have gone mad at me for saying that when I was young -
C is just about the highest-level language they ever used - Assembly/hex
all the way!

So if you really want speed then why don't you write your code in
assembly? That's the only "perfect language" - it's capable of doing
everything in the most efficient way possible on your machine.

Of course that's a hassle, so I guess you may as well use C, since
that's almost certainly only linearly worse than using assembly, and it
takes far less time to use.

Oh, but then you may as well use python, since (again) that's probably
only linearly worse than C, and well-programmed C at that - I certainly
wouldn't end up with some of the optimisations that have gone into the
python interpreter!

That's basically what my mind goes through whenever I choose a language
to use for a task - and why I almost always end up with Python.

> It's a proposition that used to bother me, until I did some actual
> programming of real world problems in Python. I've yet to really find
> a case where the application was slow enough to justify the cost of
> using multiple Python processes.

I deal with those situations a fair bit - but the solutions are normally
easy - if it's limited by waiting for IO then I use threads, if it's
limited by waiting for CPU time then I use multiple processes, or share
the job over another application (such as MySQL), or run a task over a
cluster.

If you have a task where the linear optimisation offered by multiple
cores is really important then you can either:
 * Run it over multiple processes, or multiple machines in Python
or
 * spend a year writing it in C or assembly, by which time you can buy a
new machine that will run it fine in Python.


Yes, we're coming to a point where we're going to have tens of cores in
a chip, but by that time someone far cleverer than me (possibly someone
who's on this list) will have solved that problem. The top experts in
many fields use Python, and if they weren't able to make use of multiple
core chips, then there wouldn't be any demand for them.

Tim Wintle


> 
> ~G
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: Will multithreading make python less popular?

[Tim Wintle]

On Thu, 2009-02-19 at 12:18 -0800, Paul Rubin wrote:
> If such
> speedups were useless or unimportant, we would not have blown our hard
> earned cash replacing perfectly good older hardware, so we have to
> accept the concept that speed matters and ignore those platitudes that
> say otherwise.

Kind of agree (although I use a netbook at lot at the moment, and I
don't use that because of speed-ups!)

> It used to be that new computers were faster than the old ones because
> they ran at higher clock rates.  That was great, no software changes
> at all were required to benefit from the higher speed.  Now, they get
> the additional speed by having more cores.  That's better than nothing
> but making use of it requires fixing the GIL.

My opinion on this (when talking about personal computers rather than
servers) is that:

(1)
Computers *appear* faster now because they have more cores - you can
have one doing the fancy GUI effects of Compiz etc. in the background,
while the other core actually does the work.

(2) 
Actual speedups aren't that related to either clock speed or cores at
the moment, they're related to the switch to 64-bit processors, the
massive increases in RAM and the increase in system bus speeds and other
IO (and of course graphics cards). I suspect that the next common
increase will be solid state hard disks.

e.g. I used to expect my computer to be paging all the time, although
I'd try to reduce it - these days I'm really upset when I see I've had
to page *anything* to disk!

Another massive increase (which I am willing to pay more for with the
work I do) is the processor cache - at first it was amazing when we got
significant level2 cache advertised on pc equipment, now I can fit
massive amounts of code into my 4mb level-2 cache *on my laptop*! That's
a huge impact for numerical work.

(3)
Multiple cores scale processing power linearly at best with the number
of cores (since you're normally going to be IO based at some point).
Perhaps the GIL will be relaxed a bit, but it's not going to have a
massive effect.

Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: code challenge: generate minimal expressions using only digits 1,2,3

[Tim Wintle]

On Fri, 2009-02-20 at 16:38 +, Nigel Rantor wrote:
> Luke Dunn wrote:



That was my initial thought when I read this too - but I'm not certain
that is guaranteed to find a solution (i.e. a solution that's optimal).

I'd welcome a proof that it will though, a few minutes thought hasn't
found a counter-example.

> > yes power towers are allowed
> 
> right, okay, without coding it here's my thought.
> 
> factorise the numbers you have but only allowing primes that exist in 
> your digit set.
> 
> then take that factorisation and turn any repeated runs of digits 
> multiplied by themselves into power-towers
> 
> any remainder can then be created in other ways, starting with a way 
> other than exponentiation that is able to create the largest number, 
> i.e. multiplication, then addition...
> 
> I've not got time to put it into code right now  but it shouldn't be too 
> hard...
> 
> e.g.
> 
> digits : 3, 2, 1
> 
> n : 10
> 10 = 2*5 - but we don't have 5...
> 10 = 3*3 + 1
> 10 = 3^2+1
> 3 digits
> 
> n : 27
> 27 = 3*3*3
> 27 = 3^3
> 2 digits
> 
> n : 33
> 33 = 3*3*3 + 6
> 33 = 3*3*3 + 3*2
> 33 = 3^3+3*2
> 4 digits
> 
> > exponentiation, multiplication, division, addition and subtraction. 
> > Brackets when necessary but length is sorted on number of digits not 
> > number of operators plus digits.
> >  
> > I always try my homework myself first. in 38 years of life I've 
> > learned only to do what i want, if I wanted everyone else to do my work 
> > for me I'd be a management consultant !
> > On Fri, Feb 20, 2009 at 3:52 PM, Luke Dunn  > > wrote:
> > 
> > I am teaching myself coding. No university or school, so i guess its
> > homework if you like. i am interested in algorithms generally, after
> > doing some of Project Euler. Of course my own learning process is
> > best served by just getting on with it but sometimes you will do
> > that while other times you might just choose to ask for help. if no
> > one suggests then i will probably shelve it and come back to it
> > myself when I'm fresh.
> >  
> > no it's not a real world problem but my grounding is in math so i
> > like pure stuff anyway. don't see how that is a problem, as a math
> > person i accept the validity of pure research conducted just for
> > curiosity and aesthetic satisfaction. it often finds an application
> > later anyway
> >  
> > Thanks for your helpful suggestion of trying other methods and i
> > will do that in time. my motive was to share an interesting problem
> > because a human of moderate math education can sit down with this
> > and find minimal solutions easily but the intuition they use is
> > quite subtle, hence the idea of converting the human heuristic into
> > an algorithm became of interest, and particularly a recursive one. i
> > find that the development of a piece of recursion usually comes as
> > an 'aha', and since i hadn't had such a moment, i thought i'd turn
> > the problem loose on the public. also i found no online reference to
> > this problem so it seemed ripe for sharing.
> > 
> > On Fri, Feb 20, 2009 at 3:39 PM, Nigel Rantor  > > wrote:
> > 
> > Trip Technician wrote:
> > 
> > anyone interested in looking at the following problem.
> > 
> > 
> > if you can give me a good reason why this is not homework I'd
> > love to hear it...I just don't see how this is a real problem.
> > 
> > 
> > we are trying to express numbers as minimal expressions
> > using only the
> > digits one two and three, with conventional arithmetic. so for
> > instance
> > 
> > 33 = 2^(3+2)+1 = 3^3+(3*2)
> > 
> > are both minimal, using 4 digits but
> > 
> > 33 = ((3+2)*2+1)*3
> > 
> > using 5 is not.
> > 
> > I have tried coding a function to return the minimal
> > representation
> > for any integer, but haven't cracked it so far. The naive first
> > attempt is to generate lots of random strings, eval() them
> > and sort by
> > size and value. this is inelegant and slow.
> > 
> > 
> > Wow. Okay, what other ways have you tried so far? Or are you
> > beating your head against the "search the entire problem space"
> > solution still?
> > 
> > This problem smells a lot like factorisation, so I would think
> > of it in terms of wanting to reduce the target number using as
> > few operations as possible.
> > 
> > If you allow exponentiation that's going to be your biggest
> > hitter so you know that the best you can do using 2 digits is
> > n^n where n is the largest digit you allow yourself.
> > 
> > Are you going to allow things like n^n^n or not?
> > 
> >  n
> > 
> > 
> > 
> > 
> 
> --
> http://m

Re: Forwarding keyword arguments from one function to another

[Tim Wintle]

On Sun, 2009-02-22 at 11:44 -0800, Ravi wrote:
> The following code didn't work:
> 

> def g(self, s, kwds):
> print s
> print kwds

This expects the function g to be called with the parameters "s" and
"kwds"


> def g(self, s, **kwds):
> print s
> print kwds

This expects to be passed the parameter "s", and various keyword
arguments, which will be put into the dict "kwds".

when you call 

o.g("string",**kwds)

you are passing the parameter "string" as the first parameter, and then
a sequence of keyword arguments taken from kwds, which will be passed
separately.

This is what the second form expects, but not what the first one
expects.

Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python shell scripting and errors

[Tim Wintle]

On Mon, 2009-02-23 at 09:12 -0800, Phillip B Oldham wrote:
> I've got a python script running as a daemon (using someone else's
> daemon module). It runs fine for a while, but will occasionally balk
> and die. Since its running in the background, I'm getting no error
> from it.
> 
> What's the best way to capture the output into a file for later review?

Point standard out to an open file handle?

--
http://mail.python.org/mailman/listinfo/python-list

Re: Challenge: Please break this! [Python Security]

[Tim Wintle]

On Mon, 2009-02-23 at 20:50 +, tav wrote:
> I'm keen to know your experiences even if you don't manage to write to
> the filesystem -- and especially if you do!
> 

er sorry, but:

from safelite import FileReader
reload(__builtins__)
f = open("/home/tim/nano.save","w")
f.write("oops")
f.close()

> Dinner and drinks on me for an evening -- when you are next in London
> or I am in your town -- to the first person who manages to break
> safelite.py and write to the filesystem.

I'm in London on Wednesday ;-) Unfortunately I'm going to be too busy to
take up your offer, lol


Tim Wintle




--
http://mail.python.org/mailman/listinfo/python-list

Re: Challenge: Please break this! [Python Security]

[Tim Wintle]

On Mon, 2009-02-23 at 13:20 -0800, Paul McNett wrote:
> tav wrote:
> > I'm keen to know your experiences even if you don't manage to write to
> > the filesystem -- and especially if you do!
> 
> Does it count when it breaks some standard libs that aren't even trying to 
> write to 
> the filesystem?

It appears to prevent any imports from being allowed at all.

> 
> mac:ss pmcnett$ python sbs_studio.py pkm
> Traceback (most recent call last):
>File "sbs_studio.py", line 159, in 
>  main()
>File "sbs_studio.py", line 15, in main
>  parser = OptionParser()
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/optparse.py",
>  line 
> 1191, in __init__
>  self.set_usage(usage)
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/optparse.py",
>  line 
> 1266, in set_usage
>  self.usage = _("%prog [options]")
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/gettext.py", 
> line 566, in gettext
>  return dgettext(_current_domain, message)
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/gettext.py", 
> line 530, in dgettext
>  codeset=_localecodesets.get(domain))
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/gettext.py", 
> line 465, in translation
>  mofiles = find(domain, localedir, languages, all=1)
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/gettext.py", 
> line 437, in find
>  for nelang in _expand_lang(lang):
>File 
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/gettext.py", 
> line 131, in _expand_lang
>  from locale import normalize
> ImportError: cannot import name normalize
> 
> Paul
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: more on unescaping escapes

[Tim Wintle]

On Mon, 2009-02-23 at 17:00 -0700, bvdp wrote:
> Let's see if this makes sense:
> 
>  >>> a='c:\\Program Files\\test'
>  >>> a.decode('string-escape')
> 'c:\\Program Files\test'

Hint: try running

>>> print a

and see what's written - I think that the interpreter adds extra "\"
characters to escape things and make things more easy to read.

i.e. 

>>> a = "c:\\test\\t"
>>> a
'c:\\test\\t'
>>> print a
c:\test\t
>>> 

so when it displays strings in the interpreter it includes escape
characters, when it is printed though the output is straight to stdout
and isn't escaped.

Hope that helps,

Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: Run a python script as an exe and run a new process from it

[Tim Wintle]

On Thu, 2009-02-26 at 04:55 -0800, venutaurus...@gmail.com wrote:
> Hello all,
>I've a strange requirement where I need to run a python
> script just as we run an exe (by double clicking through windows
> explorer or by typing the script name at command prompt).
I don't know how windows deals with this part
>  In that
> process I should be able to execute another python script in such a
> way that, the second script should continue running but the main one
> should terminate without effecting the second one.
standard daemon behavior I'd have thought

This shows how you would do it (with lots of comments). Assume this
should work on Windows.

http://code.activestate.com/recipes/278731/

note the two child threads to prevent zombies - may not be needed on
windows but good to use them anyway.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Run a python script as an exe and run a new process from it

[Tim Wintle]

On Thu, 2009-02-26 at 06:00 -0800, venutaurus...@gmail.com wrote:
> Thanks for the reply,
>Being a newbie to python, I am finding it difficult to
> understand the logic even after thorough reading of comments. Is there
> any simpler way where I can just run a python script from the main
> script and exit without disturbing the second one(This will end based
> on some other constraints which can be handled). Or can some one throw
> some light on where should I run the function os.system(" python
> script2.py") from the main one.

That would be a simple way of doing it :-)

--
http://mail.python.org/mailman/listinfo/python-list

Re: best way to parse a function-call-like string?

[Tim Wintle]

On Thu, 2009-02-26 at 21:29 +, m...@pixar.com wrote:
> I have some strings that look like function calls, e.g.
> 
> "junkpkg.f1"
> "junkpkg.f1()"
> "junkpkg.f1('aaa')"
> "junkpkg.f1('aaa','bbb')"
> "junkpkg.f1('aaa','bbb','ccc')"
> "junkpkg.f1('aaa','with,comma')"
> 
> and I need to split them into the function name and list of parms,
> e.g.
> 
> "junkpkg.f1", []
> "junkpkg.f1", []
> "junkpkg.f1", ['aaa']
> "junkpkg.f1", ['aaa','bbb']
> "junkpkg.f1", ['aaa','bbb','ccc']
> "junkpkg.f1", ['aaa','with,comma']
> 

quick and dirty

for s in string_list:
if "(" in s and s[-1] == ")":
  parts = s.split("(")
  fn, args = s[0],s[1][:-1].split(",")


Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: best way to parse a function-call-like string?

[Tim Wintle]

On Thu, 2009-02-26 at 21:39 +, Tim Wintle wrote:
> On Thu, 2009-02-26 at 21:29 +, m...@pixar.com wrote:
> > "junkpkg.f1", ['aaa','with,comma']
oops - missed this one, ignore my last reply.




--
http://mail.python.org/mailman/listinfo/python-list

Re: Pickle Problem

[Tim Wintle]

On Tue, 2009-03-03 at 15:33 -0800, Fab86 wrote:
> I have been trying except SearchError: however I get the error:
> 
> Traceback (most recent call last):
>   File "C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py",
> line 19, in 
> except SearchError:
> NameError: name 'SearchError' is not defined
> 
> I have searched all documents for terms along the lines of searcherror
> but am finding nothing..

Try looking at the file
C:\Downloads\MoS\yws-2.12\Python\pYsearch-3.1\timeDelay.py

and see where it imports SearchError from (or defines it)

Exceptions are just classes, so you'll have to import and reference it
like you'd reference any other class you import.

Hope that helps


Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Roulette wheel

[Tim Wintle]

On Wed, 2009-03-04 at 18:02 +, mattia wrote:
> ri = randint(0, len(rw) - 1)
> print("Random index:", rw[ri], ", value:", pop[rw[ri]])

you probably want random.choice(rw)

--
http://mail.python.org/mailman/listinfo/python-list

Re: Can Python do shopping cart?

[Tim Wintle]

On Fri, 2009-03-06 at 11:59 +0100, Marco Mariani wrote:
> Lie Ryan wrote:
> 
> > Python is Turing Complete
> 
> Well, actually no, because it doesn't support an infinite amount of memory.

Surely you can address an infinite amount of storage using infinite
length integers and a wrapper to files on disk - then it's just your
OS's limits that hold it back - so python is turing/register complete.

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: strings for beginers

[Tim Wintle]

On Sat, 2009-03-07 at 12:53 -0800, Sapote wrote:
> I have an incrementing variable disc_num that I could insert in the
> line below to create discspanisoX.iso where X is incrementing...
> 
>   burn_cmd = "mkisofs   -udf  -o /home/donkey/discspaniso.iso -graft-
> points --path-list %s" %(temp_list)

burn_cmd = "mkisofs   -udf  -o /home/donkey/discspaniso%d.iso
-graft-points --path-list %s" %(x,temp_list)

(i.e. %d for an integer, %s for a string, %f for a float etc.)

--
http://mail.python.org/mailman/listinfo/python-list

Re: strings for beginers

[Tim Wintle]

On Sat, 2009-03-07 at 21:25 +, Tim Wintle wrote:
> burn_cmd = "mkisofs   -udf  -o /home/donkey/discspaniso%d.iso
> -graft-points --path-list %s" %(x,temp_list)

obviously I meant to say

burn_cmd = "mkisofs   -udf  -o /home/donkey/discspaniso%
d.iso-graft-points --path-list %s" %(disc_num,temp_list)

- hadn't noticed the variable was disc_num.

Full explanation should be easily found on Google.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Themed TK (tk Tile) at last?!

[Tim Wintle]

On Sun, 2009-03-08 at 08:37 +1100, Python Nutter wrote:
> Looks like we finally get tkinter GUI based programs according to
> Issue# 2983 in Python 3.1a so our programs don't look like something
> out of early 1980's and can be themed to more closely match the
> underlying Operating Systems widget set!
Wow, I've never used tkinter for a proper app because of it's looks, but
I may now once I start using python 2.7 (or 3.1)

> the desire to use wxWidgets or Qt etc was high because tkinter
> widgets just look so horrid. 

How come nobody ever mentions Gtk?

> Good job python-dev team.

And especially Guilherme Polo for doing all the tough work as far as I
tell from the issue and the commit logs.
Am I right this was a Google Summer of Code project? If so then thanks
to Google as well for sponsoring it.


Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: should i move on to python3

[Tim Wintle]

On Sun, 2009-03-08 at 09:15 +1100, Python Nutter wrote:
> Maybe if everyone shares their own thinking for their own situations
> it may help.

Well, at work I do a mixture of things, some of which require python 2.3
(I know...), and some of which I can write to whatever version I want. I
generally use 2.5 for the second group. This is all work that runs live
all the time and has money running through it, so I'd rather not risk
moving to 3 for *any* of it until:

a) any security holes in python 3 have been fixed
b) it costs me more to stick with 2.x than to go through all of my code
line by line.

At home I normally write code for 2.5, as that's what comes with Ubuntu
Hardy (on my main, stable, machine), and most users will have 2.5 or 2.6
for quite a long time. For projects that are released, I'm planning to
stick to the advice Guido gave at Europython and keep working on 2.5/2.6
in trunk, but automatically generate a 3.x branch using 2to3.


I think it's going to be a bit like Java's JVMs (I can barely write
Java, but I use some Java projects) - I've got Java 5 and 6 running
different applications side by side here. In fact, I've got Sun's JVM
for 6 and 5, and IBM's JDK for Java 5 all running.

Similarly, on various machines I use CPython 2.3, 2.4, 2.5, 2.6 and
Jython 2.2 for various reasons - and I'm certainly planning on using
PyPy a large amount once it's stable.

I used the Beta of 3.0, but to be honest I haven't used it for anything
"proper" yet.

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a better way of doing this?

[Tim Wintle]

On Sun, 2009-03-08 at 15:49 +1100, Steven D'Aprano wrote:
> If the environmental costs of recycling something are worse than the
> environmental costs of throwing it away and making a new one, then
> recycling that object is actually harmful. But I digress.

Unless you live in a country that imports most of these goods, in which
case by recycling you keep money in the economy rather than buying goods
from elsewhere - it's never mentioned, but I'm fairly certain that's one
of the main reasons that the UK government loves forcing us to recycle
so much.

(obviously doesn't change how "environmentally harmful" something is)

Tim Wintle



--
http://mail.python.org/mailman/listinfo/python-list

Re: Chandler, Python, speed

[Tim Wintle]

On Sat, 2009-03-07 at 22:05 +, Ville M. Vainio wrote:
> Alan G Isaac wrote:
> 
> > 3. Chandler is not really an email client.  So specifically,
> > which of its functionalities is it slow, and what evidence
> > if any is there that Python is causing this?
> 
> I remember reading "somewhere" that the cause of slowness is/was
> architectural - perhaps it was that chandler was persisting too much stuff
> to disk, or something. In any case, this might help you google for more
> detail.

I've been using it, and the only major issue I've got is the long
shutdown times, which is caused by backing up a copy of _Everything_ to
disk, so that upgrades can happen cleanly - sure that's going to be
fixed though.

Startup time is a bit slow too, but it's designed to be left open all
the time, and it's fairly zippy once it's open IMO.

Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: Is python worth learning as a second language?

[Tim Wintle]

On Mon, 2009-03-09 at 11:19 +, Lie Ryan wrote:
> Certainly. A programmer that only knows one language would be too 
> limited. Try as many programming language as you can, and especially 
> look for programming languages that have "obscenely different" paradigm 
> than the language you already know.
I completely agree
> 
> You should know at least a language from each categories (anyone can add 
> if they feel something is missing):
> - Object oriented, example: C-family, Java, Python, etc
> - Imperative, example: C-family, Java, Python, etc
> - Functional, example: Python, Lisp/Scheme, Haskell, etc
> - Declarative, example: Haskell, Prolog
> - Logic Programming, example: Prolog, etc
> - Event driven, example: most GUI sublanguage, etc
> - Domain specific language, example: Regular Expression (yes it is a 
> programming language, regex parser is a Finite State Machine), SQL, etc
> - Concurrent programming, example: Erlang, etc
> - any other paradigms

My slight issue with this list that I think things are in too many
places. E.g. although you can do functional programming in Python (and
many do), I think it's worth trying to learn a language like lisp just
for the sake of forcing yourself to fully understand the paradigm.

I also think it's worth writing simple programs in a low level - either
in assembly, or as Turing/Register machine code.


> so basically, I still have much to learn...

so do I!


--
http://mail.python.org/mailman/listinfo/python-list

Re: Ban Xah Lee

[Tim Wintle]

On Mon, 2009-03-09 at 21:28 -0700, Luis Gonzalez wrote:
> C'mon guys, Xha Lee always wins, because fools like you get mad at him
> instead of ignoring him.

Here here!


--
http://mail.python.org/mailman/listinfo/python-list

Re: Memory efficient tuple storage

[Tim Wintle]

On Fri, 2009-03-13 at 08:59 -0700, psaff...@googlemail.com wrote:
> I'm reading in some rather large files (28 files each of 130MB). Each
> file is a genome coordinate (chromosome (string) and position (int))
> and a data point (float). I want to read these into a list of
> coordinates (each a tuple of (chromosome, position)) and a list of
> data points.
> 
> This has taught me that Python lists are not memory efficient, because
> if I use lists it gets through 100MB a second until it hits the swap
> space and I have 8GB physical memory in this machine. I can use Python
> or numpy arrays for the data points, which is much more manageable.
> However, I still need the coordinates. If I don't keep them in a list,
> where can I keep them?

If you just have one list, of objects then it's actually relatively
efficient, it's if you have lots of lists that it's inefficient.

I'm not certain without seeing your code (and my biology isn't good
enough to know the answer to my question below)

How many unique chromosome strings do you have (by equivalence)?

If the same chromosome string is being used multiple times then you may
find it more efficient to reference the same string, so you don't need
to have multiple copies of the same string in memory. That may be what
is taking up the space.


i.e. something like (written verbosely)

reference_dict = {}
list_of_coordinates = []
for (chromosome,posn) in my_file:
chromosome = reference_dict.setdefault(chromosome,chromosome)
    list_of_coordinates.append((chromosome,posn))

(or something like that)


Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: Style question - defining immutable class data members

[Tim Wintle]

On Sun, 2009-03-15 at 10:39 -0700, John Posner wrote:
> (My apologies if the thread has already covered this.) I believe I understand 
> the WHAT in this situation, but I don't understand the WHY ...

> Is there a beneficial effect of silently creating the instance attribute, 
> which outweighs the detrimental effects: (1) inconsistency, (2) the 
> "surprising" decoupling?

>From an end-user point of view (rather than the point of accessing class
methods others mentioned), here are two more reasons:

1) To save memory - there's only one object stored, where an instance
variable will require the same memory every instance (normally)


2) referencing Zope objects (although any persistent object storage will
have the same argument):

(Zope is a web framework that stores data as instances of python classes
in a persistent database.)

let's say you have the class (missing all the boilerplate)

{{{
class ZMyUser():
def __init__(self):
self.lookuplanguage = {\
"EN": "English"
}
self.language = "EN"

def get_friendly_name(self):
return self.lookuplanguage.get(self.language,"--")
}}}

And you create loads of these objects. But then you want to add French
as an option. How do you do that? All the objects have already been
created, so you could either: 

 * Run through every object in the database one by one and update the
dict as a one-off task (potentially taking a very long time as you force
the computer to pull the entire database off disk)

 * Have written  lookuplanguage as a class property, so re-initialising
the class will update the definition without any extra overhead.




--
http://mail.python.org/mailman/listinfo/python-list

Re: Style question - defining immutable class data members

[Tim Wintle]

On Mon, 2009-03-16 at 04:02 +, Tim Wintle wrote:
> On Sun, 2009-03-15 at 10:39 -0700, John Posner wrote:

Doh, reply out of thread there - I meant to reply to Rhodi's comment
further down.
> Is there any actual advantage to self.attribute picking up
> Class.attribute instead of raising a NameError?


--
http://mail.python.org/mailman/listinfo/python-list

Re: Run on Startup

[Tim Wintle]

On Wed, 2009-03-18 at 11:13 -0700, Mike Driscoll wrote:
> On Mar 18, 1:09 am, "Gabriel Genellina" 
> > Any decent installer is able to register a program so it runs on startup  
> > (InnoSetup, by example). Anyway, if you want to it it yourself, 
> > see:http://msdn.microsoft.com/en-us/library/bb776426(VS.85).aspx
> >
> > --
> > Gabriel Genellina
> 
> How is putting your program in "C:\Program Files" stupid? Almost every
> program I've ever installed on Windows defaults to that location.
> Putting a link in the Startup folder is kind of silly, I will grant
> you that. Of course, putting the shortcut there makes it easier for
> non-techie people to remove it. Otherwise you'll have to hack the
> registry to stop the startup or run msconfig and stop auto-start
> there.

The reason is that on non-english language versions it's not called "C:
\Program Files" - there's a system setting that installers use to know
the correct place to put applications. You should ask windows where it
thinks you should put applications, and then put it there.

The "Startup" folder is (I believe) depricated, and I would have guessed
that it's not called "Startup" in other languages.

> 
> Mike
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: Another of those "is" issues.

[Tim Wintle]

On Fri, 2009-03-20 at 11:20 -0700, Emanuele D'Arrigo wrote:
> >>> def aFunction():
> ... pass
> ...
> >>> f = aFunction
> >>> f is aFunction
> True   <--- Ok, this seems reasonable. Nevertheless, I suspect I
> shouldn't quite rely on it.

You can rely on this in the above - you've just assigned the name "f" to
the same object as "aFunction"

> 
> >>> class MyClass(object):
> ... def myMethod(self):
> ... pass
> ...
> >>> c = MyClass()
> >>> m = c.myMethod
> >>> m is c.myMethod
> False  <--- What? Why is that?

I believe that c.myMethod is actually a new object that's similar to
lambda self,*args,**kwargs: MyClass.myMethod(self,*args,**kwargs).

ie:

The MyClass *instance* has checked if there is an object
self.__dict__["myMethod"], and when it hasn't found it it creates a new
function which wraps the call to the base class up correctly. It's more
complicated than this though because you see the same behaviour when
checking the method on the class definition.

Seem to remember Raymond Hettinger pointing to the code that does this
as part of his "descriptor tutorial" talk at europython - but I can't
find the slides to reference.

hope that helps,

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: pygame and socket.recv

[Tim Wintle]

On Wed, 2009-04-01 at 17:58 -0700, Aaron Brady wrote:
> I tried writing a small game on a pygame layer.  The graphics are
> fine, and at the moment, it is not graphics intensive.  It is multi-
> player, and for the communication, I am sending a pickle string across
> a LAN, once per frame.
> 
> I'm observing some latency.  It seems that socket.recv isn't
> performing consistently.

Not sure I understand the question, are you blocking for the data to
come down the network before rendering the next frame?

For game programming I've always used select with non-blocking sockets
to receive data - and kept the transmissions to UDP to save time
(obviously you have to expect some data to be lost). Wire time is always
going to have too much latency for a message to be happily passed within
the time it takes to render a frame.

For syncing time I use a really simple algorithm - both machines send
each other their local [game] time every few seconds, and if the
received time is ahead of the local time then the receiving machine
updates it's time to match - that way they are always out by at most the
shortest time it takes for a packet to travel from one to the other.


Tim Wintle


--
http://mail.python.org/mailman/listinfo/python-list

Re: pygame and socket.recv

[Tim Wintle]

On Wed, 2009-04-01 at 18:45 -0700, Aaron Brady wrote:
> 
> My game loop looks like this:
> 
> poll events, get 1 at most
> send to server
> wait for server reply
> render entire frame

The look I'm suggesting is:

poll events
write to (non-blocking) socket
render frame
check non-blocking socket and add events to the event queue

> Yes, I am blocking for the data to come down the network.
> Unfortunately, if I use any "prediction," I will have to go back and
> un-render the previous frame, then redraw with the new information.
Sounds like that may have to be re-factored slightly, afraid this is why
real-time networked games are tough to make.
> 
> 40 transmissions per second in each way can't be too much to ask, it's
> just that they have to alternate, up one, down one.
IMO It's very unlikely to be a bandwidth issue. It's more likely to just
be a latency issue.

> 
> I don't understand your solution.  I can't picture it for my favorite
> RTS game or the one I'm writing.  Are you saying that the slower
> machine just jumps ahead, and its user just doesn't have the
> opportunity to make moves on the omitted frames?


a) How much to move etc. is decided based on some real-time solution
(not on the number of frames). Ideally all movement methods take a
parameter that is a delta in time. (i.e. 1/40th of a second)

b) That time is the time that is synced across machines - if machine B
has to put it's timer forward 1/50 of a second, you call all the methods
above with a timedelta of 1/50 before continuing.

c) Obviously things like which frame a sprite is on isn't really
necessary for the sake of a game, it's only game state variables that
would be required.

d) If you end up with jumping objects then you can create "ghost"
objects for the other player's objects. When you get the state of a
foreign object, update the real object with the properties, but draw the
sprite at the position of your ghost object - and every frame move the
ghost object towards the real one a little bit. That gets rid of the
effect of jumping from one place to another, but keeps collision
detection etc. correct wrt the other player.


> 
> I am using TCP, socket.SOCK_STREAM.  UDP is a potential solution, but
> it still doesn't fix my main loop.
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: python for loop

[Tim Wintle]

On Thu, 2009-04-02 at 06:28 +, Steven D'Aprano wrote:
> In set theory, you start by defining the integers like this:

> 
> 0 = len( {} )
> 1 = len( {{}} )
> 2 = len( {{}, {{}}} )
> 3 = len( {{}, {{}}, {{}, {{}}} )
> etc.
not quite len() - surely you mean something like "any object along with
an algebra in which the left hand side is equivalent to the right in the
algebra of set theory?" - at least for ordinals.

The cardinal is then (for finite numbers) the length of that.


Or, in a pythonic sense (taking 0,1,2,... to be variable names):
 0 = set()
 1 = set(0)
 2 = set(1,0)
 3 = set(2,1,0)
 3 = set(3,2,1,0)
 etc.


> How that relates to whether indexing should start at one or zero, I
> have 
> no idea.

so in this sense, range(n) is actually very close to the ordinal value
of 0 (except for being a list and not a set - but it's not in 3.0)

i.e. range(n) returns something very similar to the ordinal "n", and
with cardinality n. That seems very sensible to me.


> Oh, and speaking of... I'm shocked, SHOCKED I say, that nobody has
> given that quote about the compromise of 0.5.
"God made the integers, all else is the work of man" - Leopold Kronecker

...holding myself back from complaining about integer division in Py3K
when the philosophical question of whether irrational numbers even exist
(in a physics sense) is fairly open.

Tim W


--
http://mail.python.org/mailman/listinfo/python-list

Re: pygame and socket.recv

[Tim Wintle]

On Thu, 2009-04-02 at 06:50 -0700, Aaron Brady wrote:
> It's just that if you register a collision in between the time that
> one object has changed its position and momentum, and the time you
> learn about it, you have to retroactively edit the collision, restore
> hit points, and recalculate the other object's position and momentum,
> to name a few. 

So it sounds like what you are trying to do is emulate shared RAM
between the two players!

The problem being that you have to effectively lock the variables on one
machine while the other one reacts.

I believe the answer normally used is the client-server architecture. 

Choose one machine to act as the "server" somehow, then let that control
the entire game state. Both players pass events to this server method
(using non-blocking sockets), and render things in the position they
believe they are in. 

I believe that is how all the current RTS games work, certainly how Red
Alert worked, and is how all FPS games work (e.g. in unreal there is
always a "server", and in single player mode the clients are simply bots
and the local client application.)

You might find the docs for Unreal networking (c. 1999) useful to
explain what they did better than I can.
http://unreal.epicgames.com/Network.htm


--
http://mail.python.org/mailman/listinfo/python-list

Re: python for loop

[Tim Wintle]

On Thu, 2009-04-02 at 15:16 -0700, Emile van Sebille wrote:
> Lou Pecora wrote:
> > Confusion only comes when you try to force the 
> > defintion of one of them on the other and then say it's illogical or not 
> > natural.  Both are natural.
> 
> Consider the French 'Premiere etage' vs the American 'First Floor'

or even in the same language - "first floor" in English (UK) is very
different from "first floor" in English (US).

> 
> Emile
> 
> 
> 
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: python needs leaning stuff from other language

[Tim Wintle]

On Fri, 2009-04-03 at 13:12 -0400, Mel wrote:
> >>> I think it would also be better to have One (and prefereably Only One)
> >>> Obvious Way To Do It. That obvious way, for those who work with
> >>> Python's ‘set’ and ‘dict’, is a ‘clear’ method. It seems best to have
> >>> ‘list’ conform with this also.
> >> 
> >> Does that mean a one-off special case rule to forbid slices having a
> >> default?
> > 
> > Why would it do that?
> 
> Well, if list.clear were truly and strictly to be the only way to clear the 
> contents of a list, then assigning nothing via the default slice would have 
> to be ruled out.  `somelist[:] = []` is just a special case of assignment to 
> a slice generally.

agreed. If .clear was to be added then really assignments to slices
should be entirely removed. 

Tim W

--
http://mail.python.org/mailman/listinfo/python-list

Re: python needs leaning stuff from other language

[Tim Wintle]

On Fri, 2009-04-03 at 18:27 -0500, Robert Kern wrote:
> > agreed. If .clear was to be added then really assignments to slices
> > should be entirely removed.
> 
> Please tell me you are joking.

Well I'm not joking as such.

I've noticed that python-ideas seems to be positive on the idea, and has
a patch ready for Guido, obviously I'm not that anti it that I'd always
be complaining if it is implemented, I just see it as unnecessary:
http://mail.python.org/pipermail/python-ideas/2009-April/003933.html

(I didn't expect such strong responses btw!)


you can already do:
del mylist[:]
* or *
mylist[:] = []
* or *
mylist = []


which, although semantically similar are different as far as the
interpreter are concerned (since two of them create a new list):

(Python 2.5.2 - don't have a newer version on this machine to check)
{{{
import dis
>>> def f(a):
...   del a[:]
...
>>> def g(a):
...   a[:] = []
...
>>> def h(a):
...   a = []
...
>>> dis.dis(f)
  2   0 LOAD_FAST0 (a)
  3 DELETE_SLICE+0  
  4 LOAD_CONST   0 (None)
  7 RETURN_VALUE
>>> dis.dis(g)
  2   0 BUILD_LIST   0
  3 LOAD_FAST0 (a)
  6 STORE_SLICE+0   
  7 LOAD_CONST   0 (None)
 10 RETURN_VALUE
>>> dis.dis(h)
  2   0 BUILD_LIST   0
  3 STORE_FAST   0 (a)
  6 LOAD_CONST   0 (None)
  9 RETURN_VALUE
}}}

so it seems silly to introduce a *third* formal description of what is
(almost) semantically the same!

My knowledge of the interpreter begins to get fuzzy here, but would this
add extra overhead by the look-ups to the .clear method (really don't
have the time to compile a patched version of trunk to test!) - it might
be a constant change, but it's still a difference to what I believe is
likely to be required to be fairly high-performance when it is used.

e.g. 
{{{
>>> def f(a):
...   a.clear()
... 
>>> dis.dis(f)
  2   0 LOAD_FAST0 (a)
  3 LOAD_ATTR0 (clear)
  6 CALL_FUNCTION0
  9 POP_TOP 
 10 LOAD_CONST   0 (None)
 13 RETURN_VALUE
}}}

The current versions at least do two different things quite efficiently
- if you are clearing the list and expect the list to be small / zero
length next time, then (I believe) it's more memory efficient to try 
mylist = []

,where
del mylist[:] 

will (I'm assuming - correct me if I'm wrong) reduce the size of the
list, but using the clever list memory usage it will scale to a large
size again fairly well.

As for:
"Should we also remove .update from dict?"

Am I missing something? What is the single statement that is equivalent
to .update (although personally I don't think I have *ever* used .update
- I'm normally using a set for situations where I would require .update)


Oh, and can I have the reference for Raymond Hettinger's blessing
(mentioned in a different branch of this thread)? As far as I can see he
has only (in archives at least) replied to a question about whether it's
worth formalising the request as a PEP, and hasn't entered a vote either
way:

"""
Just ask Guido for his blessing.  The implementation is trivial.
"""
http://mail.python.org/pipermail/python-ideas/2009-April/003938.html

Ironically much of my - very limited - knowledge of the interpreter
stems from many of his talks, which were the encouragement to start
actually reading the source for the interpreter!

I'm more than willing to drop my issue if it's voted against, but it
does feel dirty to me to add extra syntax that have completely parallel
semantics. 

In general language design (not just programming languages) I personally
take any (provable) requirement for this as a _possible_ problem with
the chosen basis / semantic value function (in 1st order language
terms), although I understand that it is sometimes unavoidable in real
life.

For example, this makes me question the orthogonality of "ordered" and
"collection" with semantics dictated by their methods/features. 

Proposing that the object:
mylist[a:b:c] 

return an iterator that is non-assignable appears to fix this
non-orthogonality as far as I can immediately see (although I am *very*
tired, and haven't thought it through far enough!)


Tim Wintle



--
http://mail.python.org/mailman/listinfo/python-list

Re: is there a way to collect twitts with python?

[Tim Wintle]

On Fri, 2009-04-03 at 14:58 -0600, Michael Torrie wrote:
> Oh wow.  If this is what Twitter does to one's ability to articulate
> clearly, I hope Twitter dies a horrible death and any APIs and Python
> bindings with it!

Thank you, thank you, thank you 

everyone around me seems to love that thing (twitter), and I still can't
work out why (apart from hacks such as using it as a hosted queue for
cross-server comms, or receiving cheap sms to your app)

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Testing dynamic languages

[Tim Wintle]

On Sat, 2009-04-04 at 06:37 -0700, grkunt...@gmail.com wrote:
> If I am writing in Python, since it is dynamically, but strongly
> typed, I really should check that each parameter is of the expected
> type, or at least can respond to the method I plan on calling ("duck"
> typing). Every call should be wrapped in a try/except statement to
> prevent the method (and program) from crashing when my method is
> called with an integer instead of the expected string.

At some point you should wrap it in a try/except block - but only at the
point where you want the exception to be handled. That will normally be
quite far up, and you'll just let the exception travel back up to that
point.


for example, in a web server type of thing you might have something
vaguely like


def main:
  for request in request_itter:
try:
  headers = request.get_headers()
  dispatch = get_dispatch_fn(headers)
  response = dispatch(response)
except:
  send_500()

You probably then don't need to catch any exceptions in the get_headers,
or actual processing methods (unless you are writing to sql or
something, when you might want to wrap all statements in a try block,
and then put a "ROLLBACK;" query in the except block (and then raise the
caught exception so it goes back the the block above)

> 
> Is this the experience that Python programmer (of large projects) see?
> Do you also write unit tests to confirm that the methods actually
> check for and catch "bad" parameter types? If I am writing small one-
> off scripts, I wouldn't worry about it, but if I am writing a large
> system that must have 99+% uptime without constant monitoring, this
> really should be verified.

I write large applications with a target of 99.9% uptime, and I don't
find it a problem. occasionally I have to check parameters, but that's
not very often

> 
> Up for discussion...
> --
> http://mail.python.org/mailman/listinfo/python-list

--
http://mail.python.org/mailman/listinfo/python-list

Re: python needs leaning stuff from other language

[Tim Wintle]

On Sat, 2009-04-04 at 02:03 -0500, Robert Kern wrote:
> 
> Let's be clear: python-ideas seems positive on the idea of adding a .clear() 
> method. *Completely removing* slice assignment has not been broached there.

Yup, sorry - I did mean to refer to the initial suggestion, rather than
my comments

> 
> > (I didn't expect such strong responses btw!)
> 
> You are proposing the removal of a general, orthogonal feature (and breaking 
> code in consequence!) just because of a new syntax for a single special case 
> of 
> that feature. That is quite simply ridiculous.

Ok, I may have come across a little strongly (was very tired) - I'm not
_actually_ saying we should remove it, I'm just pointing out why
adding .clear() to lists seems to be unnecessary and slightly messy. The
suggested removal of assignments to slices is a theoretical statement.

> 
> .clear() would be non-orthogonal syntactic sugar. That's okay! Python has 
> syntactic sugar in a number of other places, too! Appropriate doses of 
> syntactic 
> sugar and non-orthogonality are precisely what lets you implement "There 
> should 
> be one-- and preferably only one --obvious way to do it." The really key word 
> in 
> that sentence is "obvious", not "one".
> 
> FWIW, removing slice assignment would be a gross form of non-orthogonality, 
> too. 
> __getitem__, __setitem__ and __delitem__ should all be able to accept the 
> same 
> indices (or else raise exceptions in the case of immutability).

hummm - I'm sure it would be confusing behaviour if it was not
available, but I'm not sure how it would be non-orthogonal

--
http://mail.python.org/mailman/listinfo/python-list

Re: python needs leaning stuff from other language

[Tim Wintle]

On Sat, 2009-04-04 at 15:36 -0500, Robert Kern wrote:
> On 2009-04-04 12:07, Tim Wintle wrote:
> >>> (I didn't expect such strong responses btw!)
> >> You are proposing the removal of a general, orthogonal feature (and 
> >> breaking
> >> code in consequence!) just because of a new syntax for a single special 
> >> case of
> >> that feature. That is quite simply ridiculous.
> >
> > Ok, I may have come across a little strongly (was very tired) - I'm not
> > _actually_ saying we should remove it, I'm just pointing out why
> > adding .clear() to lists seems to be unnecessary and slightly messy. The
> > suggested removal of assignments to slices is a theoretical statement.
> 
> But can you see why your wording might lead the rest of us to believe 
> otherwise? 
>   :-)

Yes I do - sorry, I tend to check mailing lists at the end of the day
when I am quite tired.

I think I could argue 'till the cows come home about why adding .clear
feels messy to me, bu as it's not my decision in the slightest, I think
I will end this thread here ;)

Tim

--
http://mail.python.org/mailman/listinfo/python-list

Re: binary file compare...

[Tim Wintle]

On Thu, 2009-04-16 at 21:44 -0700, Adam Olsen wrote:
> The Wayback Machine has 150 billion pages, so 2**37.  Google's index
> is a bit larger at over a trillion pages, so 2**40.  A little closer
> than I'd like, but that's still 56294995000 to 1 odds of having
> *any* collisions between *any* of the files.  Step up to SHA-256 and
> it becomes 1915619400 to
> 1.  Sadly, I can't even give you the odds for SHA-512, Qalculate
> considers that too close to infinite to display. :)

That might be true as long as your data is completely uniformly
distributed. For the example you give there's:

a) a high chance that there's "" near the top

b) a non-uniform distribution of individual words within the text.

c) a non-unifom distribution of all n-grams within the text (as there is
in natural language)

So it's very far from uniformly distributed. Just about the only
situation where I could imagine that holding would be where you are
hashing uniformly random data for the sake of testing the hash.


I believe the point being made is that comparing hash values is a
probabilistic algorithm anyway, which is fine if you're ok with that,
but for mission critical software it's crazy.

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Too early implementation

[Tim Wintle]

On Sat, 2009-04-18 at 13:48 +0200, Filip Gruszczyński wrote:
> 
> So, do you know some good methods to prevent myself from just starting
> coding (which I like very much) and do some thinking about the problem
> (which I like a little less ;-))?

The "Method" (If you can call it that) that I use is to have users throw
requests at me so fast that they normally have to get backlogged for
months at a time (I've got several years worth of backlogged work at the
moment). 

That way by the time I actually start coding I've normally read over the
issue several times and approached the problem from various angles ;-)

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: [OT] large db question about no joins

[Tim Wintle]

On Fri, 2009-04-17 at 21:16 +0100, Martin P. Hellwig wrote:
> So in other words, just lay out the data which makes the most sense
> to 
> you, the pain of recreating SQL like logic is there no matter what 
> layout you choose.

I have to say that given the amount of pain most people seem to go
through when they first learn about relational databases, it seems
ironic how much people come to expect that that's the standard way to
query data!

At the end of the day it's all just one long sequential load of 0s and
1s on disk, and some code somewhere has to do all the logic.

(I'm a relational database user btw, but it seems to happen with object
database people too - we get so used to our own paradigm that we don't
think about other ways of doing the task)

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a programming language that is combination of Python and Basic?

[Tim Wintle]

On Sun, 2009-04-19 at 05:08 +, Zaphod wrote:
> Well, most of the Linux kernel is written in C and while there *is* a 
> jump (often JMP) in most asms, you should only do so if you really
> need 
> to.  JSR (jump sub routine) is a better idea in many (most?) cases. 

Have to say that I feel jump is more than justified in some situations
(when it's jumping to within 10-20 lines of the start position, and it's
a routine that needs to be highly optimised - I'm thinking tail
recursion etc.)

(btw, how come nobody has mentioned python bytecode? Most flow control
is jumps)

Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a programming language that is combination of Python and Basic?

[Tim Wintle]

On Sun, 2009-04-19 at 06:26 +, Steven D'Aprano wrote:
> > (btw, how come nobody has mentioned python bytecode? Most flow
> control is jumps)
> 
> 
> I wrote yesterday:
> 
> "GOTO, after all, is just a jump, and we use jumps in Python all the
> time:
> 
> raise Exception
> break
> continue
> if... elif... else...
> for... else...
> etc."

Ah - apologies 

Tim

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python interpreter speed

[Tim Wintle]

On Sun, 2009-04-19 at 18:11 +0200, Ryniek90 wrote:
> Hi.
> 
> Standard Python interpreter's implementation is written in C language. C 
> code while compilation, is compilled into machine code (the fastest 
> code). Python code is compiled into into byte-code which is also some 
> sort of fast machine code. So why Python interpreter is slower than Java 
> VM? Being written in C and compilled into machine code, it should be as 
> fast as C/Asm code.
> What's wrong with that?


I can't remember Java properly, but...

Firstly, speed will depend on what you're writing. I dont' actually know
how much slower python is, but I'm sure there are some things that run
faster in python.

for example, the program:

#!/usr/bin/python
print "Hello World"

is almost certainly faster in python than Java, as the JVM takes much
longer to start up.


But basically for the actual code execution speed, it's the Virtual
machine:

Dynamic typing ...
is great for productivity, and a great language feature, but it does
mean that most operations in the virtual machine have to be wrapped in
some kind of type-checking. That adds more and more operations to the
work that has to be done to run python code.

Just in Time ...
is used in the Java VM but not in the standard python VM. It can
significantly optimise performance though.

There are various people working on adding a JIT to python - I'm most
interested in pypy, here's a basic article explaining their current
target of adding a tracing JIT ( I believe it's working on a small
subset of python):
http://morepypy.blogspot.com/2009/03/applying-tracing-jit-to-interpreter.html


(Just spotted that there was a new pypy release yesterday if people
haven't noticed)


You'll also probably notice that many of python's types (list, dict,
etc.) might be slower for small sizes than Java's are - that's because
they have been optimised to perform efficiently at any size, at the
expense of being slightly less efficient than they could be for small
sizes)


Tim Wintle

--
http://mail.python.org/mailman/listinfo/python-list

Re: The Python standard library and PEP8

[Tim Wintle]

On Sun, 2009-04-19 at 18:43 +0200, Emmanuel Surleau wrote:
> Hi there,
> 
> Exploring the Python standard library, I was surprised to see that several 
> packages (ConfigParser, logging...) use mixed case for methods all over the 
> place. I assume that they were written back when the Python styling 
> guidelines were not well-defined.

I seem to remember there was talk about this on python-dev fairly
recently, and basically they wish they had changed more of them in
python 3, but as they didn't they are likely to stay.

> 
> Given that it's rather irritating (not to mention violating the principle of 
> least surprise) to have this inconsistency, wouldn't it make sense to clean 
> up the API by marking old-style, mixed-case methods as deprecated (but 
> keep them around anyway) and add equivalent methods following the 
> lowercase_with_underscores convention?

There was some suggestion about introducing synonyms that followed PEP8
and introducing deprecation warnings as you suggested, but I can't
remember the outcome. I'd suggest checking the dev archives.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Self function

[Tim Wintle]

On Mon, 2009-05-04 at 19:51 +0100, Arnaud Delobelle wrote:
> 
> Bearophile, there is a thread on python-ideas about tail-call
> optimization at the moment.

Oooh - haven't noticed that (and don't have time to follow it), but has
anyone seen the results I got a week or so ago from briefly playing with
a couple of simple optimisations:



I was amazed how much you could improve performance by not jumping all
over the stack




signature.asc
Description: This is a digitally signed message part
--
http://mail.python.org/mailman/listinfo/python-list

Re: While Statement

[Tim Wintle]

On Fri, 2009-05-22 at 13:19 +0200, Andre Engels wrote:
> number/total = 998/999 = 0
> number/total*100 = 0*100 = 0
> float(number/total*100) = float(0) = 0.0
> 
> Change "float(number/total*100)" to "float(number)/total*100" and it
> should work:

I'd use:

 (number * 100.)/total

- works because
  *  =>  

It's a minor thing, but it's much faster to cast implicitly as you miss
the python function call overhead - it's no extra work to write, and for
numerical things it can really speed things up.

>>> a = timeit.Timer("float(200)/5*100")
>>> b = timeit.Timer("(200*100.)/5")
>>> a.timeit(1000)
12.282480955123901
>>> b.timeit(1000)
3.6434230804443359

Tim W


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python question

[Tim Wintle]

On Fri, 2009-05-22 at 05:51 -0700, Craig wrote:
> I use python 2.6.2 and i useing ubuntu 9.04 not windows.

What are you trying to install?

is it available in Synaptic package manager?


If it's a program written in python, then there may be a file called
"setup.py". If there is then open a terminal, cd to that directory, and
then type:

python setup.py install

(you may need to do "sudo" before it) - that will normally automatically
compile any extensions written in C

hope that helps.


Tim W

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: While Statement

[Tim Wintle]

On Fri, 2009-05-22 at 09:59 -0400, Dave Angel wrote:
> 
> Tim Wintle wrote:
> > On Fri, 2009-05-22 at 13:19 +0200, Andre Engels wrote:

> >> Change "float(number/total*100)" to "float(number)/total*100" and it
> >> should work:
> >> 
> >
> > I'd use:
> >
> >  (number * 100.)/total
> >
> > - works because
> >   *  =>  

> It's the old-timer in me, but I'd avoid the float entirely.  You start 
> with ints, and you want to end with ints.  So simply do the multiply 
> first, then the divide.
> 
>  number * 100/total

Agreed, to be honest I'd not fully read the original post and didn't
realise everything was an int.




> 
> will get the same answer.
> 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: why do I get this behavior from a while loop?

[Tim Wintle]

On Fri, 2009-11-27 at 17:06 +0100, S. Chris Colbert wrote:
> This seems strange to me, but perhaps I am just missing something:

> I would think that second loop should terminate at 9.9, no? 
> 
> I am missing something fundamental?

Floating points variables ...
http://en.wikipedia.org/wiki/Floating_point

0.1 isn't represented internally, so you haven't actually reached 10. -
try comparing that "10." (as it's printed) with 10.0 i.e.
 print (t == 10.)

you'll get False - even though they print the same

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Recursion head scratcher

[Tim Wintle]

On Wed, 2009-12-02 at 02:07 -0500, Joel Madigan wrote:
> 
> that it is possible to make it print the path to the finish in the
> order the steps were taken.  That is, the algorithm as written
> produces: (4,0) (4,1) (3,1) (3,2) (3,3) (2,3) (1,3) (1,2) True 
> 
> Rather than (1,2) (1,3) (2,3) (3,3) (3,2) (3,1) (4,1) (4,0) True 
> 
> Furthermore, he claims it's a "one line change" without using a stack
> or any other extra data structure

The way I see immediately is a three line change (if you include
modifying/removing the existing print statements). It will be one line
shorter to print the other order.

As a hint - think of what the python interpreter's stack looks like when
it's running your code at the moment - that's the stack you're currently
using (and need to use) to store the results you print.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Java-to-Python?

[Tim Wintle]

On Fri, 2009-12-18 at 15:44 +0100, Virgil Stokes wrote:
> I have a rather large Java package for the analysis of networks that I 
> would like to convert to Python. Many of the classes in the Java package 
> are "Serializable".
> 
> Any recommendations on Java-to-Python (2.6) would be appreciated.

I used java2python recently with quite a lot of success. I believe it
doesn't support newer java features though.

http://code.google.com/p/java2python/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A way to convert RTF to HTML?

[Tim Wintle]

On Sun, 2009-12-27 at 20:10 +, Star Glider wrote:
> the problem is that the one of the fields as text in rich text format,
> and it needs to be display without the RTF markup, of course.
> Is there any way to convert RTF to HTML?


Depending on how precisely you need to lay it out you might find it
relatively easy to do yourself (The spec is available online).

I wrote a very simple parser recently as part of a "graffle2svg" - it
ignores most styling options, but applies some of the styles in css.

http://code.google.com/p/graffle2svg/

- feel free to take rtf.py from that project under the BSD license if
it's helpful.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Pymazon 0.1beta released.

[Tim Wintle]

On Tue, 2009-12-29 at 19:44 +0100, Chris Colbert wrote:
> I'm happy to announce the first beta release of Pymazon: a Python
> implemented alternative to the Amazon mp3 downloader. 
> 
> Pymazon was created specifically to alleviate the issues surrounding
> the Linux version of the Amazon mp3 downloader (though it should run
> just fine in Windows too). 

Thanks!

I've been complaining to them for not providing 64-bit binaries (or
source) for ages! (Although hats off to them for even providing the
number of linux variants they do support)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: SimpleXMLRPCServer daemon

[Tim Wintle]

On Fri, 2010-01-29 at 07:54 -0800, Thomas Allen wrote:
> The second is that when it does crash, I don't know about it...what
> would be sufficient as a "keep-alive" script to restart it? I suppose
> I could use something like EventMachine (already installed on my
> server) to watch the PID file if it were deleted reliably.

If the server crashes then it clearly won't get around to deleting it's
pid file.

The way I do it is to use os.kill (with signal 0 IIRC) to check if the
process is still alive when the script starts. If it's not then I delete
the pid file and carry on starting up.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Skeletal animation

[Tim Wintle]

On Mon, 2009-10-05 at 18:36 +0200, Donn wrote:
> see: http://www.panda3d.org/wiki/index.php/Attaching_an_Object_to_a_Joint

+1 for Panda 3d.

Their Node graph is very intuitive for animating joints, and Panda's API
is very nice and clean.

TimW

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Object Relational Mappers are evil (a meditation)

[Tim Wintle]

On Fri, 2009-10-16 at 01:01 +0200, Mick Krippendorf wrote:
> Maybe my English (and my memory) is just not so good. I'm german, and
> here "abnormal" and "anormal" are both negations of "normal", but with
> a slight difference in meaning. "anormal" means just "not normal",
> whereas the meaning of "abnormal" is more like "perverted". That's of
> course the better word for the case at hand.

>From my understanding, the prefix "ab-" comes from latin (away /away
from) - which can also be shortened to "a" in some usages. (e.g.
abnormal)

The prefix "an" - which is more commonly shortened to "a" comes from the
greek - meaning "without" (e.g. anaerobic )

I agree that "abnormal" would be the better term here - as it does /can
have normalisation information, just not in a standardised manner.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: pointless musings on performance

[Tim Wintle]

On Tue, 2009-11-24 at 18:25 +, Antoine Pitrou wrote:
> Le Tue, 24 Nov 2009 08:58:40 -0800, Paul Boddie a écrit :
> > As you
> > point out, a lot of this RISC vs. CISC analysis (and inferences
> drawn
> > from Python bytecode analysis) is somewhat academic: the cost of the
> > JUMP_IF_FALSE instruction is likely to be minimal in the context of
> all the activity going on to evaluate the bytecodes.
> 
> Sorry, I have trouble parsing your sentence. Do you mean bytecode 
> interpretation overhead is minimal compared to the cost of actual
> useful work, or the contrary?
> (IMO both are wrong by the way)

Out of interest - has anyone else spotted that the call to
PyObject_IsTrue in the XXX_JUMP_IF_ blocks performs two unnecessary
pointer comparisons?

 ceval.c 
if (w == Py_True) {
Py_DECREF(w);
FAST_DISPATCH();
}
if (w == Py_False) {
Py_DECREF(w);
JUMPTO(oparg);
FAST_DISPATCH();
}
err = PyObject_IsTrue(w);
Py_DECREF(w);
.
.
.
==

 object.c 
PyObject_IsTrue(PyObject *v)
{
Py_ssize_t res;
if (v == Py_True)
return 1;
if (v == Py_False)
return 0;
.
.
.
==

Would it be worth in-lining the remaining part of PyObject_IsTrue in
ceval?

> Another data point I've heard is that people who have tried a very
> crude form of Python-to-C compilation (generating the exact C code
> corresponding to a function or method, using Python's C API and
> preserving dynamicity without attempting to be clever) have apparently
> reached speedups of up to 50% (in other words, "twice as fast").

That's roughly what I get with Cython - which does exactly that.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: taking python enterprise level?...

[Tim Wintle]

On Thu, 2010-02-25 at 02:26 -0800, simn_stv wrote:
> i plan to build an application, a network based application that i
> estimate (and seriously hope) would get as many as 100, 000 hits a day
> (hehe,...my dad always told me to 'AIM HIGH' ;0), not some 'facebook'
> or anything like it, its mainly for a financial transactions which
> gets pretty busy...

I've got apps running that handle *well* over 100,000 hits / process /
day using Python - although some of the heavy lifting is off-loaded to C
and MySql - obviously without actually looking at your requirements that
doesn't mean much as I don't know how much work each hit requires.

Regarding financial transactions - you'll almost certainly want to
integrate with something that already has transactional support (sql
etc) - so I expect that will bear the brunt of the load

> so my question is this would anyone have anything that would make
> python a little less of a serious candidate (cos it already is) and
> the options may be to use some other languages (maybe java, C (oh
> God))

I've avoided integrating java with my python (I'm not a big fan of java)
- but I've integrated quite a bit of C - it's fairly easy to do, and you
can just port the inner loops if you see the need arise.

> ...i am into a bit of php and building API's in php would not be
> the hard part, what i am concerned about is scalability and
> efficiency, well, as far as the 'core' is concerned.

I've heard that php can be well scaled (by compiling it to bytecode/C++)
- but my preference would always be to python.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: taking python enterprise level?...

[Tim Wintle]

On Wed, 2010-03-03 at 17:26 +0100, mk wrote:
> 
> So there *may* be some evidence that joins are indeed bad in
> practice. 
> If someone has smth specific/interesting on the subject, please post.

I have found joins to cause problems in a few cases - I'm talking about
relatively large tables though - roughly order 10^8 rows.

I'm on Mysql normally, but that shouldn't make any difference - I've
seen almost the same situation on Oracle

consider this simple example:

/*   Table A*/
CREATE TABLE TableA (
project_id BIGINT NOT NULL,
cost INT,
date DATETIME,
PRIMARY KEY (project_id, date)
);

/* Table projects */
CREATE TABLE projects (
client_id BIGINT NOT NULL,
project_id BIGINT NOT NULL,
INDEX(client_id)
);


... now the index on TableA has been optimised for queries against date
ranges on specific project ids which should more or less be sequential
(under a load of other assumptions) - but that reduces the efficiency of
the query under a join with the table "projects".

If you denormalise the table, and update the first index to be on
(client_id, project_id, date) it can end up running far more quickly -
assuming you can access the first mapping anyway - so you're still
storing the first table, with stored procedures to ensure you still have
correct data in all tables.

I'm definitely glossing over the details - but I've definitely got
situations where I've had to choose denormalisation over purity of data.


Rolled-up data tables are other situations - where you know half your
queries are grouping by field "A" it's sometimes a requirement to store
that.


Tim



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: taking python enterprise level?...

[Tim Wintle]

On Wed, 2010-03-03 at 20:39 +0100, mk wrote:
> Hello Tim,
> 
> Pardon the questions but I haven't had the need to use denormalization 
> yet, so:

> IOW you basically merged the tables like follows?
> 
> CREATE TABLE projects (
>  client_id BIGINT NOT NULL,
>  project_id BIGINT NOT NULL,
>  cost INT,
>  date DATETIME,
>  INDEX(client_id, project_id, date)
> );

Yup

> From what you write further in the mail I conclude that you have not 
> eliminated the first table, just made table projects look like I wrote 
> above, right? (and used stored procedures to make sure that both tables 
> contain the relevant data for client_id and project_id columns in both 
> tables)

Yup

> Have you had some other joins on denormalized keys? i.e. in example how 
> the join of hypothetical TableB with projects on projects.client_id 
> behave with such big tables? (bc I assume that you obviously can't 
> denormalize absolutely everything, so this implies the need of doing 
> some joins on denormalized columns like client_id).

For these joins (for SELECT statements) this _can_ end up running faster
- of course all of this depends on what kind of queries you normally end
up getting and the distribution of data in the indexes.

I've never written anything that started out with a schema like this,
but several have ended up getting denormalised as the projects have
matured and query behaviour has been tested

> > assuming you can access the first mapping anyway -
> 
> ? I'm not clear on what you mean here.

I'm refering to not eliminating the first table as you concluded

> 
> Regards,
> mk
> 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: taking python enterprise level?...

[Tim Wintle]

On Wed, 2010-03-03 at 16:23 -0500, D'Arcy J.M. Cain wrote:
> On Wed, 03 Mar 2010 20:39:35 +0100
> mk  wrote:
> > > If you denormalise the table, and update the first index to be on
> > > (client_id, project_id, date) it can end up running far more quickly -
> 
> Maybe.  Don't start with denormalization.  Write it properly and only
> consider changing if profiling suggests that that is your bottleneck.

Quite - and I'd add to cache reads as much in front end machines as is
permissible in your use case before considering denormalisation.

> With a decent database engine and proper design it will hardly ever be.

I completely agree - I'm simply responding to the request for an example
where denormalisation may be a good idea.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Evaluate my first python script, please

[Tim Wintle]

On Thu, 2010-03-04 at 10:39 -0800, Pete Emerson wrote:
> I am looking for advice along the lines of "an easier way to do this"
> or "a more python way" (I'm sure that's asking for trouble!) or
> "people commonly do this instead" or "here's a slick trick" or "oh,
> interesting, here's my version to do the same thing".

(1) I would wrap it all in a function

def main():
# your code here

if __name__ == "__main__":
main()

(2) PEP8 (python style guidelines) suggests one import per line

(3) I'd use four spaces as tab width

(4) 
I'd change this:

> for arg in sys.argv[1:]:
> for section in hostname.split('.'):
> if section == arg:
> count = count + 1
> break

to something more like:

for section in hostname.split("."):
if section in sys.argv[1:]:
count += 1

(although as you suggested I'd only calculate sys.argv[1:] once)

... or you could replace whole section between the for loop and
hosts.append with:

if sorted(hostname.split(".")) == sorted(sys.argv[1:]):
host.append(hostname)


, at a slight loss of clarity - but I think I'd stick with the more
verbose version personally.

Tim



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Evaluate my first python script, please

[Tim Wintle]

On Fri, 2010-03-05 at 07:53 -0800, Pete Emerson wrote:
> Thanks for your response, further questions inline.
> 
> On Mar 4, 11:07 am, Tim Wintle  wrote:
> > On Thu, 2010-03-04 at 10:39 -0800, Pete Emerson wrote:
> > > I am looking for advice along the lines of "an easier way to do this"
> > > or "a more python way" (I'm sure that's asking for trouble!) or
> > > "people commonly do this instead" or "here's a slick trick" or "oh,
> > > interesting, here's my version to do the same thing".
> >
> > (1) I would wrap it all in a function
> >
> > def main():
> > # your code here
> >
> > if __name__ == "__main__":
> > main()
> 
> Is this purely aesthetic reasons, or will I appreciate this when I
> write my own modules, or something else?

It's for when you reuse this code. 

Consider it's in "mymodule.py" (so run with ./mymodule.py) - if you then
make a "tests.py" (for example) you can "import mymodule" without it
automatically running your code.

re-writing it

def main(args):
#your code

if __name__ == "__main__":
main(sys.argv[1:])

would obviously be more sensible for actually writing tests.


> > ... or you could replace whole section between the for loop and
> > hosts.append with:
> >
> > if sorted(hostname.split(".")) == sorted(sys.argv[1:]):
> > host.append(hostname)
> 
> This doesn't actually work, because I'm not looking for a one to one
> mapping of args to sections of hostname, but rather a subset. So
> passing in 'prod sfo' would register a match for '001.webapp.prod.sfo'.

Ah - good point - I guess the the set intersection technique someone
else mentioned is best in that case.

Tim

-- 
http://mail.python.org/mailman/listinfo/python-list