Re: Python 3.0 unfit for serious work?

[Jay Tee]

Hey,

> Python 3.x.  I believe your fear is just a knee jerk reaction to the notion
> that there will be some stated incompatibilities between 2.x and 3.x without
> having done any investigation of the transition process.  Nobody is forcing
> you to do anything right now or completely abandon your code base.  Python
> 2.x still has a long shelf life.  Hell, 3.0a1 isn't even out yet.  If you

Thanks to the pointer to PyCon, if there is anything relevant maybe
you can send me off a mail (or post here) with some links.  About the
above : it isn't fear, I'm just telling you what I suspect might
happen.  My own little piece of code in the giant picture will remain
in python, unless i am mandated to move it to something else : I
*like* python.

I threw in my few cents here just because I suspect the python
community does not intend to move itself into the realm of
"interesting but not serious languages" like Oberon did.  I could be
wrong, both in the sense that maybe that *is* the intention, or maybe
making a backwards-incompatible "evolution" of the language won't hurt
python or it's users.  We now return to your regularly scheduled
program.

  J "back to the grind" T

-- 
http://mail.python.org/mailman/listinfo/python-list

How to test if one dict is subset of another?

[Jay Tee]

Hi,

I have some code that does, essentially, the following:

- gather information on tens of thousands of items (in this case, jobs
running on a
 compute cluster)
- store the information as a list (one per job) of Job items
(essentially wrapped
 dictionaries mapping attribute names to values)

and then does some computations on the data.  One of the things the
code needs to do, very often, is troll through the list and find jobs
of a certain class:

for j in jobs:
   if (j.get('user') == 'jeff' and j.get('state')=='running') :
  do_something()

This operation is ultimately the limiting factor in the performance.
What I would like to try, if it is possible, is instead do something
like this:

   if j.subset_attr({'user' : 'jeff', 'state' : 'running'}) :
  do_something()


where subset_attr would see if the dict passed in was a subset of the
underlying attribute dict of j:

  j1's dict : { 'user' : 'jeff', 'start' : 43, 'queue' : 'qlong',
'state' : 'running' }
  j2's dict : { 'user' : 'jeff', 'start' : 57, 'queue' : 'qlong',
'state' : 'queued' }

so in the second snippet, if j was j1 then subset_attr would return
true, for j2 the answer would be false (because of the 'state' value
not being the same).

Any suggestions?  Constraint : the answer has to work for both python
2.2 and 2.3 (and preferably all pythons after that).

 JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to test if one dict is subset of another?

[Jay Tee]

On Feb 19, 11:07 am, Peter Otten <[EMAIL PROTECTED]> wrote:

> Use a RDBMS (a database), they tend to be good at this kind of operations.

yeah, one of the options is metakit ... sqlite and buzhug both looked
promising but the constraint of pythons 2.2 and 2.3 ruled that out.
disadvantage of metakit is that it's not pure python, meaning possible
integration problems.  the system has to be deployed at 200+ sites
worldwide on a mix of RHEL 3 and 4 based systems, with some Debian
clusters thrown in, and running real production ...

hence my desire to find a pure-python solution if at all possible.
it's looking grim.

 JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Python 3.0 unfit for serious work?

[Jay Tee]

Yo,

On Feb 16, 6:07 am, Steven Bethard <[EMAIL PROTECTED]> wrote:
> Python 3.0 is determined not to be hampered by backwards incompatibility
> concerns. It's not even clear yet that your average 2.6 code will work

Then Python is pretty much determined to remove itself from
consideration
from various kinds of international projects like the one I work on.
We're already catching flack from people due to a few things that were
valid
in 2.2 that are not valid in 2.3 (I don't have the details but could
scare them
up).  The project we work on contains code from many different people
and has to
run on thousands of computers all over the world.  The installed base
at the
moment is a mix of RHEL 3, RHEL 4, and Debian, with a few other
machines thrown in.
The relevant Python versions at this moment IIRC are 2.2.3 and 2.3.4,
because these
are the native versions on those platforms.

We are estimating, due to the speed at which our applications follow
OS releases, that
we can drop RHEL 3 (and hence Python 2.2) support a year from now.  Go
figure when you
think we might be ready to require that all programs run on python
3.0.  If it's not
backwards compatible, meaning if 2.4 code doesn't run on 3.0, it's
rather likely that
strong pressure will be applied to port *away* from Python into
something less capricious.

Bottom line: practicality and beauty is always a tradeoff.  Oberon is
the most beautiful
language I ever saw, but there is almost nobody using it any more.
Too many beauty contests over who had the best proposal for a standard
library.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to test if one dict is subset of another?

[Jay Tee]

On Feb 20, 9:12 am, Paul Rubin  wrote:
 >   do_something()
>
> you'd just write:
>
> for j in (index['user']['jeff'] & index['state']['running']):
> do_something()

Hi,

it looks very cool, except that one of the constraints mentioned is
that the solution has to work properly on pythons 2.2 and 2.3.
production systems, certified OS releases, that sort of stuff ...
can't just upgrade a few thousand machines worldwide for my little
program ;-)

Thanks ... I took the previous poster's suggestion (had also been
suggested earlier) and built cached versions of the job lists, indexed
on common queries, while building the master job list.

JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to test if one dict is subset of another?

[Jay Tee]

Hi

your post had the following construct:

> for j in (index['user']['jeff'] & index['state']['running']):
> do_something()

but

Python 2.3.4 (#1, Oct 11 2006, 06:18:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> l1= [3, 4, 7, 2]
>>> l2 = [2, 3]
>>> l2 = [2, 3, 99]
>>> l1 & l2
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: unsupported operand type(s) for &: 'list' and 'list'

what am I missing?

Thanks

JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to test if one dict is subset of another?

[Jay Tee]

On Feb 20, 6:44 pm, Paul Rubin  wrote:
> They are sets, not lists.
>
>from sets import Set as set  # use in 2.3 and earlier
>
>l1= set([3, 4, 7, 2])
>l2 = set([2, 3])
>l2 = set([2, 3, 99])
>print l1 & l2

Thanks Paul, but:

bosui:~> python
Python 2.2.3 (#1, Oct 26 2003, 11:49:53)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sets import Set as set
Traceback (most recent call last):
  File "", line 1, in ?
ImportError: No module named sets

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3.0 unfit for serious work?

[Jay Tee]

Hi,

On Feb 20, 8:59 pm, Steven Bethard <[EMAIL PROTECTED]> wrote:
> You snipped the rest of that comment:
>
> "It's not even clear yet that your average 2.6 code will work on 3.0 --
> though there's a pretty large contingent trying to make this true."

Thanks for pointing this out.  I voted for the comp.lang.python
newsgroup back in the early 90's, my active days of python
'development' are over, it's all i can do to hang on to the code i've
been posting about.

> have 2.6/3.0 compatible code) by late 2011. Sure, it's a long way off,
> but you're writing 2.2 compatible code *now*. Is it really that bad to
> wait four years for Python 3.0?

As long as when python 3.0 shows up, i don't have to do a massive
rewrite.  I think I've really only had to change two or three things
over the years .. one was that I used to use words like "dict" and
"list" in my code, which broke in subtle ways when d = dict() became
legal.

I just dug out some code laying around on disk from 1994, and ran it,
unchanged, under python 2.3.5. If I can achieve this (running 2007
code under python3.0 in 2011 with no modifications), that'd be OK.

JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How to test if one dict is subset of another?

[Jay Tee]

Hi,

thanks!  the code lift from 2.3 to 2.2 worked (thank Guido et al for
BACKWARDS COMPATIBILITY ;-)) ... unfortunately I was in a hurry to get
the release out since a colleague's cluster was croaking under the
load of the old, non-indexed version.  Your solution is nicer looking
than mine, and leads to a less complex implementation.  Rolling it
into the next release is going to have to wait a few weeks, I hope I
can remember it that long.  Thanks!!

JT

On Feb 20, 8:54 pm, Paul Rubin <http://[EMAIL PROTECTED]> wrote:
> "Jay Tee" <[EMAIL PROTECTED]> writes:
> > Python 2.2.3 (#1, Oct 26 2003, 11:49:53)
> > ImportError: No module named sets
>
> Hmm, well I think the sets module in 2.3 is written in Python, so you
> could drop it into your application for use in 2.2.  Better would be
> to use the C version from 2.4 if you can.  Or you could fake it with
> dicts.  Sets are really just dicts under the skin.  Instead of
> set([1,2,3]) you'd use {1:True, 2:True, 3:True}.  To intersect
> two of them (untested):
>
> def intersection(d1,d2):
>if len(d2) < len(d1):
>  # swap them so d1 is the smaller one
>  d1,d2 = d2,d1
>r = {}
>for k in d1.iterkeys():
>  if k in d2:
>r[k] = True
>return r
>
> The d1,d2 swap is just an optimization, since access is constant-time
> you want to scan the smaller dictionary to minimize the number of
> lookups.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3.0 unfit for serious work?

[Jay Tee]

Hi,

Paul, thanks for this, I didn't realize the scope of the situation.  I
agree with your assessment to the extent that I understand what the
whole python 3.0 thing is about.

Let's see if I can scare up something I wrote about ten years ago on a
now-dead language that I really wanted to use (wound up sticking with
python instead because "it was supported" ;-)

===
to figure out how to work things.  The fact that there are three (or
four depending if you count Linz V4) different Oberon System
implementations, and several different compilers, and even four or
five separate dialects of Oberon with none of them appearing to be
really "official", gives the impression of a fragmented, directionless
development effort, and a probability bordering on 1. that
whatever you try to do will be incompatible with all but a small
subset of what's available (unless you stick to writing small programs
like in the books.)  It does not matter if you tell people that this
is not so; something has to clearly stand out as being THE STANDARD
STUFF and all the other stuff as INTERESTING BUT NONTHREATENING SIDE
PROJECTS.  The STANDARD STUFF must include a sufficient number of
=

Oberon is really really cool, seriously, ... but nobody is using it.
People working on python development are of course free to do what
they want, but so are the users ...

   J "actie-reactie is what my ex-brother-in-law would say" T

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3.0 unfit for serious work?

[Jay Tee]

On Feb 21, 1:41 am, "BJörn Lindqvist" <[EMAIL PROTECTED]> wrote:

[ citing me ]

> "if 2.4 code doesn't run on 3.0, it's rather likely that strong
> pressure will be applied to port *away* from Python into something
> less capricious."
>
> Who are these people that are applying the strong pressure? How can
> you run a international and seemingly very important project without
> knowing basic things about how to handle versioning problems?

This isn't versioning.  At least not the way I see it.  Versioning has
to do with changes in your own product.  Indeed one needs to know how
to handle it.  Lately our project has been doing OK on this front, a
couple years ago was a different story.

Underlying technology is a different story.  This should be reasonably
stable.  Small changes are inevitable but even these are a major pain,
since we have dependency links like the following:

  OS release links to
  Python release which changes an
  Third-party extension module which requires
  A third party library whose
  Version needs to be upgraded but which
  Is used by some other non-python means that
  Requires the earlier version.

Or the earlier problem, an OS upgrade comes with a new python version
on which existing code breaks.

The fraction of code in our system that's written in python is
something like 10% -- it's not a big pure-python system.

Now you're talking about major changes in the underlying technology,
forcing, at some point in the future, an extensive rewrite of the
python code.  Yes, at that point, some people would make the comment
that a language which changes to that extent by major versions is not
fit for production work, and if one was going to have to rewrite, it
would be better to rewrite in a more stable language.

And finally, remember the original post I replied to said that python
3.0 was determined not to be hampered by backwards compatibility.
This is quite a bit different than what you say here ("MIGHT cause
problems"):

> language than to continue using Python 2.x because Python 3.x, when it
> is released, MIGHT cause them some problems several YEARS from now?

If backwards compatibility is not a consideration, then it would be a
miracle if there were no problems.

JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 3.0 unfit for serious work?

[Jay Tee]

On Feb 21, 8:06 pm, [EMAIL PROTECTED] wrote:

> I don't know the specifics of your app, but why does everyone insist
> that they need to use the 'system' python?

Hey Grant, don't worry it's not a rant.  A completely valid question.
Again it's a problem of dependency management ... one of the things
we've had to deal with is one of the applications : they have a
framework largely built in python, and I think they were indeed at one
point shipping their own python because they used features not present
in the "system" python.

However, they did use the middleware installed on the system, and this
middleware is also partially written in python ... except it was
expecting the system python.  So you get into very tricky things
having to sanity check the PYTHONPATH, that the "user" python path is
not getting exported into middleware land, nor is the application's
desired PYTHONPATH being destroyed by middleware.  We had just such an
issue here last week, we had solved the problem for bourne shell
users, but csh for some reason had a different behavior, and the user
app could not longer find its python modules.

About the only code for which we don't seem to have these issues is
compiled C.  C++ is nasty because the compilers are still catching up
to the standard; the last I heard, a big segment of C++ code was stuck
because it was full of apparently illegal constructs that the previous
compiler (the g++ shipped stock with RHEL3) accepted, but the RHEL-4
native version refused to compile.

A colleague of mine pointed out today that this is an old problem, and
used to be handled by automake at compile time, with code full of
ifdefs ... now we've moved the problem to run time.  Which I think is
harder, because sometimes we don't find the problem until our binary
is halfway across the world ...

JT

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pep 3105: the end of print?

[Jay Tee]

On Feb 23, 8:48 am, I V <[EMAIL PROTECTED]> wrote:

> While that's true, C++ compiler vendors, for example, take backwards
> compatibility significantly less seriously, it seems to me. A year or so
> ago, I tried compiling something I'd written for g++ 2, using a
> then-recent-ish g++ 3; it failed spectacularly. Likewise with Visual C++ 6
> and a Visual C++ 2005. The suggestion that "working programmers"
> will reject python if a major version change introduces some backwards
> incompatibilities is not borne out by the experience of any other
> language I am aware of.

The experience with C++ in our project is similar to yours.  I think
the real reason is that the compiler tries to be more true to the
standard than to past implementations; past implementations accepted
incorrect syntax, newer "improved" compilers broke backwards
compatibility by rejecting the incorrect syntax.

Can't blame 'em, C++ syntax is hard ... don't try this at home,
kids ...

On the other hand, C++ is firmly established as a "serious" language
in our community while python is not.  So the programmers tend to be
more forgiving.  There is no perception that compiler developers are
*trying* to be difficult by changing the language to break backwards
compatibility.  That's the difference I think.

JT

-- 
http://mail.python.org/mailman/listinfo/python-list