"as" keyword woes
A bottom line / pragmatic question... hopefully not a FAQ. Why was it necessary to make "as" a reserved keyword? And more to the point, why was it necessary to prevent developers from being able to refer to attributes named "as"? For example, this code breaks as of 2.6 / 3.0: Class C: Pass obj = C() obj.bs = 2 # valid obj.as = 1 # syntax error obj.cs = 3 # valid Just to be clear: I understand why it breaks, since "as" is now a keyword, I also know that one can use workarounds such as: obj.__dict__['as'] = 1 to maintain compatibility. What I want to understand is why this parser change was necessary in order to enable new 2.6/3.0 features. Was this change potentially avoidable? Why can't the parser distinguish between a standalone " as " keyword and ".as" used as an object/attribute reference? Couldn't we have continued along just fine using a smarter parser without elevating "as" to reserved status (and thus potentially breaking a 10+ years of existing code)? Thank you for enlighting me! (Unfortunately, our project may now have to maintain a branch at 2.5.x in order to preserve compatibility with existing third-party scripts & infrastructure which routinely rely upon "as" as an object method. Sigh.) Cheers, Warren -- http://mail.python.org/mailman/listinfo/python-list
Re: "as" keyword woes
> Because it can be used at the import statement to let the imported thing > be known under another name? > Something like: > > >>> import xml.etree.ElementTree as ET Yes, but that syntax worked fine for years without "as" actually having to be a keyword. There must be something more going on here... I have to believe that was some reason or benefit gained from making it a keyword, and thus prohibiting its use anywhere else, as opposed to sticking with the status quo. Otherwise this would merely be a pointless change (perhaps even destructive!). -- http://mail.python.org/mailman/listinfo/python-list
"as" keyword woes
> > Why was it necessary to make "as" a reserved keyword? > > I can't answer for the Python developers as to why they *did* make it > a reserved word. > > But I can offer what I believe is a good reason why it *should* be a > reserved word: Because simple is better than complex, and special > cases aren't special enough to break the rules. So you prefer broken code to broken rules, eh? Your customers must love that! This is exactly the kind of ivory-tower thinking I feared might be behind the decision (form over function, damn the users to hell, etc.) > > And more to the point, why was it necessary to prevent developers > > from being able to refer to attributes named "as"? > > There aren't special rules for which names can be use in which way, > and that's a *good* thing. Any name that is valid in one area of > Python syntax is valid in all Python syntax. Except that Python syntax has proven itself to be a non-backwards compatible moving target. Eliminating cruft and adding new functionality is one thing, but introducing a whole new two-letter keyword so long after the game has begun? That seems well over the line of what can be considered reasonable. > > Why can't the parser distinguish between a standalone " as " keyword > > and ".as" used as an object/attribute reference? > > Because that would require special-casing some names as being > forbidden in syntax where other names are allowed. Special cases in > language syntax are to be avoided where feasible. Am I the only one picking up on the irony here? "as" exists largely to provide a mechanism for working around namespace conflicts -- except, apparently, conflicts involving "as". The fact that "as" itself creates an insurmountable namespace collision is just killing me! How absurd. Anyway, it seems obvious that the right decision for our customers (or more importantly, for their countless lines of autogenerated-Python log, state, and code files from the past decade) is to stick with C/Python 2.5.x for the time being and plan to make the jump to a threads-capable and hopefully backwards-compatible Python implementation as soon as possible (IronPython perhaps?). That seems like a sensible path around the breakage and enduring limitations of C/Python 2.6 or 3. Or in other words, if we're all going to go through a Python compatibility-disconnect, then we might as well kill as many birds as possible in a single pass -- and that single-threaded interpreter bird is definitely a "dodo" in danger of extinction. By the way, congratulations on 3.0 final! But to be brutally honest...in this many-core world we must all accept and deal with today, it is hard to imagine how a non-multithreaded AND non-backwards compatible Python implementation can have much of a life ahead of it, with or without an "as" keyword. I do sincerely hope I am wrong about this, but it is seems quite possible that C/Python's glory days are now behind us. And if so, then thank you all for so many wonderful years of effort and participation! C/Python has had a great run, and Python syntax, in a multiplicity of forms, will surely live on for many decades to come. Python has changed the world, and very much so for the better. Well done! Cheers, Warren -- http://mail.python.org/mailman/listinfo/python-list
Re: "as" keyword woes
> I don't know how you infer any of those from what I said, nor > from the process of introducing features in Python. None of > what you say there rings at all true with anything I've > experienced in Python's core or the attitudes surrounding > development if the language; indeed, quite the opposite. That has been my experience as well, which is why this particular action seems surprising and out of character. > Speaking of irony, you're complaining about namespace > conflicts with a -two character identifier- you've chosen. > Here's a hint: choose better names. Hey, come on now -- developers working on top of an existing language bear nowhere near the responsibility as the language & implementation maintainers. Also, note that the fields of math and science are filled with short identifiers with well-defined meanings -- brevity doesn't mean ambiguous within a given application domain! But regardless, our scripts use "as" in the same way as Python -- to change the effective appearance of an object, albeit in a representational rather than naming space. So if we're wrong in our choice, then so is Python. In addition, note that my choice of a concise method identifier affects only my users. Python's introduction of a new keyword affects the entire Python world code base, so perhaps you should be directing your "choose better names" criticism in another direction? > To match your honesty, I'm somewhat tired with the trend of > some people to hit -one- issue with Python and suddenly lash > out like children over all the same old tired crap. Have you > even looked at multiprocessing? Have you contributed to any > projects working on GIL- less implementations? Or are you > just regurgitating the same bullet points we've heard time > and time again? Multiprocessing solves some problems, but it is unsuitable for high-frequency handoffs of large (in memory) objects between many independent threads/processes -- the HPC object/data flow parallelization model in a nutshell. Furthermore, not every coder has the compsci chops to work on language & VM implementations (my PhD involved programming DNA and directing evolution in a test tube, not parse trees and state machines). But that isn't to say I didn't try: at one point I even sketched out a possible TLS-based GIL workaround for handling the issue without breaking the existing C/API. It was of course shunned by those who knew better...for performance reasons IIRC. > For chrissake, if you cannot do ANYTHING but BITCH about a > change, then you've no damn right to consider yourself a > programmer. Real programmers find solutions, not excuses. Though I have certainly bitched about the threading issue multiple times on mailing lists including baypiggies and python-dev, bitching is not the only thing I've done. Having come to grips with my own coding limitations, I also offered to contribute financial resources from my own Python-enhanced business in support of GIL-removal -- before Python 3.0 was finalized. Unfortunately, no one responded to my offer. Even today, I am indirectly proposing the solution of "as" not being a reserved keyword since it has worked just fine for years without it. Yes that's late, but I didn't see this particular trainwreck coming (since it is not actually our current code which breaks, but rather, quantities of code created years ago but which must still run with fidelity into the far off future). Installing a Python macro preprocessor is another after-the-fact possible solution which may bear further investigation... Also today, I am proposing a pragmatic general solution for projects like ours in addressing both the 2.6/3.0 compatibility and threading situations. Specifically: deliberate migration away from C/Python and on to alternate VMs which happen to support Python syntax. Obviously none of my past efforts yielded fruit -- but it is wrong and unfair of you to assume and accuse me of not trying to come up with solutions. Solutions are virtually the entire game! > > And if so, then thank you all for so many wonderful years of effort > > and participation! > > You're welcome. Don't let the door hit you on the ass on your way out. But who's leaving who exactly? Surely a language as beautiful as Python will easily transcend the limitations of its flagship implementation (if or to the extent that such an implementation cannot keep pace with the times). That's all well and good -- it may even end up being the next great leap forward for the language. I believe Guido has even said as much himself. Warren -- http://mail.python.org/mailman/listinfo/python-list
"as" keyword woes
> Now, instead of keeping that special status, it was decided to make
> it a reserved word since there's a new use case in Python 2.6 for
> it as well - catching exceptions:
>
> >>> try:
> ... 1/0
> ... except Exception as exc_object:
> ... print exc_object
> ...
> integer division or modulo by zero
>
> The Python developers always try very hard not to introduce new
> keywords to the language, but every now and then, it's better to
> go with the addition and cause some breakage.
Thank you for finally answering my original question. If the above use
(or that of with), for implementation reasons, *requires* "as" to be a
keyword, then I can understand their decision to break Python. But what
I can't understand is the decision to break 2.6 instead of 3.0. 2.x was
supposed to remain backwards compatible, with the thinking that 2.x
would be maintained in parallel for quite some time. 3.x was supposed
to be the compatibility break. Not so, apparently.
> In this case, it's easy enough to find the files that break.
> Just run compileall.py on all your files and Python 2.6 will tell
> you which ones need fixing:
>
> python2.6 -c 'import compileall;compileall.compile_dir(".")'
But that assumes source code is a closed set. Unfortunately, in our
case, our code actually writes new Python code itself in the form of
document-like-objects intended for future re-execution. In that sense,
our code base is an open set relying upon Python's backward
compatibility (albeit in a very limited ways, but no source code can be
immune to introduction of new language keywords).
> This idea of CPython not being threads-capable is FUD. CPython
> works perfectly well in multi-threaded environments.
With all due respect, calling CPython out on the fact that it delivers
the efficiency of only one single CPU core even when there are N Python
threads running with N-1 idle cores available on a system is not
misleading FUD. It is the truth.
With all due respect, calling CPython out on the fact that its
developer-users cannot even work around this problem elegantly by
instantiating multiple independent Python interpreters running
concurrently within a single process (with limited but well defined
channels of communication between them) is not misleading FUD. It is
also the truth.
Given that the next round of high-end workstations will likely have
12-16 cores, N CPython native threads will be more than an order of
magnitude (>10 fold) less efficient than N Python-like threads on a true
threads-capable VM like the CLR. 3-4 years later, it will be 100-fold
less efficient, and on and on, in a 1/(2^T) geometric rate of declining
performance. That is near-term reality, not misleading FUD.
That the powers-that-be consider the above situation working "perfectly
well in multi-threaded environments" is rather telling. That the
CPython 3.0 compatibility-break was finalized without resolution of
CPython's thread performance issues should absolutely give rise to
well-founded Fear, Uncertainty, and Doubt about the utility of the
CPython 3.0 VM implementation in the minds of current users who must
deliver performant software for a living.
> There are, of course, situations where using a multi-threaded approach
> is not necessarily the right way to approach a problem.
Yes, we have been lectured about this endlessly. We are told that
threads are bad for various reasons, that they aren't ever the right
solution, that we should be using shared memory, separate processes, or
native code, or that real multithreading would break CPython library
compatibility (!) and so on, all despite the fact that threads work
perfectly fine in other VMs, including some VMs which run Python
dialects. Threads are indeed the optimal solution to certain problems,
and those problems are still not solvable with CPython 3.0.
Is it too much to hold out hope for a native Pythonic solution to the
multithreading performance issues inside of the CPython VM itself? Only
time will tell... but time is rapidly running out.
Warren
--
http://mail.python.org/mailman/listinfo/python-list
To Troll or Not To Troll
> Yet Another Python Troll (the ivory tower reference, as well as the > abrupt shift from complaining about keywords to multiprocessing), I > have to point out that Python does add new keywords, it has done so in > the past, and there was a considerable amount of warning, including an > automated deprecation warning in the very version you are going to > recommend to your "customers' (I don't actually think you have any > customers). ROFL! I'm sorry, you're right -- this has all been a figment of my imagination... http://www.pymolwiki.org/index.php/Covers http://images.google.com/images?q=pymol So don't mind me -- I clearly don't know what I'm talking about. -- http://mail.python.org/mailman/listinfo/python-list
RE: To Troll or Not To Troll
> I still would have to call your management of the problem considerably > into question - your expertise at writing mathematical software may > not be in question, but your skills and producing and managing a > software product are. You have nobody at your organization, which > sells a product that relies on Python, who follows python-dev? Or who > even reads the changelogs for new python versions? You should have > known about the "as" keyword change *over a year ago*, even if the > import bug was masking the deprecation warning. Everything else aside, > I can't get past that issue with your complaints. I *have* gone back > now and read all the posts in all the threads and I still have not > seen a single post from you even hinting that you might have any > responsibility in the matter. Well then, let me set the record straight on that one point: I admit that it was entirely my mistake (and mine alone) to implicitly assume, by adopting such a logging & persistence architecture (dating back to 1.5.2, mind you!), that new keywords would not be introduced into the Python language so as to potentially break all existing Python code. Silly me! How unreasonable. -- http://mail.python.org/mailman/listinfo/python-list
"as" keyword woes
> I still have not > >> seen a single post from you even hinting that you might have any > >> responsibility in the matter. > > > > Well then, let me set the record straight on that one point: > > > > I admit that it was entirely my mistake (and mine alone) to implicitly > > assume, by adopting such a logging & persistence architecture (dating > > back to 1.5.2, mind you!), that new keywords would not be introduced > > into the Python language so as to potentially break all existing Python > > code. > > > > Silly me! How unreasonable. > > > > Pythons backwards compatibility policy is available here: > http://www.python.org/dev/peps/pep-0005/ Thank you! Just to end on a more positive note: As someone who makes a living from Python rather than someone who lives to make Python, I recognize that there will be ancillary casualties in every major battle. Though I may whine incessantly about all of our pre-2.5 log-file/documents being one such casualty (your various accusations notwithstanding, we did indeed patch our own code as soon as the deprecation warnings appeared in 2.5!), if the Python 2.6 "as" keyword break is truly for the greater good, then so be it. Warren -- http://mail.python.org/mailman/listinfo/python-list
Re: To Troll or Not To Troll (aka: "as" keyword woes)
> From: Ben Finney <[EMAIL PROTECTED]> > > "Chris Mellon" <[EMAIL PROTECTED]> writes: > > > Peculiarities in usenet resulted in this discussion having several > > threads and I missed some messages before I wrote this email. > > I'll put this more bluntly: Warren's messages to date > egregiously break the flow of discussion. > > Warren, in the interest of sane discussion in these forums, please: > > * preserve attribution lines on quoted material so we can see who > wrote what. > > * use the convention of ?New subject (was: Old subject)? when you > change the ?Subject? field of a message. > > * switch to a client that preserves threading in messages you send, > i.e. that properly constructs the ?References? and ?In-Reply-To? > fields. > > General advice good for everyone, of course, but particularly > apropos to this reply. Any one of the above is detrimental to > omit; striking on all three makes a discussion almost > impossible to follow. (Thank you, though, for avoiding the > worse habit of top posting!) Thank so much for the suggestions Ben. Sorry that I am personally unable to live up to your high standards, but it is nevertheless an honor to partipicate in such a helpful and mutually respectful community mailing list! Warren -- http://mail.python.org/mailman/listinfo/python-list
Re: "as" keyword woes
> Date: Fri, 05 Dec 2008 22:22:38 -0800
> From: Dennis Lee Bieber <[EMAIL PROTECTED]>
> Subject: Re: "as" keyword woes
> To: python-list@python.org
> Message-ID: <[EMAIL PROTECTED]>
>
> I'm still in the dark as to what type of data could
> even inspire the
> use of "as" as an object name... A collection of "a" objects? In which
> case, what are the "a"s?
Please let me clarify. It is not "as" as a standalone object that we
specifically miss in 2.6/3, but rather, the ability to use ".as" used as
a method or attribute name.
In other words we have lost the ability to refer to "as" as the
generalized OOP-compliant/syntax-independent method name for casting:
new_object = old_object.as(class_hint)
# For example:
float_obj = int_obj.as("float")
# or
float_obj = int_obj.as(float_class)
# as opposed to something like
float_obj = int_obj.asFloat()
# which requires a separate method for each cast, or
float_obj = (float)int_obj
# which required syntax-dependent casting [language-based rather than
object-based].
Of course, use of explicit casting syntax "(float)" is fine if you're
restricting yourself to Python and other languages which support
casting, but that solution is unavailable inside of a pure OOP
message-passing paradigm where object.method(argument) invocations are
all you have to work with.
Please note that use of object.asClassname(...) is a ubiqitous
convention for casting objects to specific classes (seen in ObjectiveC,
Java, SmallTalk, etc.).
There, I assert that 'object.as(class_reference)' is the simplest and
most elegant generalization of this widely-used convention. Indeed, it
is the only obvious concise answer, if you are limited to using methods
for casting.
Although there are other valid domain-specific uses for "as" as either a
local variable or attribute names (e.g. systematic naming: as, bs, cs),
those aren't nearly as important compared to "as" being available as the
name of a generalized casting method -- one that is now strictly denied
to users of Python 2.6 and 3.
As someone somewhat knowledgable of how parsers work, I do not
understand why a method/attribute name "object_name.as(...)" must
necessarily conflict with a standalone keyword " as ". It seems to me
that it should be possible to unambiguously separate the two without
ambiguity or undue complication of the parser.
So, assuming I now wish to propose a corrective PEP to remedy this
situation for Python 3.1 and beyond, what is the best way to get started
on such a proposal?
Cheers,
Warren
--
http://mail.python.org/mailman/listinfo/python-list
Re: "as" keyword woes
> Date: Sat, 6 Dec 2008 12:13:16 -0800 (PST) > From: Carl Banks <[EMAIL PROTECTED]> > Subject: Re: "as" keyword woes > To: python-list@python.org > Message-ID: > > (snip) > > If you write a PEP, I advise you to try to sound less whiny and than > you have in this thread. > > (snip) Ehem, well, such comments notwithstanding, I thank everyone who responded to my latest post on this topic for taking my inquiry seriously, and for providing cogent, focused, well-reasoned feedback while not resorting to name-calling, to false accusations on top of baseless assumptions, or to explicit personal attacks on my competence, sincerity, experience, credibility, or form. To you especially, I am grateful for your input for your years of service to the community and to the noble ideals you embody in the Python project. May the rest of us (not just myself) be ashamed of our lesser conduct and learn from you exemplary performance. So to summarize, having assimilated all responses over the past several days (python-list as well as python-dev, for the newcomers), I now accept the following as self-evident: -> "as", as a Python keyword, is a here to stay: Love it or leave it. -> Likewise ditto for the GIL: if you truly need Python concurrency within a single process, then use a Python implementation other than CPython. Season's greetings to all! Peace. Cheers, Warren -- http://mail.python.org/mailman/listinfo/python-list
Safe eval of insecure strings containing Python data structures?
I would like to parse arbitrary insecure text string containing nested
Python data structures in eval-compatible form:
# For example, given a "config.txt" such as:
{
'my_atom' : 1.20,
'my_dict' : { 2:50 , 'hi':'mom'},
'my_list' : [ (1,2,3), [4.5,6.9], 'foo', 0 ]
}
# I would like to do something like this:
empty_space = {'__builtins__' : {}}
try:
config = eval(open("config.txt").read(), empty_space, empty_space)
except:
config = {}
print config
# But I know for certain that the above approach is NOT secure since
object attributes can still be accessed...
So is there an equally convenient yet secure alternative available for
parsing strings containing Python data structure definitions?
Thanks in advance for any pointers!
Cheers,
Warren
--
http://mail.python.org/mailman/listinfo/python-list
RE: Safe eval of insecure strings containing Python data structures?
JSON rocks! Thanks everyone. Ben wrote: >More generally, you should never execute (via eval, exec, or whatever) >*any* instruction from an untrusted path; especially not arbitrary >data from an input stream. Wow, for the record, I completely disagree with this point of view: Today's web apps wouldn't exist without safe forms of untrusted eval/exec (Javascript anyone?). Such dogma is appropriate when dealing with the CPython VM, but not as a general principle. "Rocket fuel may be dangerous, but you ain't shooting the moon without it!" Cheers, Warren -- http://mail.python.org/mailman/listinfo/python-list
RE: Python-list Digest, Vol 61, Issue 368
> From: "Andy O'Meara" <[EMAIL PROTECTED]> > Unfortunately, a shared address region doesn't work when you have > large and opaque objects (e.g. a rendered CoreVideo movie in the > QuickTime API or 300 megs of audio data that just went through a > DSP). Then you've got the hit of serialization if you're got > intricate data structures (that would normally would need to be > serialized, such as a hashtable or something). Also, if I may speak > for commercial developers out there who are just looking to get the > job done without new code, it's usually always preferable to just a > single high level sync object (for when the job is complete) than to Just to chime as a CPython-based ISV from the scientific visualization realm, we face the same problem & limitations due to lack of threading (or at least multiple independent interpreters). A typical use case might be a 1-3 GB dataset (molecular dynamics trajectory and derived state) subjected to asynchronous random read/write by N threads each running on one of N cores in parallel. We get by jettisoning CPython almost entirely and working in C for all tasks other than the most basic operations: thread creation, workload scheduling, mutexes, and thread deletion. The biggest problem is not for the most compute-intensive tasks (where use of C is justified), but for those relatively short-running but algorithmically complex tasks which could be done much more easily from Python than from C (e.g. data organization, U.I. survey/present tasks, rarely used transformations, ad hoc scripting experiments, etc.). Cheers, Warren -- http://mail.python.org/mailman/listinfo/python-list
