Re: Future standard GUI library
On 5/20/2013 1:04 AM, Vito De Tullio wrote: Terry Jan Reedy wrote: Do you think tkinter is going to be the standard python built-in gui solution as long as python exists? AT the moment, there is nothing really comparable that is a realistic candidate to replace tkinter. FLTK? (http://www.fltk.org/index.php) tkinter is the Python wrapper of the tk library, just as wxpython is the python wrapper of the wx library. I do not see a py-fltk wrapper. -- http://mail.python.org/mailman/listinfo/python-list
Question about ast.literal_eval
Hi all
I am trying to emulate a SQL check constraint in Python. Quoting from
the PostgreSQL docs, "A check constraint is the most generic constraint
type. It allows you to specify that the value in a certain column must
satisfy a Boolean (truth-value) expression."
The problem is that I want to store the constraint as a string, and I
was hoping to use ast.literal_eval to evaluate it, but it does not work.
>>> x = 'abc'
>>> x in ('abc', xyz')
True
>>> b = "x in ('abc', 'xyz')"
>>> eval(b)
True
>>> from ast import literal_eval
>>> literal_eval(b)
ValueError: malformed node or string: <_ast.Compare object at ...>
Is there a safe way to do what I want? I am using python 3.3.
Thanks
Frank Millman
--
http://mail.python.org/mailman/listinfo/python-list
CFP: MuCoCoS Workshop at PACT-2013 (Edinburgh, Scotland, UK)
CALL FOR PAPERS: 6th International Workshop on Multi/many-Core Computing Systems (MuCoCoS-2013) September 7, 2013, Edinburgh, Scotland, UK in conjunction with the 22nd Int. Conference on Parallel Architectures and Compilation Techniques (PACT-2013) http://www.ida.liu.se/conferences/mucocos2013 AIMS AND SCOPE The pervasiveness of homogeneous and heterogeneous multi-core and many-core processors, in a large spectrum of systems from embedded and general-purpose to high-end computing systems, poses major challenges to software industry. In general, there is no guarantee that software developed for a particular architecture will be executable (that is functional) on another architecture. Furthermore, ensuring that the software preserves some aspects of performance behavior (such as temporal or energy efficiency) across different such architectures is an open research issue. Therefore, this workshop focuses on language level, system software and architectural solutions for performance portability across different architectures and for automated performance tuning. The topics of the MuCoCoS workshop include but are not limited to: * Performance measurement, modeling, analysis and tuning * Portable programming models, languages and compilation techniques * Run-time systems and hardware support mechanisms for auto-tuning * Tunable algorithms and data structures * Case studies highlighting performance portability and tuning. As the sixth workshop in the series MuCoCoS 2008 (Barcelona, Spain), MuCoCoS 2009 (Fukuoka, Japan), MuCoCoS 2010 (Krakow, Poland), MuCoCoS 2011 (Seoul, Korea), and MuCoCoS 2012 (Salt Lake City, USA), MuCoCoS 2013 will be held in Edinburgh, UK, in conjunction with the 22nd International Conference on Parallel Architectures and Compilation Techniques (PACT 2013). SUBMISSION GUIDELINES The papers should be prepared using the IEEE format (two-column, 10pt, LaTeX users please use style IEEEtran.cls), and no longer than 10 pages. Submitted papers will be carefully evaluated based on originality, significance to workshop topics, technical soundness, and presentation quality. Please submit your paper (as PDF, viewable by Adobe Reader v5.0 or higher, with all fonts embedded please) electronically using the online submission system https://www.easychair.org/conferences/?conf=mucocos2013 Submission of the paper implies that, should the paper be accepted, at least one of the authors will register and present the paper at the workshop. Accepted papers will be published in electronic form in IEEE Xplore (confirmation pending). They will also be included in the PACT USB proceedings. We also plan to invite authors of the best MuCoCoS papers after the workshop to submit their extended workshop papers to a special issue of Computing Journal (Springer) (confirmation pending). IMPORTANT DATES Submission: May 27, 2013 (Firm Deadline) Notification: June 23, 2013 Camera-ready: July 7, 2013 Workshop: September 7, 2013 WORKSHOP ORGANIZATION Christoph Kessler, Linköping University, Sweden, program chair Sabri Pllana, Linnaeus University, Sweden, co-chair PROGRAM COMMITTEE Marco Aldinucci, Univ. of Torino, Italy Beverly Bachmayer, Intel, Germany David Bader, Georgia Tech, USA Jacob Barhen, Oak Ridge National Lab, USA Siegfried Benkner, Univ. of Vienna, Austria Franz Franchetti, Carnegie Mellon University, USA Grigori Fursin, INRIA, France Jörg Keller, FernUniv. Hagen, Germany Lasse Natvig, NTNU Trondheim, Norway Mitsuhisa Sato, Univ. of Tsukuba, Japan Samuel Thibault, INRIA / Univ. of Bordeaux, France Philippas Tsigas, Chalmers University, Sweden Jakub Yaghob, Charles University Prague, Czech Republic -- http://mail.python.org/mailman/listinfo/python-list
RE: Question about ast.literal_eval
It seems to me you can't use ast.literal_eval()[1] to evaluate that kind of
expression because it's just for literals[2].
Why don't you use eval()?
[1] http://docs.python.org/2/library/ast.html#ast-helpers
[2] http://docs.python.org/2/reference/lexical_analysis.html#literals
> To: python-list@python.org
> From: fr...@chagford.com
> Subject: Question about ast.literal_eval
> Date: Mon, 20 May 2013 09:05:48 +0200
>
> Hi all
>
> I am trying to emulate a SQL check constraint in Python. Quoting from
> the PostgreSQL docs, "A check constraint is the most generic constraint
> type. It allows you to specify that the value in a certain column must
> satisfy a Boolean (truth-value) expression."
>
> The problem is that I want to store the constraint as a string, and I
> was hoping to use ast.literal_eval to evaluate it, but it does not work.
>
x = 'abc'
x in ('abc', xyz')
> True
b = "x in ('abc', 'xyz')"
eval(b)
> True
from ast import literal_eval
literal_eval(b)
> ValueError: malformed node or string: <_ast.Compare object at ...>
>
> Is there a safe way to do what I want? I am using python 3.3.
>
> Thanks
>
> Frank Millman
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, May 20, 2013 at 5:05 PM, Frank Millman wrote:
> Hi all
>
> I am trying to emulate a SQL check constraint in Python. Quoting from the
> PostgreSQL docs, "A check constraint is the most generic constraint type. It
> allows you to specify that the value in a certain column must satisfy a
> Boolean (truth-value) expression."
>
> The problem is that I want to store the constraint as a string, and I was
> hoping to use ast.literal_eval to evaluate it, but it does not work.
>
x = 'abc'
x in ('abc', xyz')
> True
b = "x in ('abc', 'xyz')"
eval(b)
> True
from ast import literal_eval
literal_eval(b)
> ValueError: malformed node or string: <_ast.Compare object at ...>
>
> Is there a safe way to do what I want? I am using python 3.3.
An SQL constraint has a whole lot that Python can't do conveniently,
and vice versa, so I think you do yourself a disservice by trying to
tie yourself to that.
The problem here is that 'in' is an operator, so this is not a
literal. One possible solution would be to separate out your operator
(and have just a handful permitted), and then use ast.literal_eval for
just the second operand - something like this:
import ast
import operator
ops = {
'in':lambda x,y: x in y, # operator.contains has the args backwards
'==':operator.eq, # or use '=' for more SQL-like syntax
'<':operator.lt,
'>':operator.gt,
}
op, value = 'in', "('abc', 'xyz')"
x = 'abc'
if ops[op](x,ast.literal_eval(value)):
print("Constraint passed")
else:
print("Ignore this one")
ChrisA
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to run a python script twice randomly in a day?
On 20May2013 09:47, Avnesh Shakya wrote: | On Mon, May 20, 2013 at 9:42 AM, Cameron Simpson wrote: | > On 19May2013 20:54, Avnesh Shakya wrote: | > |How to run a python script twice randomly in a day? Actually | > | I want to run my script randomly in a day and twice only. Please | > | help me.. how is it possible. | > | > Do you mean "run twice a day, each at random times"? | > | > If so, do the obvious: at midnight, pick two random times. Sleep | > until the first time, run the script, sleep until the second time, | > run the script. | > | > There are various ways to do the sleeping and midnight bits; they're | > up to you. | | Thanks, Can you mail documentation or link for it? I am totally new for it. 1: Please reply on-list; you asked the list, the discussion should remain there. I have added the list to the CC line. 2: Please don't top-post. Quote only the relevant bits of the previous message and reply below. If the message is long, do that in pieces. [quote] reply [quote] reply. Like a conversation. Now, to your questions. A UNIX user would use "cron" to schedule the midnight job and from the midnight job then probably use "at" to schedule the other jobs to run at specific times. See "man 1 crontab", "man 5 crontab" to submit the cron jobs and "man at" to submit the once off jobs. UNIX means Linux, Solaris, AIX, any of the BSDs (includes MacOSX), etc. You could a small python script for the midnight job, and it would submit the "at" jobs. See the "random" module to compute a pseudorandom number (and thus a random time), the "datetime" module to compute the dates and times from that number, and the "subprocess" module to submit the "at" job for the chosen run times. The modules are here: http://docs.python.org/3/py-modindex.html That presumes python 3; if you're using python 2 the docs for that are available at the same web site. You don't need to use "at". Your midnight job could just compute the time in seconds to each job time, then use the "time.sleep" function to delay until then. See the "time" module at the above link. Nobody on this list will write your program for you. Attempt to write the program, then come to the list with your attempt and any problems. People will help if you've made an initial effort, and continue to try. This question has the "feel" of a homework question; the point of homework is for you to learn by solving a problem yourself. It is fine to seek help, but you must make the effort yourself. Come back with specific questions. Cheers, -- Cameron Simpson I'm not weird; I'm gifted. -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On 20May2013 07:25, Fábio Santos wrote: | On 18 May 2013 20:33, "Dennis Lee Bieber" wrote: | > Python threads work fine if the threads either rely on intelligent | > DLLs for number crunching (instead of doing nested Python loops to | > process a numeric array you pass it to something like NumPy which | > releases the GIL while crunching a copy of the array) or they do lots of | > I/O and have to wait for I/O devices (while one thread is waiting for | > the write/read operation to complete, another thread can do some number | > crunching). | | Has nobody thought of a context manager to allow a part of your code to | free up the GIL? I think the GIL is not inherently bad, but if it poses a | problem at times, there should be a way to get it out of your... Way. The GIL makes individual python operations thread safe by never running two at once. This makes the implementation of the operations simpler, faster and safer. It is probably totally infeasible to write meaningful python code inside your suggested context manager that didn't rely on the GIL; if the GIL were not held the code would be unsafe. It is easy for a C extension to release the GIL, and then to do meaningful work until it needs to return to python land. Most C extensions will do that around non-trivial sections, and anything that may stall in the OS. So your use case for the context manager doesn't fit well. -- Cameron Simpson Gentle suggestions being those which are written on rocks of less than 5lbs. - Tracy Nelson in comp.lang.c -- http://mail.python.org/mailman/listinfo/python-list
RE: Please help with Threading
> Date: Sun, 19 May 2013 13:10:36 +1000 > From: c...@zip.com.au > To: carlosnepomuc...@outlook.com > CC: python-list@python.org > Subject: Re: Please help with Threading > > On 19May2013 03:02, Carlos Nepomuceno wrote: > | Just been told that GIL doesn't make things slower, but as I > | didn't know that such a thing even existed I went out looking for > | more info and found that document: > | http://www.dabeaz.com/python/UnderstandingGIL.pdf > | > | Is it current? I didn't know Python threads aren't preemptive. > | Seems to be something really old considering the state of the art > | on parallel execution on multi-cores. > | What's the catch on making Python threads preemptive? Are there any ongoing > projects to make that? > > Depends what you mean by preemptive. If you have multiple CPU bound > pure Python threads they will all get CPU time without any of them > explicitly yeilding control. But thread switching happens between > python instructions, mediated by the interpreter. I meant operating system preemptive. I've just checked and Python does not start Windows threads. > The standard answers for using multiple cores is to either run > multiple processes (either explicitly spawning other executables, > or spawning child python processes using the multiprocessing module), > or to use (as suggested) libraries that can do the compute intensive > bits themselves, releasing the while doing so so that the Python > interpreter can run other bits of your python code. I've just discovered the multiprocessing module[1] and will make some tests with it later. Are there any other modules for that purpose? I've found the following articles about Python threads. Any suggestions? http://www.ibm.com/developerworks/aix/library/au-threadingpython/ http://pymotw.com/2/threading/index.html http://www.laurentluce.com/posts/python-threads-synchronization-locks-rlocks-semaphores-conditions-events-and-queues/ [1] http://docs.python.org/2/library/multiprocessing.html > Plenty of OS system calls (and calls to other libraries from the > interpreter) release the GIL during the call. Other python threads > can run during that window. > > And there are other Python implementations other than CPython. > > Cheers, > -- > Cameron Simpson > > Processes are like potatoes. - NCR device driver manual > -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
[Corrected top-posting] >> To: python-list@python.org From: fr...@chagford.com Subject: Question about ast.literal_eval Date: Mon, 20 May 2013 09:05:48 +0200 Hi all I am trying to emulate a SQL check constraint in Python. Quoting from the PostgreSQL docs, "A check constraint is the most generic constraint type. It allows you to specify that the value in a certain column must satisfy a Boolean (truth-value) expression." The problem is that I want to store the constraint as a string, and I was hoping to use ast.literal_eval to evaluate it, but it does not work. On 20/05/2013 09:34, Carlos Nepomuceno wrote: It seems to me you can't use ast.literal_eval()[1] to evaluate that kind of expression > because it's just for literals[2]. Why don't you use eval()? Because users can create their own columns, with their own constraints. Therefore the string is user-modifiable, so it cannot be trusted. Frank -- http://mail.python.org/mailman/listinfo/python-list
RE: Please help with Threading
> Date: Mon, 20 May 2013 17:45:14 +1000 > From: c...@zip.com.au > To: fabiosantos...@gmail.com > Subject: Re: Please help with Threading > CC: python-list@python.org; wlfr...@ix.netcom.com > > On 20May2013 07:25, Fábio Santos wrote: > | On 18 May 2013 20:33, "Dennis Lee Bieber" wrote: > |> Python threads work fine if the threads either rely on intelligent > |> DLLs for number crunching (instead of doing nested Python loops to > |> process a numeric array you pass it to something like NumPy which > |> releases the GIL while crunching a copy of the array) or they do lots of > |> I/O and have to wait for I/O devices (while one thread is waiting for > |> the write/read operation to complete, another thread can do some number > |> crunching). > | > | Has nobody thought of a context manager to allow a part of your code to > | free up the GIL? I think the GIL is not inherently bad, but if it poses a > | problem at times, there should be a way to get it out of your... Way. > > The GIL makes individual python operations thread safe by never > running two at once. This makes the implementation of the operations > simpler, faster and safer. It is probably totally infeasible to > write meaningful python code inside your suggested context > manager that didn't rely on the GIL; if the GIL were not held the > code would be unsafe. I just got my hands dirty trying to synchronize Python prints from many threads. Sometimes they mess up when printing the newlines. I tried several approaches using threading.Lock and Condition. None of them worked perfectly and all of them made the code sluggish. Is there a 100% sure method to make print thread safe? Can it be fast??? > It is easy for a C extension to release the GIL, and then to do > meaningful work until it needs to return to python land. Most C > extensions will do that around non-trivial sections, and anything > that may stall in the OS. > > So your use case for the context manager doesn't fit well. > -- > Cameron Simpson > > Gentle suggestions being those which are written on rocks of less than 5lbs. > - Tracy Nelson in comp.lang.c > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, May 20, 2013 at 5:50 PM, Frank Millman wrote: > On 20/05/2013 09:34, Carlos Nepomuceno wrote: >> Why don't you use eval()? >> > > Because users can create their own columns, with their own constraints. > Therefore the string is user-modifiable, so it cannot be trusted. Plenty of reason right there :) Is it a requirement that they be able to key in a constraint as a single string? We have a similar situation in one of the systems at work, so we divided the input into three(ish) parts: pick a field, pick an operator (legal operators vary according to field type - integers can't be compared against regular expressions, timestamps can use >= and < only), then enter the other operand. Sure, that cuts out a few possibilities, but you get 99.9%+ of all usage and it's easy to sanitize. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
My use case was a tight loop processing an image pixel by pixel, or crunching a CSV file. If it only uses local variables (and probably hold a lock before releasing the GIL) it should be safe, no? My idea is that it's a little bad to have to write C or use multiprocessing just to do simultaneous calculations. I think an application using a reactor loop such as twisted would actually benefit from this. Sure, it will be slower than a C implementation of the same loop, but isn't fast prototyping a very important feature of the Python language? On 20 May 2013 08:45, "Cameron Simpson" wrote: > On 20May2013 07:25, Fábio Santos wrote: > | On 18 May 2013 20:33, "Dennis Lee Bieber" wrote: > | > Python threads work fine if the threads either rely on > intelligent > | > DLLs for number crunching (instead of doing nested Python loops to > | > process a numeric array you pass it to something like NumPy which > | > releases the GIL while crunching a copy of the array) or they do lots > of > | > I/O and have to wait for I/O devices (while one thread is waiting for > | > the write/read operation to complete, another thread can do some number > | > crunching). > | > | Has nobody thought of a context manager to allow a part of your code to > | free up the GIL? I think the GIL is not inherently bad, but if it poses a > | problem at times, there should be a way to get it out of your... Way. > > The GIL makes individual python operations thread safe by never > running two at once. This makes the implementation of the operations > simpler, faster and safer. It is probably totally infeasible to > write meaningful python code inside your suggested context > manager that didn't rely on the GIL; if the GIL were not held the > code would be unsafe. > > It is easy for a C extension to release the GIL, and then to do > meaningful work until it needs to return to python land. Most C > extensions will do that around non-trivial sections, and anything > that may stall in the OS. > > So your use case for the context manager doesn't fit well. > -- > Cameron Simpson > > Gentle suggestions being those which are written on rocks of less than > 5lbs. > - Tracy Nelson in comp.lang.c > -- http://mail.python.org/mailman/listinfo/python-list
RE: Question about ast.literal_eval
> To: python-list@python.org > From: fr...@chagford.com > Subject: Re: Question about ast.literal_eval > Date: Mon, 20 May 2013 09:50:02 +0200 > > [Corrected top-posting] > >>> To: python-list@python.org >>> From: fr...@chagford.com >>> Subject: Question about ast.literal_eval >>> Date: Mon, 20 May 2013 09:05:48 +0200 >>> >>> Hi all >>> >>> I am trying to emulate a SQL check constraint in Python. Quoting from >>> the PostgreSQL docs, "A check constraint is the most generic constraint >>> type. It allows you to specify that the value in a certain column must >>> satisfy a Boolean (truth-value) expression." >>> >>> The problem is that I want to store the constraint as a string, and I >>> was hoping to use ast.literal_eval to evaluate it, but it does not work. >>> > > On 20/05/2013 09:34, Carlos Nepomuceno wrote: > >> It seems to me you can't use ast.literal_eval()[1] to evaluate that kind of >> expression >> because it's just for literals[2]. >> >> Why don't you use eval()? >> > > Because users can create their own columns, with their own constraints. > Therefore the string is user-modifiable, so it cannot be trusted. I understand your motivation but I don't know what protection ast.literal_eval() is offering that eval() doesn't. > Frank > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20/05/2013 09:55, Chris Angelico wrote: On Mon, May 20, 2013 at 5:50 PM, Frank Millman wrote: On 20/05/2013 09:34, Carlos Nepomuceno wrote: Why don't you use eval()? Because users can create their own columns, with their own constraints. Therefore the string is user-modifiable, so it cannot be trusted. Plenty of reason right there :) Is it a requirement that they be able to key in a constraint as a single string? We have a similar situation in one of the systems at work, so we divided the input into three(ish) parts: pick a field, pick an operator (legal operators vary according to field type - integers can't be compared against regular expressions, timestamps can use >= and < only), then enter the other operand. Sure, that cuts out a few possibilities, but you get 99.9%+ of all usage and it's easy to sanitize. ChrisA It is not a requirement, no. I just thought it would be a convenient short-cut. I had in mind something similar to your scheme above, so I guess I will have to bite the bullet and implement it. Thanks Frank -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, May 20, 2013 at 5:55 PM, Carlos Nepomuceno wrote: > I understand your motivation but I don't know what protection > ast.literal_eval() is offering that eval() doesn't. eval will *execute code*, while literal_eval will not. That's the protection. With ast.literal_eval, all that can happen is that it produces a single result value. In this case, unfortunately, that's insufficient; a comparison needs to be done, ergo it's not an entire literal. But something else in the ast module may be able to serve, or maybe literal_eval can do the bulk of the work. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20/05/2013 09:55, Carlos Nepomuceno wrote: Why don't you use eval()? Because users can create their own columns, with their own constraints. Therefore the string is user-modifiable, so it cannot be trusted. I understand your motivation but I don't know what protection ast.literal_eval() is offering that eval() doesn't. Quoting from the manual - "Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None." The operative word is 'safely'. I don't know the details, but it prevents the kinds of exploits that can be carried out by malicious code using eval(). I believe it is the same problem as SQL injection, which is solved by using parameterised queries. Frank -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, 20 May 2013 10:55:35 +0300, Carlos Nepomuceno wrote:
> I understand your motivation but I don't know what protection
> ast.literal_eval() is offering that eval() doesn't.
eval will evaluate any legal Python expression:
py> eval("__import__('os').system('echo Mwahaha! Now you are pwned!') or 42")
Mwahaha! And now you are pwned!
42
ast.literal_eval() does exactly what the name says: it will evaluate any
legal Python LITERAL, including ints, floats, lists, dicts and strings,
but not arbitrary expressions.
py> ast.literal_eval('123')
123
py> ast.literal_eval('[123, None, "spam"]')
[123, None, 'spam']
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On 20May2013 10:53, Carlos Nepomuceno wrote: | I just got my hands dirty trying to synchronize Python prints from many threads. | Sometimes they mess up when printing the newlines. | I tried several approaches using threading.Lock and Condition. | None of them worked perfectly and all of them made the code sluggish. Show us some code, with specific complaints. Did you try this? _lock = Lock() def lprint(*a, **kw): global _lock with _lock: print(*a, **kw) and use lprint() everywhere? For generality the lock should be per file: the above hack uses one lock for any file, so that's going to stall overlapping prints to different files; inefficient. There are other things than the above, but at least individual prints will never overlap. If you have interleaved prints, show us. | Is there a 100% sure method to make print thread safe? Can it be fast??? Depends on what you mean by "fast". It will be slower than code with no lock; how much would require measurement. Cheers, -- Cameron Simpson My own suspicion is that the universe is not only queerer than we suppose, but queerer than we *can* suppose. - J.B.S. Haldane "On Being the Right Size" in the (1928) book "Possible Worlds" -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20 May 2013 09:19, "Frank Millman" wrote: > Quoting from the manual - > > "Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None." > > The operative word is 'safely'. I don't know the details, but it prevents the kinds of exploits that can be carried out by malicious code using eval(). Literals are only a subset of expressions. The documentation is a bit misleading, by stating it accepts a "python expression". This individual is rightfully confused. -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On Mon, May 20, 2013 at 6:35 PM, Cameron Simpson wrote: > _lock = Lock() > > def lprint(*a, **kw): > global _lock > with _lock: > print(*a, **kw) > > and use lprint() everywhere? Fun little hack: def print(*args,print=print,lock=Lock(),**kwargs): with lock: print(*args,**kwargs) Question: Is this a cool use or a horrible abuse of the scoping rules? ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Diacretical incensitive search
On Fri, 2013-05-17, Olive wrote:
> One feature that seems to be missing in the re module (or any tools
> that I know for searching text) is "diacretical incensitive search". I
> would like to have a match for something like this:
> re.match("franc", "français")
...
> The algorithm to write such a function is trivial but there are a
> lot of mark we can put on a letter. It would be necessary to have the
> list of "a"'s with something on it. i.e. "à,á,ã", etc. and this for
> every letter. Trying to make such a list by hand would inevitably lead
> to some symbols forgotten (and would be tedious).
Ok, but please remember that the diacriticals are of varying importance.
The english "naïve" is easily recognizable when written as "naive".
The swedish word "får" cannot be spelled "far" and still be understood.
This is IMHO out of the scope of re, and perhaps case-insensitivity
should have been too. Perhaps it /would/ have been, if regular
expressions hadn't come from the ASCII world where these things are
easy.
/Jorgen
--
// Jorgen GrahnO o .
--
http://mail.python.org/mailman/listinfo/python-list
Re: Future standard GUI library
On 2013-05-20 08:00, Terry Jan Reedy wrote: On 5/20/2013 1:04 AM, Vito De Tullio wrote: Terry Jan Reedy wrote: Do you think tkinter is going to be the standard python built-in gui solution as long as python exists? AT the moment, there is nothing really comparable that is a realistic candidate to replace tkinter. FLTK? (http://www.fltk.org/index.php) tkinter is the Python wrapper of the tk library, just as wxpython is the python wrapper of the wx library. I do not see a py-fltk wrapper. It exists, but it's really old. http://pyfltk.sourceforge.net/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
It is pretty cool although it looks like a recursive function at first ;) On 20 May 2013 10:13, "Chris Angelico" wrote: > On Mon, May 20, 2013 at 6:35 PM, Cameron Simpson wrote: > > _lock = Lock() > > > > def lprint(*a, **kw): > > global _lock > > with _lock: > > print(*a, **kw) > > > > and use lprint() everywhere? > > Fun little hack: > > def print(*args,print=print,lock=Lock(),**kwargs): > with lock: > print(*args,**kwargs) > > Question: Is this a cool use or a horrible abuse of the scoping rules? > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On 20May2013 19:09, Chris Angelico wrote: | On Mon, May 20, 2013 at 6:35 PM, Cameron Simpson wrote: | > _lock = Lock() | > | > def lprint(*a, **kw): | > global _lock | > with _lock: | > print(*a, **kw) | > | > and use lprint() everywhere? | | Fun little hack: | | def print(*args,print=print,lock=Lock(),**kwargs): | with lock: | print(*args,**kwargs) | | Question: Is this a cool use or a horrible abuse of the scoping rules? I carefully avoided monkey patching print itself:-) That's... mad! I can see what the end result is meant to be, but it looks like a debugging nightmare. Certainly my scoping-fu is too weak to see at a glance how it works. -- Cameron Simpson I will not do it as a hack I will not do it for my friends I will not do it on a MacI will not write for Uncle Sam I will not do it on weekends I won't do ADA, Sam-I-Am - Gregory Bond -- http://mail.python.org/mailman/listinfo/python-list
RE: Please help with Threading
> Date: Mon, 20 May 2013 18:35:20 +1000
> From: c...@zip.com.au
> To: carlosnepomuc...@outlook.com
> CC: python-list@python.org
> Subject: Re: Please help with Threading
>
> On 20May2013 10:53, Carlos Nepomuceno wrote:
> | I just got my hands dirty trying to synchronize Python prints from many
> threads.
> | Sometimes they mess up when printing the newlines.
> | I tried several approaches using threading.Lock and Condition.
> | None of them worked perfectly and all of them made the code sluggish.
>
> Show us some code, with specific complaints.
>
> Did you try this?
>
> _lock = Lock()
>
> def lprint(*a, **kw):
> global _lock
> with _lock:
> print(*a, **kw)
>
> and use lprint() everywhere?
It works! Think I was running the wrong script...
Anyway, the suggestion you've made is the third and latest attempt that I've
tried to synchronize the print outputs from the threads.
I've also used:
### 1st approach ###
lock = threading.Lock()
[...]
try:
lock.acquire()
[thread protected code]
finally:
lock.release()
### 2nd approach ###
cond = threading.Condition()
[...]
try:
[thread protected code]
with cond:
print '[...]'
### 3rd approach ###
from __future__ import print_function
def safe_print(*args, **kwargs):
global print_lock
with print_lock:
print(*args, **kwargs)
[...]
try:
[thread protected code]
safe_print('[...]')
Except for the first one all kind of have the same performance. The
problem was I placed the acquire/release around the whole code block,
instead of only the print statements.
Thanks a lot! ;)
> For generality the lock should be per file: the above hack uses one
> lock for any file, so that's going to stall overlapping prints to
> different files; inefficient.
>
> There are other things than the above, but at least individual prints will
> never overlap. If you have interleaved prints, show us.
>
> | Is there a 100% sure method to make print thread safe? Can it be fast???
>
> Depends on what you mean by "fast". It will be slower than code
> with no lock; how much would require measurement.
>
> Cheers,
> --
> Cameron Simpson
>
> My own suspicion is that the universe is not only queerer than we suppose,
> but queerer than we *can* suppose.
> - J.B.S. Haldane "On Being the Right Size"
> in the (1928) book "Possible Worlds"
--
http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On Mon, May 20, 2013 at 7:54 PM, Cameron Simpson wrote: > On 20May2013 19:09, Chris Angelico wrote: > | On Mon, May 20, 2013 at 6:35 PM, Cameron Simpson wrote: > | > _lock = Lock() > | > > | > def lprint(*a, **kw): > | > global _lock > | > with _lock: > | > print(*a, **kw) > | > > | > and use lprint() everywhere? > | > | Fun little hack: > | > | def print(*args,print=print,lock=Lock(),**kwargs): > | with lock: > | print(*args,**kwargs) > | > | Question: Is this a cool use or a horrible abuse of the scoping rules? > > I carefully avoided monkey patching print itself:-) > > That's... mad! I can see what the end result is meant to be, but > it looks like a debugging nightmare. Certainly my scoping-fu is too > weak to see at a glance how it works. Hehe. Like I said, could easily be called abuse. Referencing a function's own name in a default has to have one of these interpretations: 1) It's a self-reference, which can be used to guarantee recursion even if the name is rebound 2) It references whatever previously held that name before this def statement. Either would be useful. Python happens to follow #2; though I can't point to any piece of specification that mandates that, so all I can really say is that CPython 3.3 appears to follow #2. But both interpretations make sense, and both would be of use, and use of either could be called abusive of the rules. Figure that out. :) The second defaulted argument (lock=Lock()), of course, is a common idiom. No abuse there, that's pretty Pythonic. This same sort of code could be done as a decorator: def serialize(fn): lock=Lock() def locked(*args,**kw): with lock: fn(*args,**kw) return locked print=serialize(print) Spelled like this, it's obvious that the argument to serialize has to be the previous 'print'. The other notation achieves the same thing, just in a quirkier way :) ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Future standard GUI library
On 5/20/13 1:04 AM, Vito De Tullio wrote: FLTK? (http://www.fltk.org/index.php) FLTK is even uglier than non-themed Tkinter: non-native on every platform. Tkinter wraps native widgets on MacOS and WIndows, but FLTK draws its own widgets everywhere. -- Kevin Walzer Code by Kevin/Mobile Code by Kevin http://www.codebykevin.com http://www.wtmobilesoftware.com -- http://mail.python.org/mailman/listinfo/python-list
RE: How to write fast into a file in python?
Oh well! Just got a flashback from the old times at the 8-bit assembly line. Dirty deeds done dirt cheap! lol > Date: Sun, 19 May 2013 16:44:55 +0100 > From: pyt...@mrabarnett.plus.com > To: python-list@python.org > Subject: Re: How to write fast into a file in python? > > On 19/05/2013 04:53, Carlos Nepomuceno wrote: >> >>> Date: Sat, 18 May 2013 22:41:32 -0400 >>> From: da...@davea.name >>> To: python-list@python.org >>> Subject: Re: How to write fast into a file in python? >>> >>> On 05/18/2013 01:00 PM, Carlos Nepomuceno wrote: Python really writes '\n\r' on Windows. Just check the files. >>> >>> That's backwards. '\r\n' on Windows, IF you omit the b in the mode when >>> creating the file. >> >> Indeed! My mistake just made me find out that Acorn used that inversion on >> Acorn MOS. >> >> According to this[1] (at page 449) the OSNEWL routine outputs '\n\r'. >> >> What the hell those guys were thinking??? :p >> > Doing it that way saved a few bytes. > > Code was something like this: > > FFE3 .OSASCI CMP #&0D > FFE5 BNE OSWRCH > FFE7 .OSNEWL LDA #&0A > FFE9 JSR OSWRCH > FFEC LDA #&0D > FFEE .OSWRCH ... > > This means that the contents of the accumulator would always be > preserved by a call to OSASCI. > >> "OSNEWL >> This call issues an LF CR (line feed, carriage return) to the currently >> selected >> output stream. The routine is entered at &FFE7." >> >> [1] http://regregex.bbcmicro.net/BPlusUserGuide-1.07.pdf >> > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On 5/20/2013 6:09 AM, Chris Angelico wrote: Referencing a function's own name in a default has to have one of these interpretations: 1) It's a self-reference, which can be used to guarantee recursion even if the name is rebound 2) It references whatever previously held that name before this def statement. The meaning must be #2. A def statement is nothing more than a fancy assignment statement. This: def foo(a): return a + 1 is really just the same as: foo = lambda a: a+1 (in fact, they compile to identical bytecode). More complex def's don't have equivalent lambdas, but are still assignments to the name of the function. So your "apparently recursive" print function is no more ambiguous "x = x + 1". The x on the right hand side is the old value of x, the x on the left hand side will be the new value of x. # Each of these updates a name x = x + 1 def print(*args,print=print,lock=Lock(),**kwargs): with lock: print(*args,**kwargs) Of course, if you're going to use that code, a comment might be in order to help the next reader through the trickiness... --Ned. -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On 05/20/2013 03:55 AM, Fábio Santos wrote: My use case was a tight loop processing an image pixel by pixel, or crunching a CSV file. If it only uses local variables (and probably hold a lock before releasing the GIL) it should be safe, no? Are you making function calls, using system libraries, or creating or deleting any objects? All of these use the GIL because they use common data structures shared among all threads. At the lowest level, creating an object requires locked access to the memory manager. Don't forget, the GIL gets used much more for Python internals than it does for the visible stuff. -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20/05/2013 10:07, Frank Millman wrote:
On 20/05/2013 09:55, Chris Angelico wrote:
Is it a requirement that they be able to key in a constraint as a
single string? We have a similar situation in one of the systems at
work, so we divided the input into three(ish) parts: pick a field,
pick an operator (legal operators vary according to field type -
integers can't be compared against regular expressions, timestamps can
use >= and < only), then enter the other operand. Sure, that cuts out
a few possibilities, but you get 99.9%+ of all usage and it's easy to
sanitize.
ChrisA
It is not a requirement, no. I just thought it would be a convenient
short-cut.
I had in mind something similar to your scheme above, so I guess I will
have to bite the bullet and implement it.
Can anyone see anything wrong with the following approach. I have not
definitely decided to do it this way, but I have been experimenting and
it seems to work.
I store the boolean test as a json'd list of 6-part tuples. Each element
of the tuple is a string, defined as follows -
0 - for the first entry in the list, the word 'check' (a placeholder -
it is discarded at evaluation time), for any subsequent entries the word
'and' or 'or'.
1 - left bracket - either '(' or ''.
2 - column name to check - it will be validated on entry.
3 - operator - must be one of '=', '!=', '<', '>', '<=', '>=', 'in',
'is', 'is not'. At evaluation time, '=' is changed to '=='.
4 - value to compare - at evaluation time I call
str(literal_eval(value)) to ensure that it is safe.
5 - right bracket - either ')' or ''.
At evaluation time I loop through the list, construct the boolean test
as a string, and call eval() on it.
Here are some examples -
check = []
check.append(('check', '', 'name', 'in', "('abc', 'xyz')", ''))
check = []
check.append(('check', '', 'value', '>=', '0', ''))
check = []
check.append(('check', '(', 'descr', 'is not', 'None', ''))
check.append(('and', '', 'alt', 'is', 'None', ')'))
check.append(('or', '(', 'descr', 'is', 'None', ''))
check.append(('and', '', 'alt', 'is not', 'None', ')'))
I don't plan to check the logic - I will just display the exception if
it does not evaluate.
It seems safe to me. Can anyone see a problem with it?
Frank
--
http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
=On Mon, May 20, 2013 at 8:46 PM, Ned Batchelder wrote: > On 5/20/2013 6:09 AM, Chris Angelico wrote: >> >> Referencing a function's own name in a default has to have one of >> these interpretations: >> >> 1) It's a self-reference, which can be used to guarantee recursion >> even if the name is rebound >> 2) It references whatever previously held that name before this def >> statement. > > > The meaning must be #2. A def statement is nothing more than a fancy > assignment statement. Sure, but the language could have been specced up somewhat differently, with the same syntax. I was fairly confident that this would be universally true (well, can't do it with 'print' per se in older Pythons, but for others); my statement about CPython 3.3 was just because I hadn't actually hunted down specification proof. > So your "apparently recursive" print function is no more > ambiguous "x = x + 1". The x on the right hand side is the old value of x, > the x on the left hand side will be the new value of x. > > # Each of these updates a name > x = x + 1 > > def print(*args,print=print,lock=Lock(),**kwargs): > with lock: > print(*args,**kwargs) Yeah. The decorator example makes that fairly clear. > Of course, if you're going to use that code, a comment might be in order to > help the next reader through the trickiness... Absolutely!! ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, 20 May 2013 15:26:02 +0200, Frank Millman wrote:
> Can anyone see anything wrong with the following approach. I have not
> definitely decided to do it this way, but I have been experimenting and
> it seems to work.
>
> I store the boolean test as a json'd list of 6-part tuples. Each element
> of the tuple is a string, defined as follows -
>
> 0 - for the first entry in the list, the word 'check' (a placeholder -
> it is discarded at evaluation time), for any subsequent entries the word
> 'and' or 'or'.
>
> 1 - left bracket - either '(' or ''.
>
> 2 - column name to check - it will be validated on entry.
>
> 3 - operator - must be one of '=', '!=', '<', '>', '<=', '>=', 'in',
> 'is', 'is not'. At evaluation time, '=' is changed to '=='.
>
> 4 - value to compare - at evaluation time I call
> str(literal_eval(value)) to ensure that it is safe.
>
> 5 - right bracket - either ')' or ''.
>
> At evaluation time I loop through the list, construct the boolean test
> as a string, and call eval() on it.
[...]
> It seems safe to me. Can anyone see a problem with it?
It seems safe to me too, but then any fool can come up with a system
which they themselves cannot break :-)
I think the real worry is validating the column name. That will be
critical. Personally, I would strongly suggest writing your own mini-
evaluator that walks the list and evaluates it by hand. It isn't as
convenient as just calling eval, but *definitely* safer.
If you do call eval, make sure you supply the globals and locals
arguments. The usual way is:
eval(expression, {'__builtins__': None}, {})
which gives you an empty locals() and a minimal, (mostly) safe globals.
Finally, as a "belt-and-braces" approach, I wouldn't even call eval
directly, but call a thin wrapper that raises an exception if the
expression contains an underscore. Underscores are usually the key to
breaking eval, so refusing to evaluate anything with an underscore raises
the barrier very high.
And even with all those defences, I wouldn't allow untrusted data from
the Internet anywhere near this. Just because I can't break it, doesn't
mean it's safe.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Mon, May 20, 2013 at 11:26 PM, Frank Millman wrote:
> 0 - for the first entry in the list, the word 'check' (a placeholder - it is
> discarded at evaluation time), for any subsequent entries the word 'and' or
> 'or'.
>
> 1 - left bracket - either '(' or ''.
>
> 5 - right bracket - either ')' or ''.
I think what you have is safe, but extremely complicated to work with.
Six separate pieces, and things have to be in the right slots... I
think you've spent too many "complexity points" on the above three
components, and you're getting too little return for them. What
happens if the nesting is mucked up? Could get verrry messy to check.
Combining multiple conditions with a mixture of ands and ors is a
nightmare to try to explain (unless you just point to the Python docs,
which IMO costs you even more complexity points); the only safe option
is to parenthesize everything. The system I pushed for at work (which
was finally approved and implemented a few months ago) is more or less
this: conditions are grouped together into blocks; for each group, you
can choose whether it's "all" or "any" (aka and/or), and you choose
whether the overall result is all-groups or any-group. That still
costs a few complexity points (and, btw, our *actual* implementation
is a bit more complicated than that, but I think we could cut it down
to what I just described here without loss of functionality), but it
gives the bulk of what people will actually want without the
complexities of point-and-click code.
The downside of that sort of system is that it requires a two-level
tree. On the flip side, that's often how people will be thinking about
their conditions anyway (eg using a pair of conditions ">" and "<" to
implement a range check - conceptually it's a single check), so that
won't cost too much.
ChrisA
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Tue, May 21, 2013 at 2:12 AM, Steven D'Aprano wrote: > Personally, I would strongly suggest writing your own mini- > evaluator that walks the list and evaluates it by hand. It isn't as > convenient as just calling eval, but *definitely* safer. Probably faster, too, for what it's worth - eval is pretty expensive. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Harmonic distortion of a input signal
Non sense. The discrete fft algorithm is valid only if the number of data points you transform does correspond to a power of 2 (2**n). Keywords to the problem: apodization, zero filling, convolution product, ... eg. http://en.wikipedia.org/wiki/Convolution jmf -- http://mail.python.org/mailman/listinfo/python-list
Re: Harmonic distortion of a input signal
Am 20.05.13 19:23, schrieb jmfauth: Non sense. Dito. The discrete fft algorithm is valid only if the number of data points you transform does correspond to a power of 2 (2**n). Where did you get this? The DFT is defined for any integer point number the same way. Just if you want to get it fast, you need to worry about the length. For powers of two, there is the classic Cooley-Tukey. But there do exist FFT algorithms for any other length. For example, there is the Winograd transform for a set of small numbers, there is "mixed-radix" to reduce any length which can be factored, and there is finally Bluestein which works for any size, even for a prime. All of the aforementioned algorithms are O(log n) and are implemented in typical FFT packages. All of them should result (up to rounding differences) in the same thing as the naive DFT sum. Therefore, today Keywords to the problem: apodization, zero filling, convolution product, ... Not for a periodic signal of integer length. eg. http://en.wikipedia.org/wiki/Convolution How long do you read this group? Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Harmonic distortion of a input signal
Oops, I thought we were posting to comp.dsp. Nevertheless, I think numpy.fft does mixed-radix (can't check it now) Am 20.05.13 19:50, schrieb Christian Gollwitzer: Am 20.05.13 19:23, schrieb jmfauth: Non sense. Dito. The discrete fft algorithm is valid only if the number of data points you transform does correspond to a power of 2 (2**n). Where did you get this? The DFT is defined for any integer point number the same way. Just if you want to get it fast, you need to worry about the length. For powers of two, there is the classic Cooley-Tukey. But there do exist FFT algorithms for any other length. For example, there is the Winograd transform for a set of small numbers, there is "mixed-radix" to reduce any length which can be factored, and there is finally Bluestein which works for any size, even for a prime. All of the aforementioned algorithms are O(log n) and are implemented in typical FFT packages. All of them should result (up to rounding differences) in the same thing as the naive DFT sum. Therefore, today Keywords to the problem: apodization, zero filling, convolution product, ... Not for a periodic signal of integer length. eg. http://en.wikipedia.org/wiki/Convolution How long do you read this group? Christian -- http://mail.python.org/mailman/listinfo/python-list
pip and different branches?
We use github and we work on many different branches at the same time. The problem is that we have >5 repos now, and for each repo we might have the same branches on all of them. Now we use pip and install requirements such as: git+ssh://g...@github.com/repo.git@dev Now the problem is that the requirements file are also under revision control, and constantly we end up in the situation that when we merge branches the branch settings get messed up, because we forget to change them. I was looking for a solution for this that would allow me to: - use the branch of the "main" repo for all the dependencies - fallback on master if that branch doesn't exist I thought about a few options: 1. create a wrapper for PIP that manipulates the requirement file, that now would become templates. In this way I would have to know however if a branch exist or not, and I didn't find a way to do that without cloning the repo. 2. modify PIP to not fail when checking out a non existing branch, so that if it's not found it falls back on master automatically. 3. use some git magic hooks but I'm not sure what exactly 4. stop using virtualenv + pip and use something smarter that handles this. Any suggestions? Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
I didn't know that. On 20 May 2013 12:10, "Dave Angel" wrote: > Are you making function calls, using system libraries, or creating or deleting any objects? All of these use the GIL because they use common data structures shared among all threads. At the lowest level, creating an object requires locked access to the memory manager. > > > Don't forget, the GIL gets used much more for Python internals than it does for the visible stuff. I did not know that. It's both interesting and somehow obvious, although I didn't know it yet. -- http://mail.python.org/mailman/listinfo/python-list
Re: What was the project that made you feel skilled in Python?
TBH, I think that the first thing that I did that made me feel that I could hold my own was when I had my first (and only thus far) patch accepted into the stdlib. To me, there's a /big/ difference between throwing together a package that a few people may find useful and putting a patch together that's accepted by a particular module expert and committed to the stdlib. To intermediate learners, I would strongly advocate getting their hands dirty with some part of the stdlib. Really (imho), there's really no better place to learn. Yes, in whole, it's a large project, but there are quite a few small(er) modules that, once their environment is set up, are self-contained and easy to follow along. Even if they don't get anything committed, learning from Python experts is far more useful than any other path that I've personally taken. Having said that, another great learning experience for me was when I wrote my OAuth 2.0 client (https://github.com/demianbrecht/sanction) and brought the initial implementation (460'ish LoC) to 66 LoC (pre-2/3 support). In part, this was due to taking a different design approach, but it was also in part due to taking advantage of Python idioms rather than simply using approaches that I had used in other languages. On Sun, May 19, 2013 at 4:30 AM, Ned Batchelder wrote: > Hi all, I'm trying to come up with more project ideas for intermediate > learners, somewhat along the lines of > http://bit.ly/intermediate-python-projects . > > So here's a question for people who remember coming up from beginner: as you > moved from exercises like those in Learn Python the Hard Way, up to your own > self-guided work on small projects, what project were you working on that > made you feel independent and skilled? What program first felt like your > own work rather than an exercise the teacher had assigned? > > I don't want anything too large, but big enough that there's room for > design, and multiple approaches, etc. > > Thanks in advance! > > --Ned. > -- > http://mail.python.org/mailman/listinfo/python-list -- Demian Brecht http://demianbrecht.github.com -- http://mail.python.org/mailman/listinfo/python-list
Re: What was the project that made you feel skilled in Python?
Hi Demian, Can I ask what you mean by working through the stdlib? As in writing code pieces utilizing each module from the stdlib? Also, you're talking about "patches" in the stdlib? Is there a separate library of patches? Forgive me if I'm google-failing hard over here. -- http://mail.python.org/mailman/listinfo/python-list
Re: What was the project that made you feel skilled in Python?
On 5/20/2013 3:36 PM, Thomas Murphy wrote: talking about "patches" in the stdlib? Is there a separate library of patches? http://bugs.python.org http://docs.python.org/devguide/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Future standard GUI library
On 2013-05-20, Terry Jan Reedy wrote: > On 5/20/2013 1:04 AM, Vito De Tullio wrote: >> Terry Jan Reedy wrote: >> Do you think tkinter is going to be the standard python built-in gui solution as long as python exists? >>> >>> AT the moment, there is nothing really comparable that is a realistic >>> candidate to replace tkinter. >> >> FLTK? (http://www.fltk.org/index.php) > > tkinter is the Python wrapper of the tk library, just as wxpython is > the python wrapper of the wx library. I do not see a py-fltk wrapper. http://pyfltk.sourceforge.net/ -- Grant Edwards grant.b.edwardsYow! Is it 1974? What's at for SUPPER? Can I spend gmail.commy COLLEGE FUND in one wild afternoon?? -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
Chris Angelico於 2013年5月20日星期一UTC+8下午5時09分13秒寫道： > On Mon, May 20, 2013 at 6:35 PM, Cameron Simpson wrote: > > > _lock = Lock() > > > > > > def lprint(*a, **kw): > > > global _lock > > > with _lock: > > > print(*a, **kw) > > > > > > and use lprint() everywhere? > > > > Fun little hack: > > > > def print(*args,print=print,lock=Lock(),**kwargs): > > with lock: > > print(*args,**kwargs) > > > > Question: Is this a cool use or a horrible abuse of the scoping rules? > > > > ChrisA OK, if the python interpreter has a global hiden print out buffer of ,say, 2to 16 K bytes, and all string print functions just construct the output string from the format to this string in an efficient low level way, then the next question would be that whether the uses can use functions in this low level buffer for other string formatting jobs. -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On Tue, May 21, 2013 at 11:44 AM, 8 Dihedral wrote: > OK, if the python interpreter has a global hiden print out > buffer of ,say, 2to 16 K bytes, and all string print functions > just construct the output string from the format to this string > in an efficient low level way, then the next question > would be that whether the uses can use functions in this > low level buffer for other string formatting jobs. You remind me of George. http://www.chroniclesofgeorge.com/ Both make great reading when I'm at work and poking around with random stuff in our .SQL file of carefully constructed mayhem. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On Monday, May 20, 2013 2:05:48 AM UTC-5, Frank Millman wrote:
> Hi all
>
>
>
> I am trying to emulate a SQL check constraint in Python. Quoting from
>
> the PostgreSQL docs, "A check constraint is the most generic constraint
>
> type. It allows you to specify that the value in a certain column must
>
> satisfy a Boolean (truth-value) expression."
>
>
>
> The problem is that I want to store the constraint as a string, and I
>
> was hoping to use ast.literal_eval to evaluate it, but it does not work.
>
>
>
> >>> x = 'abc'
>
> >>> x in ('abc', xyz')
>
> True
>
> >>> b = "x in ('abc', 'xyz')"
>
> >>> eval(b)
>
> True
>
> >>> from ast import literal_eval
>
> >>> literal_eval(b)
>
> ValueError: malformed node or string: <_ast.Compare object at ...>
>
>
>
> Is there a safe way to do what I want? I am using python 3.3.
>
>
>
> Thanks
>
>
>
> Frank Millman
You might find the asteval module (https://pypi.python.org/pypi/asteval)
useful. It provides a relatively safe "eval", for example:
>>> import asteval
>>> a = asteval.Interpreter()
>>> a.eval('x = "abc"')
>>> a.eval('x in ("abc", "xyz")')
True
>>> a.eval('import os')
NotImplementedError
import os
'Import' not supported
>>> a.eval('__import__("os")')
NameError
__import__("os")
name '__import__' is not defined
This works by maintaining an internal namespace (a flat dictionary), and
walking the AST generated for the expression. It supports most Python syntax,
including if, for, while, and try/except blocks, and function definitions, and
with the notable exceptions of eval, exec, class, lambda, yield, and import.
This requires Python2.6 and higher, and does work with Python3.3.
Of course, it is not guaranteed to be completely safe, but it does disallow
imports, which seems like the biggest vulnerability concern listed here.
Currently, there is no explicit protection against long-running calculations
for denial of service attacks. If you're exposing an SQL database to
user-generated code, that may be worth considering.
Cheers,
--Matt Newville
--
http://mail.python.org/mailman/listinfo/python-list
RE: Please help with Threading
sys.stdout.write() does not suffer from the newlines mess up when printing from many threads, like print statement does. The only usage difference, AFAIK, is to add '\n' at the end of the string. It's faster and thread safe (really?) by default. BTW, why I didn't find the source code to the sys module in the 'Lib' directory? > Date: Tue, 21 May 2013 11:50:17 +1000 > Subject: Re: Please help with Threading > From: ros...@gmail.com > To: python-list@python.org > > On Tue, May 21, 2013 at 11:44 AM, 8 Dihedral > wrote: >> OK, if the python interpreter has a global hiden print out >> buffer of ,say, 2to 16 K bytes, and all string print functions >> just construct the output string from the format to this string >> in an efficient low level way, then the next question >> would be that whether the uses can use functions in this >> low level buffer for other string formatting jobs. > > You remind me of George. > http://www.chroniclesofgeorge.com/ > > Both make great reading when I'm at work and poking around with random > stuff in our .SQL file of carefully constructed mayhem. > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run a python script twice randomly in a day?
On 20May2013 15:05, Avnesh Shakya wrote: | Thanks a lot. No worries, but ... AGAIN: - please DO NOT top post. Post below, trimming the quoted material. - please POST TO THE LIST, not just to me. This is a public discussion. Now... | I did something. | I have created test.sh file in which i put- | | #!/bin/bash | cd /home/avin/cronJob | python try.py Ok, good. Some minor remarks: Personally, I always use: #!/bin/sh instead of requiring bash. All UNIX systems have sh, bash is only common. And even when present, it may not be in /bin. /bin/sh is always there, and unless you're doing something quite unusual, it works just fine. | then i went on terminal - | and run crontab -e | and wrote- | */2 * * * * bash /home/avin/cronJob/test.sh | and saved it. IIRC, this runs every two minutes. Good for testing, but not your original spec. Also, if you make the shell script (test.sh) executable you do not need to specify the interpreter. Treat your script like any other command! So: chmod +rx /home/avin/cronJob/test.sh and then your cron line can look like this: */2 * * * * /home/avin/cronJob/test.sh Also, treat your script the same way as your shell script, start it with a #! like this: #!/usr/bin/python Make it executable: chmod +rx /home/avin/cronJob/try.py and then you don't need to say "python" in your shell script: ./try.py (You need the ./ because the current directory is not in your command search path ($PATH).) | It's working fine. | but when I m using like | | import random | a = random.randrange(0, 59) | */a * * * * bash /home/avin/cronJob/test.sh | then it's showing error becose of varable 'a', so now how can i take | variable? I take it that this is your python program intended to schedule the two randomly timed runs? As a start, it must all be python. The first two lines are. The third line is a crontab line. So as a start, you need to look more like this: #!/usr/bin/python import random a = random.randrange(0, 59) cronline = '*/%d * * * * /home/avin/cronJob/test.sh' % (a,) print(cronline) At least then you can see the cron line you're making. It still does not add it to a cron job. Some remarks: - randrange() is like other python ranges: it does not include the end value. So your call picks a number from 0..58, not 0..59. Say randrange(0,60). Think "start, length". - My recollection is that you wanted to run a script twice a day at random times. Your cron line doesn't do that. - If you're picking random run times you want to schedule a once-off job for each to run at a particular times. Cron schedules repeating jobs. To run at a particular time you want an "at" job. - You need to do one of two things in the pick-a-time script: - pick a time, then sleep until that time and then directly invoke the try.py script or - pick a time, then use the "at" command to schedule the try.py (or test.sh) script. The first approach would look a bit like this (totally untested): #!/usr/bin/python import random import subporcess import time # choose range time in the next 24 hours when = random.randrange(0, 24 * 3600) # sleep that many seconds time.sleep(when) subprocess.call(['/home/avin/cronJob/test.sh']) For two runs, pick two times. Swap them into order. Sleep twice, once until the first time and then once until the second time. Etc. The second approach (using "at") would not sleep. instead, compute (using the datetime module) the date and time each job should run, and invoke "at" using the subprocess module, piping the text "/home/avin/cronJob/test.sh\n" to it. Cheers, -- Cameron Simpson On a related topic, has anyone looked at doing a clean-room copy of CSS a la RC2 and RC4 a few years back? I know one or two people have looked at this in an informal manner, but we couldn't find anyone who hadn't already seen the DeCSS code to act as the clean person (it says a lot for the status of their "trade secret" that we couldn't actually find anyone who didn't already know it). - Peter Gutmann -- http://mail.python.org/mailman/listinfo/python-list
RE: Please help with Threading
> On Tue, May 21, 2013 at 11:44 AM, 8 Dihedral > wrote: >> OK, if the python interpreter has a global hiden print out >> buffer of ,say, 2to 16 K bytes, and all string print functions >> just construct the output string from the format to this string >> in an efficient low level way, then the next question >> would be that whether the uses can use functions in this >> low level buffer for other string formatting jobs. > > You remind me of George. > http://www.chroniclesofgeorge.com/ > > Both make great reading when I'm at work and poking around with random > stuff in our .SQL file of carefully constructed mayhem. > > ChrisA lol I need more cowbell!!! Please!!! lol -- http://mail.python.org/mailman/listinfo/python-list
Re: How to run a python script twice randomly in a day?
Thanks a lot. I got it. On Tue, May 21, 2013 at 6:42 AM, Cameron Simpson wrote: > On 20May2013 15:05, Avnesh Shakya wrote: > | Thanks a lot. > > No worries, but ... > > AGAIN: > - please DO NOT top post. Post below, trimming the quoted material. > - please POST TO THE LIST, not just to me. This is a public discussion. > > Now... > > | I did something. > | I have created test.sh file in which i put- > | > | #!/bin/bash > | cd /home/avin/cronJob > | python try.py > > Ok, good. Some minor remarks: > > Personally, I always use: > > #!/bin/sh > > instead of requiring bash. All UNIX systems have sh, bash is only > common. And even when present, it may not be in /bin. /bin/sh is > always there, and unless you're doing something quite unusual, it > works just fine. > > | then i went on terminal - > | and run crontab -e > | and wrote- > | */2 * * * * bash /home/avin/cronJob/test.sh > | and saved it. > > IIRC, this runs every two minutes. Good for testing, but not your original > spec. > > Also, if you make the shell script (test.sh) executable you do not > need to specify the interpreter. Treat your script like any other > command! So: > > chmod +rx /home/avin/cronJob/test.sh > > and then your cron line can look like this: > > */2 * * * * /home/avin/cronJob/test.sh > > Also, treat your script the same way as your shell script, start > it with a #! like this: > > #!/usr/bin/python > > Make it executable: > > chmod +rx /home/avin/cronJob/try.py > > and then you don't need to say "python" in your shell script: > > ./try.py > > (You need the ./ because the current directory is not in your command > search path ($PATH).) > > | It's working fine. > | but when I m using like > | > | import random > | a = random.randrange(0, 59) > | */a * * * * bash /home/avin/cronJob/test.sh > | then it's showing error becose of varable 'a', so now how can i take > | variable? > > I take it that this is your python program intended to schedule the two > randomly timed runs? > > As a start, it must all be python. The first two lines are. The third line > is > a crontab line. > > So as a start, you need to look more like this: > > #!/usr/bin/python > import random > a = random.randrange(0, 59) > cronline = '*/%d * * * * /home/avin/cronJob/test.sh' % (a,) > print(cronline) > > At least then you can see the cron line you're making. It still > does not add it to a cron job. > > Some remarks: > > - randrange() is like other python ranges: it does not include the end > value. > So your call picks a number from 0..58, not 0..59. > Say randrange(0,60). Think "start, length". > > - My recollection is that you wanted to run a script twice a day at random > times. > Your cron line doesn't do that. > > - If you're picking random run times you want to schedule a once-off > job for each to run at a particular times. Cron schedules repeating > jobs. To run at a particular time you want an "at" job. > > - You need to do one of two things in the pick-a-time script: > - pick a time, then sleep until that time and then directly > invoke the try.py script > or > - pick a time, then use the "at" command to schedule the try.py > (or test.sh) script. > > The first approach would look a bit like this (totally untested): > > #!/usr/bin/python > import random > import subporcess > import time > # choose range time in the next 24 hours > when = random.randrange(0, 24 * 3600) > # sleep that many seconds > time.sleep(when) > subprocess.call(['/home/avin/cronJob/test.sh']) > > For two runs, pick two times. Swap them into order. Sleep twice, > once until the first time and then once until the second time. Etc. > > The second approach (using "at") would not sleep. instead, compute > (using the datetime module) the date and time each job should run, > and invoke "at" using the subprocess module, piping the text > "/home/avin/cronJob/test.sh\n" to it. > > Cheers, > -- > Cameron Simpson > > On a related topic, has anyone looked at doing a clean-room copy of CSS > a la RC2 and RC4 a few years back? I know one or two people have > looked at this in an informal manner, but we couldn't find anyone who > hadn't already seen the DeCSS code to act as the clean person (it says > a lot for the status of their "trade secret" that we couldn't actually > find anyone who didn't already know it). > - Peter Gutmann > -- http://mail.python.org/mailman/listinfo/python-list
Re: Please help with Threading
On Tue, 21 May 2013 05:53:46 +0300, Carlos Nepomuceno wrote: > BTW, why I didn't find the source code to the sys module in the 'Lib' > directory? Because sys is a built-in module. It is embedded in the Python interpreter. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
PEP 378: Format Specifier for Thousands Separator
Is there a way to format integers with thousands separator (digit grouping)
like the format specifier of str.format()?
I'm currently using the following:
>>> sys.stdout.write('Number = %s\n' % '{:,.0f}'.format(x))
Number = 12,345
'x' is unsigned integer so it's like using a sledgehammer to crack a nut!
I'd like to have something like:
sys.stdout.write('Number = %,u\n' % x)
Is that possible? How can I do it if not already available?
--
http://mail.python.org/mailman/listinfo/python-list
Re: PEP 378: Format Specifier for Thousands Separator
In article ,
Carlos Nepomuceno wrote:
> Is there a way to format integers with thousands separator (digit grouping)
> like the format specifier of str.format()?>
> I'm currently using the following:>
> >>> sys.stdout.write('Number = %s\n' % '{:,.0f}'.format(x))
> Number = 12,345>
> 'x' is unsigned integer so it's like using a sledgehammer to crack a nut!>
> I'd like to have something like:
> sys.stdout.write('Number = %,u\n' % x)
> Is that possible? How can I do it if not already available?
For Python 3.2+ or 2.7, why not just:
>>> print('Number = {:,}'.format(x))
Number = 12,345
--
Ned Deily,
n...@acm.org
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 21/05/2013 04:39, matt.newvi...@gmail.com wrote:
You might find the asteval module (https://pypi.python.org/pypi/asteval) useful. It
provides a relatively safe "eval", for example:
>>> import asteval
>>> a = asteval.Interpreter()
>>> a.eval('x = "abc"')
>>> a.eval('x in ("abc", "xyz")')
True
>>> a.eval('import os')
NotImplementedError
import os
'Import' not supported
>>> a.eval('__import__("os")')
NameError
__import__("os")
name '__import__' is not defined
This works by maintaining an internal namespace (a flat dictionary), and
walking the AST generated for the expression. It supports most Python syntax,
including if, for, while, and try/except blocks, and function definitions, and
with the notable exceptions of eval, exec, class, lambda, yield, and import.
This requires Python2.6 and higher, and does work with Python3.3.
Of course, it is not guaranteed to be completely safe, but it does disallow
imports, which seems like the biggest vulnerability concern listed here.
Currently, there is no explicit protection against long-running calculations
for denial of service attacks. If you're exposing an SQL database to
user-generated code, that may be worth considering.
Thanks for this, Matt. I will definitely look into it.
Frank
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20/05/2013 18:12, Steven D'Aprano wrote:
On Mon, 20 May 2013 15:26:02 +0200, Frank Millman wrote:
Can anyone see anything wrong with the following approach. I have not
definitely decided to do it this way, but I have been experimenting and
it seems to work.
[...]
It seems safe to me too, but then any fool can come up with a system
which they themselves cannot break :-)
Thanks for the detailed response.
I think the real worry is validating the column name. That will be
critical.
I would not pass the actual column name to eval(), I would use it to
retrieve a value from a data object and pass that to eval(). However,
then your point becomes 'validating the value retrieved'. I had not
thought about that. I will investigate further.
Personally, I would strongly suggest writing your own mini-
evaluator that walks the list and evaluates it by hand. It isn't as
convenient as just calling eval, but *definitely* safer.
I am not sure I can wrap my mind around mixed 'and's, 'or's, and brackets.
[Thinking aloud]
Maybe I can manually reduce each internal test to a True or False,
substitute them in the list, and then call eval() on the result.
eval('(True and False) or (False or True)')
I will experiment with that.
If you do call eval, make sure you supply the globals and locals
arguments. The usual way is:
eval(expression, {'__builtins__': None}, {})
which gives you an empty locals() and a minimal, (mostly) safe globals.
Thanks - I did not know about that.
Finally, as a "belt-and-braces" approach, I wouldn't even call eval
directly, but call a thin wrapper that raises an exception if the
expression contains an underscore. Underscores are usually the key to
breaking eval, so refusing to evaluate anything with an underscore raises
the barrier very high.
And even with all those defences, I wouldn't allow untrusted data from
the Internet anywhere near this. Just because I can't break it, doesn't
mean it's safe.
All good advice - much appreciated.
Frank
--
http://mail.python.org/mailman/listinfo/python-list
Re: Question about ast.literal_eval
On 20/05/2013 18:13, Chris Angelico wrote:
On Mon, May 20, 2013 at 11:26 PM, Frank Millman wrote:
0 - for the first entry in the list, the word 'check' (a placeholder - it is
discarded at evaluation time), for any subsequent entries the word 'and' or
'or'.
1 - left bracket - either '(' or ''.
5 - right bracket - either ')' or ''.
I think what you have is safe, but extremely complicated to work with.
Six separate pieces, and things have to be in the right slots... I
think you've spent too many "complexity points" on the above three
components, and you're getting too little return for them. What
happens if the nesting is mucked up? Could get verrry messy to check.
Combining multiple conditions with a mixture of ands and ors is a
nightmare to try to explain (unless you just point to the Python docs,
which IMO costs you even more complexity points); the only safe option
is to parenthesize everything. The system I pushed for at work (which
was finally approved and implemented a few months ago) is more or less
this: conditions are grouped together into blocks; for each group, you
can choose whether it's "all" or "any" (aka and/or), and you choose
whether the overall result is all-groups or any-group. That still
costs a few complexity points (and, btw, our *actual* implementation
is a bit more complicated than that, but I think we could cut it down
to what I just described here without loss of functionality), but it
gives the bulk of what people will actually want without the
complexities of point-and-click code.
The downside of that sort of system is that it requires a two-level
tree. On the flip side, that's often how people will be thinking about
their conditions anyway (eg using a pair of conditions ">" and "<" to
implement a range check - conceptually it's a single check), so that
won't cost too much.
You may be right, Chris, but I don't think my approach is all that bad.
The vast majority of tests will be simple - either a single line, or two
lines for a range check, with no brackets at all.
If the requirement is more complicated than that, well, I don't think
the complication can be avoided, and at least this approach gives full
control.
FWIW, I use the same approach to allow users to construct their own
WHERE clauses in custom views. Again, the vast majority are simple, but
there are times when it can get complicated.
The proof of the pudding will be when I try to get ordinary users to get
their own hands dirty - I am not there yet. If I ever get this released,
the business model will be free software, but support will be charged
for. So if a user gets out of his/her depth, there will be assistance
available.
Time will tell who is right ... ;-)
Frank
--
http://mail.python.org/mailman/listinfo/python-list
