Re: Writing list of dictionaries to CSV
Op Wednesday 6 May 2015 07:32 CEST schreef Kashif Rana: > thanks for the feedback. I think its problem with excel itself, > showing wrong value. Because when I opened the csv file in text > editor, I can see correct value but opening in excel showing wrong > value. What I can do to see correct in excel as well. A Python mailing list is not the right place to ask that. You can better use a Microsoft (office) mailing list. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On Wednesday 06 May 2015 15:58, Ian Kelly wrote: > On Tue, May 5, 2015 at 7:27 PM, Steven D'Aprano > wrote: >> Only the minimum is statistically useful. > > I disagree. The minimum tells you how fast the code *can* run, under > optimal circumstances. The mean tells you how fast it *realistically* > runs, under typical load. Both can be useful to measure. Er, not even close. Running code using timeit is in no way the same as running code "for real" under realistic circumstances. The fact that you are running the function or code snippet in isolation, in its own scope, via exec, rather than as part of some larger script or application, should be a hint. timeit itself has overhead, so you cannot measure the time taken by the operation alone, you can only measure the time taken by the operation within the timeit environment. We have no reason to think that the distribution of noise under timeit will be even vaguely similar to the noise when running in production. The purpose of timeit is to compare individual algorithms, in as close as possible to an equal footing with as little noise as possible. If you want to profile code used in a realistic application, use a profiler, not timeit. And even that doesn't tell you how fast the code would be alone, because the profiler adds overhead. Besides, "typical load" is a myth -- there is no such thing. A high-end Windows web server getting ten thousand hits a minute, a virtual machine starved for RAM, a Mac laptop, a Linux server idling away with a load of 0.1 all day... any of those machines could run your code. How can you *possibly* say what is typical? The very idea is absurd. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Encrypt python files
On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: > Hi, > > What are the ways to encrypt python files? No, I just want to hide the scripts from others. -- https://mail.python.org/mailman/listinfo/python-list
DRM is self-defeating (was: Encrypt python files)
Palpandi writes: > On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: > > What are the ways to encrypt python files? > > No, I just want to hide the scripts from others. Which others? You can hide the scripts from them by never showing those others the scripts. If you don't trust a recipient, don't let them receive the file. I suspect you are asking “how can I distribute Python modules to people without those people being able to read them”? You can't, because executing the file requires reading its contents. Anyone who has a machine that can execute the file has a machine that must, by necessity, read its contents. This is the dilemma of those who think they want Digital Restrictions Management (DRM): attempting to treat recipients as untrustworthy, while still telling them they can use the files. If you a person is in possession of the file you're trying to restrict, on a machine they possess, with a key needed to unlock the content, then that person is in possession of everything needed to unlock the content. On the other hand, if one of those (e.g. the key to unlock the content) is missing, then the file is useless for whatever you're saying the recipient can do with it. The only feasible solution is to distribute files only to those recipients you want to have them, and can trust to do with them as you ask. If you don't trust a recipient, don't hand the files to them. On the other hand, if you're asking something else, you will need to be much more explicit. You have not helped us understand what you want thus far. -- \“This sentence contradicts itself — no actually it doesn't.” | `\ —Douglas Hofstadter | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: DRM is self-defeating (was: Encrypt python files)
On Wed, May 6, 2015 at 5:41 PM, Ben Finney wrote: > The only feasible solution is to distribute files only to those > recipients you want to have them, and can trust to do with them as you > ask. If you don't trust a recipient, don't hand the files to them. in today's world, that basically gives two good options: 1) Distribute your code far and wide, let every man and his dog clone your source code, and license it so they're allowed to; or 2) Host your code on a server that people don't get access to, and have them interact with it remotely. Either way works really well. You can have closed-source software that people never directly see, yet benefit from (and potentially pay you), or you can let people see everything they're running. The problem only comes when you try to hybridize. (Aside: Most people carry mobile phones they don't truly own/control, so they're restricted by what the device manufacturer offers them. Despite looking superficially like the first case, it's really more like the second.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: DRM is self-defeating (was: Encrypt python files)
- On Wed, May 6, 2015 9:41 AM CEST Ben Finney wrote: >Palpandi writes: > >> On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: > >> > What are the ways to encrypt python files? >> >> No, I just want to hide the scripts from others. > >Which others? You can hide the scripts from them by never showing those >others the scripts. If you don't trust a recipient, don't let them >receive the file. >The only feasible solution is to distribute files only to those >recipients you want to have them, and can trust to do with them as you >ask. If you don't trust a recipient, don't hand the files to them. Can tools like py2exe or cx_freeze also be used to obfuscate python source code? -- https://mail.python.org/mailman/listinfo/python-list
Json Comaprision
Hi All: I wanted to compare two json files ignoring few of the keys in the json files. Can anybody suggest me few things? Thanks, Siva -- https://mail.python.org/mailman/listinfo/python-list
Re: Encrypt python files
On Wednesday 06 May 2015 17:23, Palpandi wrote: > On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: >> Hi, >> >> What are the ways to encrypt python files? > > No, I just want to hide the scripts from others. Why, are you ashamed of your code? Python is free, open source software. Hiding the code from others is not a priority for the developers of the language. Besides, you can't hide the code unless you only operate the application via a web service, or similar. As soon as you give people a copy of the code, whether it is binary code or source code, they have a copy of it and can look at it and work out how what it does. If "hiding the code" was good for security, why are there so many viruses and spybots and worms and other malware for Windows? No, as far as I am concerned, trying to hide the code is a waste of time with Python. But if you absolutely must, you can distribute the .pyc files instead of the .py files, and that will discourage casual tinkerers from poking around in the program. The .pyc file is compiled to byte-code rather than source code, so it's not readable without running it through a decompiler. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: DRM is self-defeating (was: Encrypt python files)
On Wed, May 6, 2015 at 6:24 PM, Albert-Jan Roskam via Python-list wrote: >>The only feasible solution is to distribute files only to those >>recipients you want to have them, and can trust to do with them as you >>ask. If you don't trust a recipient, don't hand the files to them. > > Can tools like py2exe or cx_freeze also be used to obfuscate python source > code? But barely. At very worst, you suppress your comments and variable names, but all your actual functionality is all there still - it has to be, or it won't run. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Encrypt python files
- On Wed, May 6, 2015 11:04 AM CEST Steven D'Aprano wrote: >On Wednesday 06 May 2015 17:23, Palpandi wrote: > >> On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: >> Hi, >> >> What are the ways to encrypt python files? >> >> No, I just want to hide the scripts from others. > >Why, are you ashamed of your code? > >Python is free, open source software. Hiding the code from others is not a >priority for the developers of the language. Besides, you can't hide the >code unless you only operate the application via a web service, or similar. >As soon as you give people a copy of the code, whether it is binary code or >source code, they have a copy of it and can look at it and work out how what >it does. > >If "hiding the code" was good for security, why are there so many viruses >and spybots and worms and other malware for Windows? > >No, as far as I am concerned, trying to hide the code is a waste of time >with Python. But if you absolutely must, you can distribute the .pyc files >instead of the .py files, and that will discourage casual tinkerers from >poking around in the program. The .pyc file is compiled to byte-code rather >than source code, so it's not readable without running it through a >decompiler. I used the marshal module before as a faster alternative to shelve (I marshalled a huge dictionary). I always understood marshal files require the *exact* same interpreter version (incl. built number). Do the same limitations apply for .pyc files? Isn't it the same format? We have a VCS with an option to use private repos. I am always wary of users who want to use this feature. Why would you not want to share your code with your colleagues? Embarrassed about unreadable crappy code, perhaps? -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
Steven D'Aprano wrote: The first one just does a name lookup and then throws the result away. The second one looks up names a and b, then adds them together, throwing away the result. Actually, it's worse than that. :-) It's possible for a to have an __add__ method with a side effect, although that would be evil. It's also possible for merely looking up a name to have a side effect, if it's done in the right context -- but that would be even more evil. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
BartC wrote:
So why pretend that ++ and -- don't exist?
Probably because Python would gain very little from
having them.
Main uses of ++ in C are things like integer for
loops:
for (i = 0; i < 10; i++) {...
and stepping through arrays:
a[i++] = b[j++];
Python code usually operates at a higher level than
that.
--
Greg
--
https://mail.python.org/mailman/listinfo/python-list
Re: Encrypt python files
On Wed, 6 May 2015 07:45 pm, Albert-Jan Roskam wrote: > I used the marshal module before as a faster alternative to shelve (I > marshalled a huge dictionary). I always understood marshal files require > the *exact* same interpreter version (incl. built number). Do the same > limitations apply for .pyc files? Isn't it the same format? marshal is currently up to version 4 of the (otherwise undocumented) file format. Obviously that's much less than the number of minor versions of Python, so, no, marshal does not change every release. https://docs.python.org/3/library/marshal.html .pyc files, on the other hand, have a magic number which changes for each minor release of Python. I think that, in principle, it could change in a point release (say, between 3.4.1 and 3.4.2) but in practice I don't believe that has ever happened. I think it is safe to assume that pyc files will be portable across any minor release (e.g. 3.4.x for any x) but not across changes to the minor or major release. I don't think that is an outright promise, but it is a strong convention. And of course, other implementations may not even use .pyc files, if they don't use the same sort of byte code. E.g. Jython compiles to JVM byte code: foo.py foo$py.class Nick Coghlan has a very good blog post about the uses of pyc only distributed software: http://www.curiousefficiency.org/posts/2011/04/benefits-and-limitations-of-pyc-only.html And Ned Batchelder has a good discussion of their internals: http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Json Comaprision
On 05/05/2015 20:55, pra devOPS wrote: Hi All: I wanted to compare two json files ignoring few of the keys in the json files. Can anybody suggest me few things? Thanks, Siva https://docs.python.org/3/library/json.html#module-json -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
Op 05-05-15 om 18:24 schreef Rustom Mody: > Yeah I happen to me in that minuscule minority that regards '= denotes > assignment' a bigger mistake than ++ Nice to know I'm not alone. I Especially think it is a mistake, because it is then used as a reason for not allowing something like if a = b - 1: arguing it would lead to some difficult bugs. Which in my mind is arguing backwards. Either you think an assigment in a condition is useful or harmful. In the first case you then look for an assignment token or assignment syntax that is not that likely to lead to difficult to discover bugs instead of letting a possible misleading token or syntax prevent you from implementing something useful. In the second case you just state why you think an assignment in a condition is harmful. No need to hide behind awkward syntax. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On 06/05/2015 12:19, Gregory Ewing wrote:
BartC wrote:
So why pretend that ++ and -- don't exist?
Probably because Python would gain very little from
having them.
Main uses of ++ in C are things like integer for
loops:
for (i = 0; i < 10; i++) {...
and stepping through arrays:
a[i++] = b[j++];
Python code usually operates at a higher level than
that.
I think even in Python it is sometimes necessary to increment things (as
the OP did).
But I had in mind not implementing ++ and --, but detecting them and
issuing a warning, so forcing someone to type "+ +" or to use
parentheses, which is unlikely to be much of an imposition as how often
are two unary pluses going to be used together?
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On Wednesday, May 6, 2015 at 6:09:08 PM UTC+5:30, Antoon Pardon wrote: > Op 05-05-15 om 18:24 schreef Rustom Mody: > > > Yeah I happen to me in that minuscule minority that regards '= denotes > > assignment' a bigger mistake than ++ > > Nice to know I'm not alone. I Especially think it is a mistake, because > it is then used as a reason for not allowing something like > > if a = b - 1: > > arguing it would lead to some difficult bugs. > > Which in my mind is arguing backwards. Either you think an assigment > in a condition is useful or harmful. In the first case you then look > for an assignment token or assignment syntax that is not that likely > to lead to difficult to discover bugs instead of letting a possible > misleading token or syntax prevent you from implementing something > useful. > > In the second case you just state why you think an assignment in a > condition is harmful. No need to hide behind awkward syntax. Nice to know we agree though I am not sure I agree with your agreement :-) In APL assignment is ← goto is → [and of course equality and negated equality are = and ≠ ] But if you've seen hair-raising APL one-liners with multiple ← and even → stuffed in... No on second thoughts we probably agree... I am just in the second camp: assignment in conditions is trouble; no need for syntax to argue that -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On Mon, May 4, 2015, at 18:02, BartC wrote: > (I think I would have picked up "++" and "--" as special tokens even if > increment/decrement ops weren't supported. Just because they would > likely cause errors through misunderstanding.) There's precedent for not doing this in C itself - even though "=+" (from very early versions of C, also =-, =*, =&) no longer acts as an add-in-place operator, it's not recognized as a special token to prevent errors either. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On Wed, May 6, 2015 at 11:11 PM, wrote: > On Mon, May 4, 2015, at 18:02, BartC wrote: >> (I think I would have picked up "++" and "--" as special tokens even if >> increment/decrement ops weren't supported. Just because they would >> likely cause errors through misunderstanding.) > > There's precedent for not doing this in C itself - even though "=+" > (from very early versions of C, also =-, =*, =&) no longer acts as an > add-in-place operator, it's not recognized as a special token to prevent > errors either. Given that the in-place operators changed to +=, -=, etc very early on, I doubt there's anyone who is actually confused by them. And it'd be extremely annoying to have to stop and think about parsing rules when taking or dereferencing pointers: /* This works */ x = &y; /* Why shouldn't this? */ x=&y; To the greatest extent possible, spaces around assignment operators should be the domain of style guides, not syntax. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On 05/06/2015 02:26 AM, Steven D'Aprano wrote: On Wednesday 06 May 2015 14:05, Steven D'Aprano wrote: My interpretation of this is that the difference has something to do with the cost of multiplications. Multiplying upwards seems to be more expensive than multiplying downwards, a result I never would have predicted, but that's what I'm seeing. I can only guess that it has something to do with the way multiplication is implemented, or perhaps the memory management involved, or something. Who the hell knows? I had guessed that the order of multiplication would make a big difference, once the product started getting bigger than the machine word size. Reason I thought that is that if you multiply starting at the top value (and end with multiplying by 2) you're spending more of the time multiplying big-ints. That's why I made sure that both Cecil's and my implementations were counting up, so that wouldn't be a distinction. I'm still puzzled, as it seems your results imply that big-int*int is faster than int*int where the product is also int. That could use some more testing, though. I still say a cutoff of about 10% is where we should draw the line in an interpretive system. Below that, you're frequently measuring noise and coincidence. Remember the days when you knew how many cycles each assembly instruction took, and could simply add them up to compare algorithms? -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On Wed, May 6, 2015 at 11:12 PM, Dave Angel wrote: > I had guessed that the order of multiplication would make a big difference, > once the product started getting bigger than the machine word size. > > Reason I thought that is that if you multiply starting at the top value (and > end with multiplying by 2) you're spending more of the time multiplying > big-ints. > > That's why I made sure that both Cecil's and my implementations were > counting up, so that wouldn't be a distinction. > > I'm still puzzled, as it seems your results imply that big-int*int is faster > than int*int where the product is also int. Are you using Python 2 or Python 3 for your testing? In Py3, there's no type difference, and barely no performance difference as you cross the word-size boundary. (Bigger numbers are a bit slower to work with, but not hugely.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On 05/06/2015 09:55 AM, Chris Angelico wrote: On Wed, May 6, 2015 at 11:12 PM, Dave Angel wrote: I had guessed that the order of multiplication would make a big difference, once the product started getting bigger than the machine word size. Reason I thought that is that if you multiply starting at the top value (and end with multiplying by 2) you're spending more of the time multiplying big-ints. That's why I made sure that both Cecil's and my implementations were counting up, so that wouldn't be a distinction. I'm still puzzled, as it seems your results imply that big-int*int is faster than int*int where the product is also int. Are you using Python 2 or Python 3 for your testing? In Py3, there's no type difference, and barely no performance difference as you cross the word-size boundary. (Bigger numbers are a bit slower to work with, but not hugely.) Both Cecil and I are using 3.x I'm using 3.4 in particular. And I know int covers both big-int and int32. that's why I called it big-int, rather than long. I was, however, mistaken. it's not that threshold that we're crossing here, but another one, for MUCH larger numbers. factorial of 10 and of 20 have 456473 and 97350 digits, respectively. In binary, that would be about 190k bytes and 404k bytes, respectively. I was seeing factorial of 20 taking about 4.5 times as long as factorial of 10. All the other increments seemed fairly proportional. I'll bet the difference is something like the memory allocator using a different algorithm for blocks above 256k. Or the cache logic hitting a threshold. If it's caching, of course the threshold will differ wildly between machine architectures. If it's the memory allocator, that could easily vary between Python versions as well. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
Steven D'Aprano writes: > Multiplying upwards seems to be more expensive than multiplying > downwards... I can only guess that it has something to do with the way > multiplication is implemented, or perhaps the memory management > involved, or something. Who the hell knows? It seems pretty natural if multiplication uses the obvious quadratic-time pencil and paper algorithm. The cost of multiplying m*n is basically w(m)*w(n) where w(x) is the width of x in machine words. So for factorial where m is the counter and n is the running product, w(m) is always 1 while w(n) is basically log2(n!). From from math import log def xfac(seq): cost = logfac = 0.0 for i in seq: logfac += log(i,2) cost += logfac return cost def upward(n): return xfac(xrange(1,n+1)) def downward(n): return xfac(xrange(n,1,-1)) print upward(4),downward(4) I get: 10499542692.6 11652843833.5 A lower number for upward than downward. The difference isn't as large as your timings, but I think it still gives some explanation of the effect. -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
Paul Rubin writes: > Steven D'Aprano writes: >> Multiplying upwards seems to be more expensive than multiplying >> downwards... I can only guess that it has something to do with the way >> multiplication is implemented, or perhaps the memory management >> involved, or something. Who the hell knows? > > It seems pretty natural if multiplication uses the obvious > quadratic-time pencil and paper algorithm. The cost of multiplying m*n > is basically w(m)*w(n) where w(x) is the width of x in machine words. > So for factorial where m is the counter and n is the running product, > w(m) is always 1 while w(n) is basically log2(n!). From > > from math import log > def xfac(seq): > cost = logfac = 0.0 > for i in seq: > logfac += log(i,2) > cost += logfac > return cost > > def upward(n): return xfac(xrange(1,n+1)) > def downward(n): return xfac(xrange(n,1,-1)) > > print upward(4),downward(4) > > I get: 10499542692.6 11652843833.5 > > A lower number for upward than downward. The difference isn't as large > as your timings, but I think it still gives some explanation of the > effect. Yes, plus the time for memory allocation. Since the code uses "r *= ...", space is reallocated when the result doesn't fit. The new size is probably proportional to the current (insufficient) size. This means that overall, you'll need fewer reallocations, because allocations are made in bigger chunks. -- Alain. -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On Wed, May 6, 2015 at 9:12 AM, Paul Rubin wrote: > Steven D'Aprano writes: >> Multiplying upwards seems to be more expensive than multiplying >> downwards... I can only guess that it has something to do with the way >> multiplication is implemented, or perhaps the memory management >> involved, or something. Who the hell knows? > > It seems pretty natural if multiplication uses the obvious > quadratic-time pencil and paper algorithm. The cost of multiplying m*n > is basically w(m)*w(n) where w(x) is the width of x in machine words. > So for factorial where m is the counter and n is the running product, > w(m) is always 1 while w(n) is basically log2(n!). From > > from math import log > def xfac(seq): > cost = logfac = 0.0 > for i in seq: > logfac += log(i,2) > cost += logfac > return cost > > def upward(n): return xfac(xrange(1,n+1)) > def downward(n): return xfac(xrange(n,1,-1)) > > print upward(4),downward(4) > > I get: 10499542692.6 11652843833.5 > > A lower number for upward than downward. The difference isn't as large > as your timings, but I think it still gives some explanation of the > effect. That was my initial thought as well, but the problem is that this actually predicts the *opposite* of what is being reported: upward should be less expensive, not more. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On Wed, 6 May 2015 10:40 pm, BartC wrote: > But I had in mind not implementing ++ and --, but detecting them and > issuing a warning, That's a job for a linter, not the compiler. The compiler should be as flexible as possible in what it accepts: a ,b=12+3 * 4,"hello" . upper () is perfectly legal code. The compiler shouldn't force you to write good looking code, apart from what is prohibited altogether. Both + and - are unary prefix operators, so you can apply + and - to any expression -- even an expression that already has a unary prefix operator: py> - --- +++ + - - + -- +++ --- 999 -999 Is that ugly, horrible code that nobody in their right mind would use in production? Absolutely. But the compiler can and should accept it, and linters (or human reviewers) should warn about it. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On Wed, May 6, 2015 at 1:08 AM, Steven D'Aprano wrote: > On Wednesday 06 May 2015 15:58, Ian Kelly wrote: > >> On Tue, May 5, 2015 at 7:27 PM, Steven D'Aprano >> wrote: >>> Only the minimum is statistically useful. >> >> I disagree. The minimum tells you how fast the code *can* run, under >> optimal circumstances. The mean tells you how fast it *realistically* >> runs, under typical load. Both can be useful to measure. > > Er, not even close. Running code using timeit is in no way the same as > running code "for real" under realistic circumstances. The fact that you are > running the function or code snippet in isolation, in its own scope, via > exec, rather than as part of some larger script or application, should be a > hint. timeit itself has overhead, so you cannot measure the time taken by > the operation alone, you can only measure the time taken by the operation > within the timeit environment. We have no reason to think that the > distribution of noise under timeit will be even vaguely similar to the noise > when running in production. You also can't be sure that the base time taken by the operation in your development environment will be comparable to the time taken in production; different system architectures may produce different results, and what is faster on your workstation may be slower on a server. Also, different algorithms may react to load differently. For example, an algorithm that goes to different parts of memory frequently may start thrashing sooner than an algorithm with better spatial locality if the system is paging a lot. I'll grant that just computing the means on a workstation that is not under a controlled load is not the best way to measure this -- but a difference in mean that is not simply proportional to the difference in min is still potentially useful information. > The purpose of timeit is to compare individual algorithms, in as close as > possible to an equal footing with as little noise as possible. If you want > to profile code used in a realistic application, use a profiler, not timeit. > And even that doesn't tell you how fast the code would be alone, because the > profiler adds overhead. > > Besides, "typical load" is a myth -- there is no such thing. A high-end > Windows web server getting ten thousand hits a minute, a virtual machine > starved for RAM, a Mac laptop, a Linux server idling away with a load of 0.1 > all day... any of those machines could run your code. How can you *possibly* > say what is typical? The very idea is absurd. Agreed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Json Comaprision
In pra devOPS writes: > I wanted to compare two json files ignoring few of the keys in the json > files. > Can anybody suggest me few things? Load each json file into a python object, delete the keys you don't care about, and compare the two objects. -- John Gordon Imagine what it must be like for a real medical doctor to gor...@panix.comwatch 'House', or a real serial killer to watch 'Dexter'. -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
Ian Kelly writes: > That was my initial thought as well, but the problem is that this > actually predicts the *opposite* of what is being reported: upward > should be less expensive, not more. Wait, what? Hmm, you're right. Needed coffee, will think about it more later. -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On 2015-05-06 06:32, Kashif Rana wrote:
Hello guys
thanks for the feedback. I think its problem with excel itself, showing wrong
value. Because when I opened the csv file in text editor, I can see correct
value but opening in excel showing wrong value. What I can do to see correct in
excel as well.
Regards
You could tell it to quote any value that's not a number:
w = csv.DictWriter(f, pol_keys, quoting=csv.QUOTE_NONNUMERIC)
It looks like all of the values you have are strings, so they'll all be
quoted.
I would hope that Excel will then treat it as a string; it would be
stupid if it didn't! :-)
On Tuesday, May 5, 2015 at 9:09:40 PM UTC+4, Kashif Rana wrote:
Hello Experts
When I am writing list of dictionaries to CSV file, the key 'schedule' has
value 'Mar 2012' becomes Mar-12. I really do not have clue why thats happening.
Below is the code.
dic_1 = {'action': 'permit',
'dst-address': 'maxprddb-scan-167, maxprddb-scan-168, maxprddb-scan-169',
'from': 'DMZ Web',
'id': '1000',
'log': 'Enable, session-init',
'name': 'Test Rule Temporary ',
'service': '1521, Oraccle-Maximo-Scan-1550, PING-ALL',
'src-address': 'sparkeregap1, sparkeregap2',
'to': 'Trust'}
dic_2 {'action': 'permit',
'dst-address': 'sparkcas01, sparkcas02, email.ab.spark.net',
'from': 'DMZ Web',
'id': '4000',
'log': 'Enable, session-init',
'schedule': 'Mar 2012',
'service': 'SMTP',
'src-address': 'sparkeregap1, sparkeregap2',
'to': 'Trust'}
my_list =
[{'to': 'Trust', 'service': '1521, Oraccle-Maximo-Scan-1550, PING-ALL',
'from': 'DMZ Web', 'dst-address': 'maxprddb-scan-167, maxprddb-scan-168,
maxprddb-scan-169', 'name': 'Test Rule Temporary ', 'action': 'permit', 'id':
'1000', 'src-address': 'sparkeregap1, sparkeregap2', 'log': 'Enable,
session-init'}, {'to': 'Trust', 'from': 'DMZ Web', 'dst-address': 'sparkcas01,
sparkcas02, email.ab.spark.net', 'service': 'SMTP', 'schedule': 'Mar 2012',
'action': 'permit', 'id': '4000', 'src-address': 'sparkeregap1, sparkeregap2',
'log': 'Enable, session-init'}]
pol_keys = ['id', 'name', 'from', 'to', 'src-address', 'dst-address',
'service', 'action', 'nat_status', 'nat_type', 'nat_src_ip', 'nat_dst_ip',
'nat_dst_port', 'log', 'schedule']
with open('test.csv', 'wb') as f:
w = csv.DictWriter(f, pol_keys)
w.writeheader()
w.writerows(my_list)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On 2015-05-06 19:08, MRAB wrote: > You could tell it to quote any value that's not a number: > > w = csv.DictWriter(f, pol_keys, > quoting=csv.QUOTE_NONNUMERIC) > > It looks like all of the values you have are strings, so they'll > all be quoted. > > I would hope that Excel will then treat it as a string; it would be > stupid if it didn't! :-) Sadly, Excel *is* that stupid based on the tests I tried just now. :-( Regardless of whether "Mar 2015" is quoted or unquoted in the source CSV file, Excel tries to outwit you and mangles the presentation. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On Wed, May 6, 2015 at 12:22 PM, Tim Chase wrote: > On 2015-05-06 19:08, MRAB wrote: >> You could tell it to quote any value that's not a number: >> >> w = csv.DictWriter(f, pol_keys, >> quoting=csv.QUOTE_NONNUMERIC) >> >> It looks like all of the values you have are strings, so they'll >> all be quoted. >> >> I would hope that Excel will then treat it as a string; it would be >> stupid if it didn't! :-) > > Sadly, Excel *is* that stupid based on the tests I tried just now. :-( > > Regardless of whether "Mar 2015" is quoted or unquoted in the source > CSV file, Excel tries to outwit you and mangles the presentation. Quoting a value in csv doesn't mean it's a string; it just means that it's a single field. You *can* force Excel to treat a value as a string by prefixing it with an apostrophe, though. -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On 2015-05-06 12:27, Ian Kelly wrote: > On Wed, May 6, 2015 at 12:22 PM, Tim Chase > wrote: > > On 2015-05-06 19:08, MRAB wrote: > >> You could tell it to quote any value that's not a number: > >> > >> w = csv.DictWriter(f, pol_keys, > >> quoting=csv.QUOTE_NONNUMERIC) > >> > >> It looks like all of the values you have are strings, so they'll > >> all be quoted. > >> > >> I would hope that Excel will then treat it as a string; it would > >> be stupid if it didn't! :-) > > > > Sadly, Excel *is* that stupid based on the tests I tried just > > now. :-( > > > > Regardless of whether "Mar 2015" is quoted or unquoted in the > > source CSV file, Excel tries to outwit you and mangles the > > presentation. > > Quoting a value in csv doesn't mean it's a string; it just means > that it's a single field. > > You *can* force Excel to treat a value as a string by prefixing it > with an apostrophe, though. Excel takes the apostrophe in the CSV file and puts it in the content, rather than stripping it as an escape/formatting character: c:\temp> type test.csv "'Mar 2015",Mar 2015,3,2015 "Apr 2015",Apr 2015,4,2015 "2015-12",2015-12,12,2015 c:\temp> start test.csv A1 has the unformatted text, but includes the apostrophe in the value. B1, A2, and B2 get munged like the OP described to the form "Apr-15". The items in row #3 come through untouched. At least on Excel 2003 on WinXP which is what I happen to have on hand. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Bitten by my C/Java experience
On 06/05/2015 17:03, Steven D'Aprano wrote: On Wed, 6 May 2015 10:40 pm, BartC wrote: But I had in mind not implementing ++ and --, but detecting them and issuing a warning, That's a job for a linter, not the compiler. The compiler should be as flexible as possible in what it accepts: a ,b=12+3 * 4,"hello" . upper () is perfectly legal code. The compiler shouldn't force you to write good looking code, apart from what is prohibited altogether. Both + and - are unary prefix operators, so you can apply + and - to any expression -- even an expression that already has a unary prefix operator: py> - --- +++ + - - + -- +++ --- 999 -999 Is that ugly, horrible code that nobody in their right mind would use in production? Absolutely. But the compiler can and should accept it, and linters (or human reviewers) should warn about it. Linters were mentioned a day or two back. Take a horse to water... -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On 05/06/2015 11:36 AM, Alain Ketterlin wrote: Yes, plus the time for memory allocation. Since the code uses "r *= ...", space is reallocated when the result doesn't fit. The new size is probably proportional to the current (insufficient) size. This means that overall, you'll need fewer reallocations, because allocations are made in bigger chunks. That sounds plausible, but a=5; a*=4 does not update in place. It calculates and creates a new object. Updating lists can work as you say, but an int is immutable. It's an optimization that might be applied if the code generator were a lot smarter, (and if the ref count is exactly 1), but it would then be confusing to anyone who used id(). -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On 06/05/2015 17:17, Ian Kelly wrote: On Wed, May 6, 2015 at 1:08 AM, Steven D'Aprano Besides, "typical load" is a myth -- there is no such thing. A high-end Windows web server getting ten thousand hits a minute, a virtual machine starved for RAM, a Mac laptop, a Linux server idling away with a load of 0.1 all day... any of those machines could run your code. How can you *possibly* say what is typical? The very idea is absurd. Agreed. I must disagree with this, on the grounds that a "typical load" of old cobblers are frequently spouted on this list. Practicality beats purity any day of the week. But no, this '=' is misused, it should have been ':='. Who really cares, if you don't like the language, pick another one, there's thousands of them to choose from. Ah, another chance for me to plug the virtues of CORAL 66 and CORAL 250. Capability violation anybody? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
extracting zip item to in-memory
I have a zip file containing several files and I want to extract out just the .xml file. I have that code. Curious if this xml file can be extracted into memory. If so, how to? I only need to move the file around, and maybe read some tags. Thanks for any help! python 2.7 -- https://mail.python.org/mailman/listinfo/python-list
Re: extracting zip item to in-memory
On 05/06/2015 04:27 PM, noydb wrote: I have a zip file containing several files and I want to extract out just the .xml file. I have that code. Curious if this xml file can be extracted into memory. If so, how to? I only need to move the file around, and maybe read some tags. Thanks for any help! python 2.7 See https://docs.python.org/2.7/library/zipfile.html#zipfile.ZipFile.open To open a particular member and get back a file-like object. Once you have that, you can use the read() method of that object. Once you've coded this, if it doesn't work, post what you've got with a description of what doesn't work, and somebody here will be able to fix it up. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: extracting zip item to in-memory
On 06/05/2015 21:27, noydb wrote: I have a zip file containing several files and I want to extract out just the .xml file. I have that code. Curious if this xml file can be extracted into memory. If so, how to? I only need to move the file around, and maybe read some tags. Thanks for any help! python 2.7 https://docs.python.org/2/library/xml.html#module-xml -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Encrypt python files
On 5/6/2015 12:23 AM, Palpandi wrote: On Wednesday, May 6, 2015 at 12:07:13 PM UTC+5:30, Palpandi wrote: Hi, What are the ways to encrypt python files? No, I just want to hide the scripts from others. See http://stackoverflow.com/questions/261638/how-do-i-protect-python-code Emile -- https://mail.python.org/mailman/listinfo/python-list
PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API
Hi.
I don't like that subprocess.Popen(['command']) only works on Windows if there
is a command.exe in %PATH%. As a Windows user you would normally expect that
also command.bat and command.cmd can be run that way.
There are simple workarounds like Popen(..., shell=True) but that is a heavy
overhead for .exe files.
Currently I use pywin32 and call Popen([win32api.FindExecutable('command')[1]])
as a workaround. This has zero overhead.
It should be default for Popen to call FindExecutable internally.
Was this discussed before?
Is it worth a PEP?
Or at least an issue?
Cheers,
Stefan
--
https://mail.python.org/mailman/listinfo/python-list
Re: CHALLENGE HELP - GOOGLE DEVELOPER DAY
On Tue, 05 May 2015 09:59:03 -0700, worship.brother wrote: > Archaeologists have found a scroll with the following texts: First you need to visit the hidden Temple of Offler, where you will find the Tears of Offler (they're the big gems set into the statue just above the teeth) after making your way through the maze with the big rolling ball, spiky pit, deadfall and spinning blade traps. Then you offer the tears off Offler up on the altar of Cthulhu[1]. Either your brain will melt when Cthulhu appears to you, or you will be blessed with the inspiration to write the code to solve your problem. If the latter, and the code doesn't work, show us the code and we might be able to make suggestions for you. [1] If you thought getting Offler's Tears was hard, wait until you try reaching the altar of Cthulhu. -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: [SciPy-User] Is there existing code to log-with-bells-on for runtime algorithm diagnostics?
Just to follow up on this thread, for interested readers' future reference... On Tue, Apr 21, 2015 at 4:22 PM, Robert Kern wrote: > On Tue, Apr 21, 2015 at 8:02 PM, Rob Clewley wrote: >> In fact, I'm trying to build a general purpose tool for exploring the >> inner workings of numerical algorithms for teaching and learning >> purposes, e.g. for graduate student training or for figuring out >> parameter choices in difficult applications. > > The term you want to search for is "structured logging". > > http://www.structlog.org/en/stable/ > http://eliot.readthedocs.org/en/stable/ > https://twiggy.readthedocs.org/en/latest/logging.html#structured-logging > http://netlogger.lbl.gov/ > I posted a new blog entry about my prototypical diagnosis and visualization tools for python numerical algorithms, built over matplotlib: http://robclewley.github.io/logging-and-diagnostic-tools-for-numeric-python-algorithms/ They utilize structlog, and I hooked up a noSQL DB (tinydb) to the logging to enable search capabilities of the log post-mortem. Comments and PRs most welcome. Thanks to everyone for the pointers. -Rob -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On Tue, 05 May 2015 22:32:28 -0700, Kashif Rana wrote: > thanks for the feedback. I think its problem with excel itself, showing > wrong value. Because when I opened the csv file in text editor, I can > see correct value but opening in excel showing wrong value. What I can > do to see correct in excel as well. You need to format your CSV date into a date format that Excel understands when it imports it. First thing to try would be to export some dates from excel as CSV and see what format excel puts them in. The see if excel recognises them as dates when you re-import the same file. If excel recognises it's own csv exported dates, reformat your dates to match the excel ones when you generate the csv. Otherwise, you might need to convert the dates to a numeric value and tell excel to format the field as date after input. -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API
On 05/06/2015 06:11 PM, Stefan Zimmermann wrote:
Hi.
I don't like that subprocess.Popen(['command']) only works on Windows if there
is a command.exe in %PATH%.
As a Windows user you would normally expect that also command.bat and
command.cmd can be run that way.
and command.com.
If it's really unfortunate that you picked "command" for your sample
program name. Since command.com was the shell in MSDOS, I was about to
point you to COMSPEC to address your problem.
There's nothing Windows-specific about that behaviour. In Linux, there
are bash commands that can only be run by using shell=True. Fortunately
Popen didn't make the mistake of pretending it's a shell.
There is lots more to running a batch file than launching it. The whole
syntax of the rest of the commandline differs when you're doing that.
There are simple workarounds like Popen(..., shell=True) but that is a heavy
overhead for .exe files.
And the reason there's such an overhead is because you're requesting the
services of the shell. If you don't need those services, use shell=False.
Currently I use pywin32 and call Popen([win32api.FindExecutable('command')[1]])
as a workaround. This has zero overhead.
It should be default for Popen to call FindExecutable internally.
Was this discussed before?
Is it worth a PEP?
Or at least an issue?
Cheers,
Stefan
--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On 2015-05-06 23:31, Denis McMahon wrote: > On Tue, 05 May 2015 22:32:28 -0700, Kashif Rana wrote: > > thanks for the feedback. I think its problem with excel itself, > > showing wrong value. Because when I opened the csv file in text > > editor, I can see correct value but opening in excel showing > > wrong value. What I can do to see correct in excel as well. > > You need to format your CSV date into a date format that Excel > understands when it imports it. That's part of the problem. Excel is taking the OP's literal values and interpreting them as dates (and formatting them in the way Excel wants to) rather than accepting them as literal values. As ChrisA posted earlier, you have to use Excel's Import functionality (there are several ways to get this wizard, but not all ways of opening a .csv trigger the wizard), then specify those particular columns as "Text" rather than "General" (which auto-detects as a Date). This prevents Excel from overthinking the problem, allowing Excel to import the text as-is like the OP wants. > First thing to try would be to export some dates from excel as CSV > and see what format excel puts them in. > > The see if excel recognises them as dates when you re-import the > same file. If you create the formatting on such cells the way that the OP wants them, then export, then re-import, Excel still mangles them since the CSV spec doesn't preserve formatting. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV
On 2015-05-06, Denis McMahon wrote: > You need to format your CSV date into a date format that Excel > understands when it imports it. > > First thing to try would be to export some dates from excel as CSV and > see what format excel puts them in. Beware of assuming that Excel can import its own exported data. You may be disappointed. -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing list of dictionaries to CSV [correction]
On 2015-05-06 20:22, Tim Chase wrote: > As ChrisA posted earlier, you have to use Excel's Import > functionality (there are several ways to get this wizard, but not > all ways of opening a .csv trigger the wizard), then specify those > particular columns as "Text" rather than "General" Sorry, it was Dennis Lee Bieber responding to ChrisA with that suggestion. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API
On Thu, May 7, 2015 at 10:58 AM, Dave Angel wrote:
> There's nothing Windows-specific about that behaviour. In Linux, there are
> bash commands that can only be run by using shell=True. Fortunately Popen
> didn't make the mistake of pretending it's a shell.
But bash commands aren't the same as shell scripts. For instance, if
you want to enumerate bash aliases, you can't exec() to the 'alias'
command, because there isn't one. But shell scripts *can* be exec'd:
$ grep $ exec_demo.*
exec_demo.c:#include
exec_demo.c:#include
exec_demo.c:int main()
exec_demo.c:{
exec_demo.c: printf("This part is coming from C code.\n");
exec_demo.c: int err=execl("./exec_demo.sh", 0);
exec_demo.c: printf("exec() failed! %d\n",err);
exec_demo.c:}
exec_demo.sh:#!/bin/sh
exec_demo.sh:echo This part ran from the shell.
exec_demo.sh:echo Hello, world!
$ ./a.out
This part is coming from C code.
This part ran from the shell.
Hello, world!
$ pike -e 'Process.exec("./exec_demo.sh");'
This part ran from the shell.
Hello, world!
$ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
This part ran from the shell.
Hello, world!
(Python doesn't seem to have any way to 'exec', but a subprocess comes
to the same thing.)
I don't know about Windows, but it seems reasonable to be able to be
able to run many types of program equally, including batch files. But
maybe Windows is just weak that way.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Throw the cat among the pigeons
On Wed, 06 May 2015 09:12:05 -0400, Dave Angel wrote: > Remember the days when you knew how many cycles each assembly > instruction took, and could simply add them up to compare algorithms? I do! I do! :-) And then the MC68020 came out, and instruction execution overlapped in weird (but predictable) ways, and interrupts would have different (unpredictable) latency depending on how much internal state of the CPU had to be saved and restored. Man, am I *old*. Dan -- https://mail.python.org/mailman/listinfo/python-list
Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API
On Thursday 07 May 2015 12:19, Chris Angelico wrote:
> On Thu, May 7, 2015 at 10:58 AM, Dave Angel wrote:
>> There's nothing Windows-specific about that behaviour. In Linux, there
>> are
>> bash commands that can only be run by using shell=True. Fortunately
>> Popen didn't make the mistake of pretending it's a shell.
>
> But bash commands aren't the same as shell scripts. For instance, if
> you want to enumerate bash aliases, you can't exec() to the 'alias'
> command, because there isn't one. But shell scripts *can* be exec'd:
Um, are we still talking about Python here? exec("alias") fails with
NameError on all the versions of Python I know. *semi-wink*
I'm guessing you're taking about some other exec, it might be a good idea
that on a Python mailing list you don't assume that we're all going to
understand the context :-)
> $ grep $ exec_demo.*
> exec_demo.c:#include
> exec_demo.c:#include
> exec_demo.c:int main()
> exec_demo.c:{
> exec_demo.c: printf("This part is coming from C code.\n");
> exec_demo.c: int err=execl("./exec_demo.sh", 0);
> exec_demo.c: printf("exec() failed! %d\n",err);
> exec_demo.c:}
> exec_demo.sh:#!/bin/sh
> exec_demo.sh:echo This part ran from the shell.
> exec_demo.sh:echo Hello, world!
> $ ./a.out
> This part is coming from C code.
> This part ran from the shell.
> Hello, world!
> $ pike -e 'Process.exec("./exec_demo.sh");'
> This part ran from the shell.
> Hello, world!
Okay, so C code can call the shell. So can Pike.
> $ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
> This part ran from the shell.
> Hello, world!
And so can Python. I'm not entirely sure what point you are trying to make
here, or how it relates to the OP's problem that when he calls
subprocess.Popen(['foo'])
he expects it to run any of foo.exe, foo.cmd, foo.bat (and possibly any
other number of executable files). Are you agreeing with him or disagreeing?
Apart from any other number of problems, surely having "foo" alone run
foo.exe, foo.bat etc. is at best confusing and at worst a security risk?
What if you have *both* foo.exe and foo.bat in the same directory?
> (Python doesn't seem to have any way to 'exec', but a subprocess comes
> to the same thing.)
According to `man exec` on my Linux system, I don't think that is correct.
The exec* family of functions "replaces the current process image with a new
process image", they don't run in a subprocess.
I think the Python equivalent of Unix exec* commands are the various
os.exec* functions.
> I don't know about Windows, but it seems reasonable to be able to be
> able to run many types of program equally, including batch files. But
> maybe Windows is just weak that way.
Hmmm. I'm not sure if this is relevant, or if I'm going off on a tangent,
but if I write a short bash script and set the execute permission:
steve@runes:~$ chmod u+x test.sh
steve@runes:~$ cat test.sh
echo "Running shell script"
subprocess.call fails unless I set shell=True:
py> p = subprocess.Popen('./test.sh', shell=True)
py> Running shell script
py> p = subprocess.Popen('./test.sh', shell=False)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/local/lib/python2.7/subprocess.py", line 1308, in
_execute_child
raise child_exception
OSError: [Errno 8] Exec format error
How is this any different from needing to specify shell=True for .bat and
.cmd files under Windows? This is not a rhetorical question, I actually want
to know.
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: PEP idea: On Windows, subprocess should implicitly support .bat and .cmd scripts by using FindExecutable from win32 API
On Thu, May 7, 2015 at 1:33 PM, Steven D'Aprano
wrote:
> On Thursday 07 May 2015 12:19, Chris Angelico wrote:
>
>> On Thu, May 7, 2015 at 10:58 AM, Dave Angel wrote:
>>> There's nothing Windows-specific about that behaviour. In Linux, there
>>> are
>>> bash commands that can only be run by using shell=True. Fortunately
>>> Popen didn't make the mistake of pretending it's a shell.
>>
>> But bash commands aren't the same as shell scripts. For instance, if
>> you want to enumerate bash aliases, you can't exec() to the 'alias'
>> command, because there isn't one. But shell scripts *can* be exec'd:
>
> Um, are we still talking about Python here? exec("alias") fails with
> NameError on all the versions of Python I know. *semi-wink*
>
> I'm guessing you're taking about some other exec, it might be a good idea
> that on a Python mailing list you don't assume that we're all going to
> understand the context :-)
Fair point. I'm talking about the underlying exec* family of
functions, which are used on Unix-like systems to do normal process
execution. More on that later.
>> $ grep $ exec_demo.*
>> exec_demo.c:#include
>> exec_demo.c:#include
>> exec_demo.c:int main()
>> exec_demo.c:{
>> exec_demo.c: printf("This part is coming from C code.\n");
>> exec_demo.c: int err=execl("./exec_demo.sh", 0);
>> exec_demo.c: printf("exec() failed! %d\n",err);
>> exec_demo.c:}
>> exec_demo.sh:#!/bin/sh
>> exec_demo.sh:echo This part ran from the shell.
>> exec_demo.sh:echo Hello, world!
>> $ ./a.out
>> This part is coming from C code.
>> This part ran from the shell.
>> Hello, world!
>
>> $ pike -e 'Process.exec("./exec_demo.sh");'
>> This part ran from the shell.
>> Hello, world!
>
> Okay, so C code can call the shell. So can Pike.
>
>
>> $ python -c 'import subprocess; subprocess.call(["./exec_demo.sh"])'
>> This part ran from the shell.
>> Hello, world!
>
> And so can Python. I'm not entirely sure what point you are trying to make
> here, or how it relates to the OP's problem that when he calls
>
> subprocess.Popen(['foo'])
>
> he expects it to run any of foo.exe, foo.cmd, foo.bat (and possibly any
> other number of executable files). Are you agreeing with him or disagreeing?
I'm stating that this works on Unix with shell=False. Consequently,
the OP's request that it work thusly on Windows ought to be possible,
and ought to be reasonable.
> Apart from any other number of problems, surely having "foo" alone run
> foo.exe, foo.bat etc. is at best confusing and at worst a security risk?
> What if you have *both* foo.exe and foo.bat in the same directory?
There's a specific search order. Back in the days of DOS, it was
simply "com, then exe, then bat", but on modern Windowses, I think
it's governed by an environment variable (similarly to PATH for the
directories to search in). It's identical security risk to the
possibility of putting something earlier in $PATH; if you insist on
running a specific executable, you put the entire path, and then
there's no problem. (Actually, I'd consider this to be a feature, not
a bug - it's the equivalent of shadowing built-ins in a Python module.
When you want it, it's really useful.)
>> (Python doesn't seem to have any way to 'exec', but a subprocess comes
>> to the same thing.)
>
> According to `man exec` on my Linux system, I don't think that is correct.
> The exec* family of functions "replaces the current process image with a new
> process image", they don't run in a subprocess.
>
> I think the Python equivalent of Unix exec* commands are the various
> os.exec* functions.
Oh, I forgot about os.exec. That's a better equivalent:
$ python3 -c 'import os; os.execl("./exec_demo.sh","exec_demo.sh")'
This part ran from the shell.
Hello, world!
When you call subprocess.* to run something, what happens is generally
that Python forks and exec's to the new process. The parent of the
fork keeps running (or maybe waits on the child immediately), the
child uses one of the exec family of functions to replace itself with
the new process. This is different from Windows, where there's a
standard API function for "start this program in a new process".
Advantages of the fork/exec model include that you can do stuff in
between the two steps - for instance, you fork, then you change your
user credentials, root directory, current directory, nice value, usage
limits, etc, etc, etc, etc, prior to exec'ing. Unix systems don't need
a way to say "run this process with this root directory", because it
can all be built up from primitives.
So yes, in terms of the ability to locate other executables, it makes
no difference whether you fork first or not - the exec call is the
same.
> Hmmm. I'm not sure if this is relevant, or if I'm going off on a tangent,
> but if I write a short bash script and set the execute permission:
>
> steve@runes:~$ chmod u+x test.sh
> steve@runes:~$ cat test.sh
> echo "Running shell script"
>
> subprocess.call fails unless I set shell=True:
>
> py> p = subprocess.Popen('./test.sh',
