Re: Executing untrusted scripts in a sandboxed environment
On Sat, Oct 6, 2012 at 8:22 AM, Robin Krahl wrote: > Hi all, > > I need to execute untrusted scripts in my Python application. To avoid > security issues, I want to use a sandboxed environment. This means that the > script authors have no access to the file system. They may only access > objects, modules and classes that are "flagged" or "approved" for scripting. > > I read that I will not be able to do this with Python scripts. (See > SandboxedPython page in the Python wiki [0] and several SE.com questions, e. > g. [1].) So my question is: What is the best way to "embed" a script engine > in a sandboxed environment that has access to the Python modules and classes > that I provide? With extreme difficulty. A while back (couple years maybe? I don't remember), I ignored everyone's warnings and tried to make a sandboxed Python, embedded in a C++ application. It failed in sandboxing. With just some trivial tinkering using Python's introspection facilities, a couple of python-list people managed to read and write files, and other equally dangerous actions. Shortly thereafter, we solved the problem completely... by switching to JavaScript. Embedding CPython in an application simply doesn't afford sandboxing. To what extent do you actually need to run untrusted Python? Can you, for instance, sandbox the entire process (which wasn't an option for what we were doing)? Perhaps chrooting the Python interpreter will do what you need. But there may still be leaks, I don't know. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Coexistence of Python 2.x and 3.x on same OS
Using Python on Windows is a dream. Python uses and needs the system, but the system does not use Python. Every Python version is installed in its own isolated space, site-packages included and without any defined environment variable. Every Python can be seen as a different application. Knowing this, it is a no-problem to use the miscellaneous versions; can be with the console, with an editor, with .bat or .cmd files, with the Windows "start menu" launcher, ... like any application. The file extension is a double sword. Do not use it or unregister it, the msi installer allows to do this. It is the same task/problem as with any file, .txt, .png, ... The new Python launcher is a redondant tool. A point of view from a multi-users desktop user. In my mind, it is a mistake to deliver a ready preconfigurated installation. "TeX" (I may say, as usual) is doing fine. A "TeX" installation consists usually only in "TeX" engines installation/configuration. It let the user work the way he wishes. jmf -- http://mail.python.org/mailman/listinfo/python-list
Re: write binary with struct.pack_into
palmeira wrote:
> Dear pythonists,
>
> I'm having a problem with read/write binary in python.
> I have a binary file that I need to read information, extract a array,
> modify this array and put these values into file again in same binary
> format.
> I need to use unpack_from and pack_into because sometimes gonna need
> read/write in the middle of file.
Use pack/unpack and file.seek() instead.
> Script:
>
> import struct
> bloco='>%df' %(252) #Binary format
>
> # READ
> fa=open('testIN.bin')
> my_array=struct.unpack_from(bloco,fa.read()[0*4:251*4])# my_aray = 252
> elements array
> ## This read is OK!
>
> #WRITE
> fb=open('testOUT.bin')
> test=struct.pack_into(bloco,fb.write()[0*4:251*4]) # ERROR in this WRITE
However, I think you have picked the wrong API. So:
# untested
import sys
import array
offset = 0
N = 252
a = array.array("f")
with open("testIN.bin", "rb") as f:
f.seek(offset)
a.read(f, N)
if sys.byteorder == "little":
a.byteswap()
# process a
if sys.byteorder == "little":
a.byteswap()
with open("testOUT.bin", "wb") as f:
f.seek(offset)
a.write(f)
--
http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
On Sat, 06 Oct 2012 08:27:25 +0200, Manuel Pégourié-Gonnard wrote: > Hi, > > I was looking at the example found here [1] which begins with: > > [1] http://docs.python.org/py3k/library/imp.html#examples > > def __import__(name, globals=None, locals=None, fromlist=None): > # Fast path: see if the module has already been imported. try: > return sys.modules[name] > except KeyError: > pass > > I was wondering if the formulation > > if name in sys.modules: > return sys.modules[name] > > would be equivalent. IOW, is using try/except here only a matter of > style or a necessity? Mostly style, but not entirely. If you expect that most of the time the module will be found, the try...except version will be faster. If you expect that most of the time the module will not be found, the "if name in" version will be faster. But see also: > I'm suspecting that maybe, in multithreaded environments, the second > option may be subject to a race condition, if another thread removes > name frome sys.modules between the if and the return, but as I'm not > very familiar (yet) with Python threads, I'm not sure it is a real > concern here. In practice, no, it would be very unusual for another thread to remove the name from sys.modules. So don't do that :) But in principle, yes, it is a race condition and yes it is a (small) concern. Since it is so easy to avoid even this tiny risk, why not use the try...except version and avoid it completely? -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
Manuel Pégourié-Gonnard wrote: >Hi, >I was looking at the example found here [1] which begins with: > >[1] http://docs.python.org/py3k/library/imp.html#examples > >def __import__(name, globals=None, locals=None, fromlist=None): ># Fast path: see if the module has already been imported. >try: >return sys.modules[name] >except KeyError: >pass > >I was wondering if the formulation > >if name in sys.modules: >return sys.modules[name] > >would be equivalent. IOW, is using try/except here only a matter of >style or a necessity? Somewhere I read a text regarding 'try:' versus 'if'. If you take the probabitility into consideration, how many times the test will fail or succeed, there are two possibilities: - If the test will fail only on very rare occasions: Use 'try:'. When the statement(s) in the try-path succeed, the try:/except: construct will not consume additional execution time (not even for the test). The statements will just be executed as they are. Only in the (rare) case of failure, the exception handling will take additional execution time (a considerably big amount). The fact, that in python it is not named 'error handling', but 'exception handling' reflects this. The failure should be the exception, also in matters of occurrence. - If the relation between success and failure is not predictable, or if the case of failure will be frequent, use 'if'. The failure of a 'try:' gives you a penalty in form of consumption of a high amount of execution time. So, in these constellations, it is better to accept the relatively small amount of execution time taken by the explicit test. Obviously, you can use 'try:' only, if there is the possibility to produce an exception on failure. In the other cases you must use the explicit test by 'if'. >I'm suspecting that maybe, in multithreaded environments, the second >option may be subject to a race condition, if another thread removes >name frome sys.modules between the if and the return, but as I'm not >very familiar (yet) with Python threads, I'm not sure it is a real >concern here. Your idea sounds reasonable, but I also am not yet familiar with threads. >And maybe there are other reasons I'm completely missing for prefering >EAFP over LBYL here? One reason might be what I described above. Best regards, Günther -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing untrusted scripts in a sandboxed environment
On Saturday, 6 October 2012 04:00:08 UTC+5:30, Robin Krahl wrote: > Hi all, > > I need to execute untrusted scripts in my Python application. To avoid > security issues, I want to use a sandboxed environment. This means that the > script authors have no access to the file system. They may only access > objects, modules and classes that are "flagged" or "approved" for scripting. > > I read that I will not be able to do this with Python scripts. (See > SandboxedPython page in the Python wiki [0] and several SE.com questions, e. > g. [1].) So my question is: What is the best way to "embed" a script engine > in a sandboxed environment that has access to the Python modules and classes > that I provide? > > Thanks for your help. > > Best regards, > Robin > > [0] http://wiki.python.org/moin/SandboxedPython > [1] > http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python >From http://wiki.python.org/moin/SandboxedPython "The Java and CLR/.NET runtimes support restricted execution, and these can be utilised through the Jython and IronPython variants of Python (as well as by other languages, obviously)." You can also check out http://doc.pypy.org/en/latest/sandbox.html for PyPy's sandbox -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing untrusted scripts in a sandboxed environment
On Saturday, 6 October 2012 12:49:29 UTC+5:30, Chris Angelico wrote: > On Sat, Oct 6, 2012 at 8:22 AM, Robin Krahl wrote: > > > Hi all, > > > > > > I need to execute untrusted scripts in my Python application. To avoid > > security issues, I want to use a sandboxed environment. This means that the > > script authors have no access to the file system. They may only access > > objects, modules and classes that are "flagged" or "approved" for scripting. > > > > > > I read that I will not be able to do this with Python scripts. (See > > SandboxedPython page in the Python wiki [0] and several SE.com questions, > > e. g. [1].) So my question is: What is the best way to "embed" a script > > engine in a sandboxed environment that has access to the Python modules and > > classes that I provide? > > > > With extreme difficulty. A while back (couple years maybe? I don't > > remember), I ignored everyone's warnings and tried to make a sandboxed > > Python, embedded in a C++ application. It failed in sandboxing. With > > just some trivial tinkering using Python's introspection facilities, a > > couple of python-list people managed to read and write files, and > > other equally dangerous actions. Shortly thereafter, we solved the > > problem completely... by switching to JavaScript. > > > > Embedding CPython in an application simply doesn't afford sandboxing. > > To what extent do you actually need to run untrusted Python? Can you, > > for instance, sandbox the entire process (which wasn't an option for > > what we were doing)? Perhaps chrooting the Python interpreter will do > > what you need. But there may still be leaks, I don't know. > > > > ChrisA Something like ast.literal_eval may be useful. -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing untrusted scripts in a sandboxed environment
On Sat, Oct 6, 2012 at 7:10 PM, Ramchandra Apte wrote: > On Saturday, 6 October 2012 12:49:29 UTC+5:30, Chris Angelico wrote: >> On Sat, Oct 6, 2012 at 8:22 AM, Robin Krahl wrote: >> > What is the best way to "embed" a script engine in a sandboxed environment >> > that has access to the Python modules and classes that I provide? >> >> With extreme difficulty. > > Something like ast.literal_eval may be useful. Not really; it's hardly sufficient. That sort of feature is handy for making an expression evaluator; for instance, you could implement a powerful calculator with it. But it's far too limited for most applications. The main problem is permitting some of the basic builtins (like True, False, len(), etc), without those objects being used as gateways. Did you know, for instance, that len.__self__.open() can be used to read and write files on the file system? ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
Günther Dietrich scripsit : > Somewhere I read a text regarding 'try:' versus 'if'. If you take the > probabitility into consideration, how many times the test will fail or > succeed, there are two possibilities: [...] Ok, thanks for the details! -- Manuel Pégourié-Gonnard - http://people.math.jussieu.fr/~mpg/ -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
Steven D'Aprano scripsit : > If you expect that most of the time the module will be found, the > try...except version will be faster. If you expect that most of the time > the module will not be found, the "if name in" version will be faster. > Ok. In the particular case of __import__, I guess speed is not crucial since I doubt import often happen within a program's inner loop. But I'll remember that point for other cases anyway. >> I'm suspecting that maybe, in multithreaded environments, the second >> option may be subject to a race condition, if another thread removes >> name frome sys.modules between the if and the return, but as I'm not >> very familiar (yet) with Python threads, I'm not sure it is a real >> concern here. > > In practice, no, it would be very unusual for another thread to remove > the name from sys.modules. So don't do that :) > That wasn't my intention. But sometimes other people may be "creative" :) > But in principle, yes, it is a race condition and yes it is a (small) > concern. Since it is so easy to avoid even this tiny risk, why not use > the try...except version and avoid it completely? > Ok. Thanks for your explanations. -- Manuel Pégourié-Gonnard - http://people.math.jussieu.fr/~mpg/ -- http://mail.python.org/mailman/listinfo/python-list
Unpaking Tuple
Hi, I am using python 2.6. I need a way to make following code working without any ValueError . >>> a, b, c, d = (1,2,3,4) >>> a, b, c, d = (1,2,3). Note: Number of values in the tuple will change dynamically. I know in python 3, you can do `a, b, c, *d = (1, 2, 3)` and then d will contain any elements that didn't fit into a,b,c. Regards, Saju -- http://mail.python.org/mailman/listinfo/python-list
Re: Unpaking Tuple
On Sat, Oct 6, 2012 at 3:09 AM, sajuptpm wrote: > Hi, > > I am using python 2.6. > > I need a way to make following code working without any ValueError . a, b, c, d = (1,2,3,4) a, b, c, d = (1,2,3). > > Note: Number of values in the tuple will change dynamically. Then you arguably want a list, not a tuple. But at any rate: shortfall = 4 - len(your_tuple) your_tuple += (None,) * shortfall # assuming None is a suitable default a, b, c, d = your_tuple If you also need to handle the "too many items" case, use slicing: a, b, c, d = your_tuple[:4] Cheers, Chris -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing untrusted scripts in a sandboxed environment
On 05/10/2012 23:22, Robin Krahl wrote: Hi all, I need to execute untrusted scripts in my Python application. To avoid security issues, I want to use a sandboxed environment. This means that the script authors have no access to the file system. They may only access objects, modules and classes that are "flagged" or "approved" for scripting. I read that I will not be able to do this with Python scripts. (See SandboxedPython page in the Python wiki [0] and several SE.com questions, e. g. [1].) So my question is: What is the best way to "embed" a script engine in a sandboxed environment that has access to the Python modules and classes that I provide? Thanks for your help. Best regards, Robin [0] http://wiki.python.org/moin/SandboxedPython [1] http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python As good a starting point as any http://www.velocityreviews.com/forums/t716131-challenge-escape-from-the-pysandbox.html ? Also throw "python experimental sandbox" into your search engine and follow your nose, something might come up smelling of roses :) -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
On 10/06/2012 02:27 AM, Manuel Pégourié-Gonnard wrote: > Hi, > > I was looking at the example found here [1] which begins with: > > [1] http://docs.python.org/py3k/library/imp.html#examples > > def __import__(name, globals=None, locals=None, fromlist=None): > # Fast path: see if the module has already been imported. > try: > return sys.modules[name] > except KeyError: > pass > > I was wondering if the formulation > > if name in sys.modules: > return sys.modules[name] > > would be equivalent. IOW, is using try/except here only a matter of > style or a necessity? > > I'm suspecting that maybe, in multithreaded environments, the second > option may be subject to a race condition, if another thread removes > name frome sys.modules between the if and the return, but as I'm not > very familiar (yet) with Python threads, I'm not sure it is a real > concern here. > > And maybe there are other reasons I'm completely missing for prefering > EAFP over LBYL here? > > Thanks in advance for your comments. > Guidelines for writing library code may very well be different than for writing your application. And if your application is trying to do something similar with *import*, chances are that it's calling a library function that already starts with the test against sys.modules. So if this is an application question, the answer is probably "don't do either one, just do the import, checking for the exceptions that it may throw." The distinction in performance between the success and failure modes of the try/catch isn't nearly as large as one of the other responses might lead you to believe. For example, a for loop generally terminates with a raise (of StopIteration exception), and that doesn't convince us to replace it with a while loop. Besides, in this case, the except code effectively includes the entire import, which would completely swamp the overhead of the raise. If we assume the question was more generally about EAFT vs. LBYL, and not just about the case of accessing the system data structure sys.modules, then the issues change somewhat. If we do a LBYL, we have to know that we've covered all interesting cases with our test. Multithreading is one case where we can get a race condition. There are times when we might be able to know either that there are not other threads, or that the other threads don't mess with the stuff we're testing. For example, there are enough problems with import and threads that we might just have a development policy that (in this program) we will do all our imports before starting any additional threads, and that we will never try to unload an import, single threaded or not. But for other conditions, we might be affected either by the system or by other processes within it. Or even affected by other asynchronous events over a network. If we do an EAFP, then we have to figure out what exceptions are possible. However, adding more exceptions with different treatments is quite easy, and they don't all have to be done at the same level. Some may be left for our caller to deal with. I think the major thing that people mind about try/catch is that it seems to break up the program flow. However, that paradigm grows on you as you get accustomed to it. -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: sum function
On Saturday, 6 October 2012 02:09:56 UTC+5:30, Dave Angel wrote: > On 10/05/2012 04:09 PM, Mike wrote: > > > Terry, > > > > > > I am not using the mail client. I am just posting on the site. > > > > And which site would that be (that you're using)? There are a few. I'm > > guessing you use google-groups. And all of them get gatewayed to the > > actual list, with differing numbers of bugs. > > > > I use email to access it directly. I solve one of the duplicate-message > > problems with google groups by automatically deleting any message > > addressed to google-groups. There are about 100 such messages each month. > > > > Another problem is that lots of these gateways post to both the > > newsgroup and to the python-list. > I found out earlier that this was why my posts was being double-posted in Google Groups. > > > > > > > Something wrong with this site. When you do individual reply, it does the > > double posting which it shouldn't. See "Ramachandra Apte's" reply. It is > > posted twice too. > > > > > > Thanks > > > > > > > > > > > > > > > -- > > > > DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Unpaking Tuple
In article , Chris Rebert wrote: > But at any rate: > shortfall = 4 - len(your_tuple) > your_tuple += (None,) * shortfall # assuming None is a suitable default > a, b, c, d = your_tuple > > If you also need to handle the "too many items" case, use slicing: > a, b, c, d = your_tuple[:4] I usually handle both of those cases at the same time: >>> a, b, c, d = (my_tuple + (None,) * 4)[:4] -- http://mail.python.org/mailman/listinfo/python-list
Re: Executing untrusted scripts in a sandboxed environment
On Oct 5, 2012, at 6:32 PM, Robin Krahl wrote: > Hi all, > > I need to execute untrusted scripts in my Python application. To avoid > security issues, I want to use a sandboxed environment. This means that the > script authors have no access to the file system. They may only access > objects, modules and classes that are "flagged" or "approved" for scripting. > > I read that I will not be able to do this with Python scripts. (See > SandboxedPython page in the Python wiki [0] and several SE.com questions, e. > g. [1].) So my question is: What is the best way to "embed" a script engine > in a sandboxed environment that has access to the Python modules and classes > that I provide? Checkout udacity.com I think there is a writeup on stackoverflow on how they accomplished their sandbox runtime env. > > Thanks for your help. > > Best regards, >Robin > > [0] http://wiki.python.org/moin/SandboxedPython > [1] > http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python > -- > http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: write binary with struct.pack_into
On 2012-10-06, Dennis Lee Bieber wrote:
> On Fri, 5 Oct 2012 20:27:36 -0700 (PDT), palmeira
> declaimed the following in gmane.comp.python.general:
>
>>
>> #WRITE
>> fb=open('testOUT.bin')
>
> Unless you specify otherwise, open() defaults to read-only
It also defaults to 'text' mode which does cr/lf translaction. That
will break both reads and writes on any binary file containing 0x0a
and 0x0d bytes.
--
Grant
--
http://mail.python.org/mailman/listinfo/python-list
Re: write binary with struct.pack_into
On Sat, Oct 6, 2012 at 11:26 PM, Grant Edwards wrote:
> On 2012-10-06, Dennis Lee Bieber wrote:
>> On Fri, 5 Oct 2012 20:27:36 -0700 (PDT), palmeira
>> declaimed the following in gmane.comp.python.general:
>>
>>>
>>> #WRITE
>>> fb=open('testOUT.bin')
>>
>> Unless you specify otherwise, open() defaults to read-only
>
> It also defaults to 'text' mode which does cr/lf translaction. That
> will break both reads and writes on any binary file containing 0x0a
> and 0x0d bytes.
And, in Python 3, it'll be returning Unicode characters too, unless
the decode fails. You definitely want binary mode.
ChrisA
--
http://mail.python.org/mailman/listinfo/python-list
Re: Combinations of lists
On 4 October 2012 16:12, Steen Lysgaard wrote: > 2012/10/4 Joshua Landau : > > On 3 October 2012 21:15, Steen Lysgaard wrote: > >> > >> Hi, > >> > >> thanks for your interest. Sorry for not being completely clear, yes > >> the length of m will always be half of the length of h. > > > > > > (Please don't top post) > > > > I have a solution to this, then. > > It's not short or fast, but it's a lot faster than yours. > > This is quite naive, because I don't know how to properly implement > > force_unique_combinations, but it runs. I hope this is right. If you need > > significantly more speed your best chance is probably Cython or C, > although > > I don't doubt 10x more speed may well be possible from within Python. > > > > > > Also, 8 Dihedral is a bot, or at least pretending like crazy to be > one. > > Great, I've now got a solution much faster than what I could come up with. > Thanks to the both of you. > Don't use it though :P. I've something better, now I've used a few sanity-points up [it's much more interesting to solve *other* people's problems]. Please note that my implementations (old and new) return duplicates when the second list contains duplicates. It's fixable, but I'll only bother if you need it fixed. It runs in a very consistent 55% of the time, but it is longer. Here y'are. """Super algorithm.""" > > from itertools import combinations > from collections import Counter > > def multiples(counter): > """Counter -> set. > > Returns the set of values that occur multiple times. > """ > multiples = set() > > for item, number in counter.items(): > if number > 1: > multiples.add(item) > > return multiples > > #@profile > def pairwise_combinations(counter, countermultiples, counterset, length, > charmap): > # length is a LIE! > """Get the permutations of two lists. > > Do not call this directly unless you want to hassle yourself. > Use the wrapper provided, list_permute, instead. > """ > > # We will do the combinations in pairs, as each pair will not have order > and so > # [1, 2, 3, 4] is equivilant to [2, 1, 4, 3] but not [1, 3, 2, 4]. > > # This means that we get the full permutations without ever filtering. > > # Each pair along is a combination. > # We are combination-ing a set to prevent dulicates. > # As the combinations are of length 2, the only ones this will > # miss are of the type [x, x] (two of the same). > # This is accounted for by countermultiples. > pairs = combinations(counterset, 2) > > # Prepend with this > length -= 1 > prefix_char = charmap[length] > > # There's not reason to recurse, so don't bother with a lot of stuff > if not length: > for p_a, p_b in pairs: > yield [prefix_char+p_a, prefix_char+p_b] > for p in countermultiples: > yield [prefix_char+p, prefix_char+p] > return > > for p_a, p_b in pairs: > # Edit scope > # The recursion wont be able to use items we've already used > counter[p_a] -= 1 > counter_p_a = counter[p_a] # Quickref > if counter_p_a == 0: counterset.remove(p_a) # None left > elif counter_p_a == 1: countermultiples.remove(p_a) # Not plural > > counter[p_b] -= 1 > counter_p_b = counter[p_b] # Quickref > if counter_p_b == 0: counterset.remove(p_b) # None left > elif counter_p_b == 1: countermultiples.remove(p_b) # Not plural > > # Recurse > # Do the same, but for the next pair along > own_yield = [prefix_char+p_a, prefix_char+p_b] > for delegated_yield in pairwise_combinations(counter, countermultiples, > counterset, length, charmap): > yield own_yield + delegated_yield > > # Reset scope > counter[p_a] += 1 > if counter_p_a == 0: counterset.add(p_a) > elif counter_p_a == 1: countermultiples.add(p_a) > > counter[p_b] += 1 > if counter_p_b == 0: counterset.add(p_b) > elif counter_p_b == 1: countermultiples.add(p_b) > > > # Now do the same for countermultiples > # This is not itertools.chain'd because this way > # is faster and I get to micro-optomize inside > for p in countermultiples: > # Edit scope > # The recursion wont be able to use items we've already used > counter[p] -= 2 > counter_p = counter[p] # Quickref > > if counter_p == 0: > counterset.remove(p) # None left > countermultiples.remove(p) # Must have been in countermultiples, none left > > elif counter_p == 1: > countermultiples.remove(p) # Not plural > > # Recurse > # Do the same, but for the next pair along > own_yield = [prefix_char+p, prefix_char+p] > for delegated_yield in pairwise_combinations(counter, countermultiples, > counterset, length, charmap): > yield own_yield + delegated_yield > > # Reset scope > counter[p] += 2 > > if counter_p == 0: > counterset.add(p) > countermultiples.add(p) > > elif counter_p == 1: > countermultiples.add(p) > > def list_permute(first, second): > """Get the permutations of two lists as according to what you want, which > isn't really the > permutations of two lists but something close to it. It does what it > needs to, I think. > > This DOES NOT work when contains duplicates, as the result has > duplicates. Th
Re: write binary with struct.pack_into
First, you should consider reading the documentation of
struct.unpack_from and struct.pack_into at
http://docs.python.org/library/struct.html quite carefully. It says,
that these commands take a parameter called offset, which names the
location of the data in a buffer (e.g. an opened file).
example:
bloco='>%df' % (252)# Format string (252 floats)
fa = open('testIN.bin', 'rb') # open for reading in binary mode
off = 0 # suppose i want to read block at beginning of file
my_array=struct.unpack_from(bloco, fa, off) #read data
now write them to another file:
fb = open('testOUT.bin', 'r+b') # open for updating in binary mode
off = 0 # suppose i want to write block at beginning of file
struct.pack_into(bloco, fb, off, *my_array) #write data to testOUT
--
http://mail.python.org/mailman/listinfo/python-list
Re: parse an environment file
> The only canned solution for parsing a bash script is bash. Think > about it the other way around: If you wanted to have a Python variable > made available to a bash script, the obvious thing to do is to invoke > Python. It's the same thing. I scratched my own itch: http://code.activestate.com/recipes/578280-parse-profile/. -- http://mail.python.org/mailman/listinfo/python-list
Re: parse an environment file
On Sun, Oct 7, 2012 at 4:14 AM, Jason Friedman wrote: >> The only canned solution for parsing a bash script is bash. Think >> about it the other way around: If you wanted to have a Python variable >> made available to a bash script, the obvious thing to do is to invoke >> Python. It's the same thing. > > I scratched my own itch: > http://code.activestate.com/recipes/578280-parse-profile/. And there you have it! A subset of bash syntax and a Python parser. Job done! ChrisA -- http://mail.python.org/mailman/listinfo/python-list
I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
I am currently starting my PhD in software quality assurance and have been doing a lot of reading round this subject. I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool i.e. a tool/methodology that can accept many languages for the following areas: • Problems in code/coding errors • Compiler bugs • Language bugs • Users mathematical model I would greatly appreciate any input and advice in this area, feel free to repost on this topic and/or contact me at: owens.darryl@gmail.com Thank you in advance Darryl Owens -- http://mail.python.org/mailman/listinfo/python-list
Re: fmap(), "inverse" of Python map() function
On Saturday, October 6, 2012 5:01:40 AM UTC+5:30, Devin Jeanpierre wrote: > On Fri, Oct 5, 2012 at 7:24 PM, Ian Kelly wrote: > > > I realize that. My point is that the function *feels* more like a > > > variant of reduce than of map. > > > > > >> If it's meant as a complaint, it's a poor one. > > > > > > It's not. > > > > Fair enough all around. Sorry for misunderstanding. > > > > -- Devin Thanks to all who replied. Always good to learn something new. - Vasudev -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
On 10/6/2012 7:36 AM, Dave Angel wrote: The distinction in performance between the success and failure modes of the try/catch isn't nearly as large as one of the other responses might lead you to believe. For example, a for loop generally terminates with a raise (of StopIteration exception), and that doesn't convince us to replace it with a while loop. For statement generally loop many times, up to millions of times, without an exception being raised, whereas while statements test the condition each time around the loop. So the rule 'if failure is rare (less than 10-20%) use try', applies here. For if/them versus try/except, I don't worry too much about it. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
[ann] pysha3 0.2.1 released
Hello, today I've released pysha3 0.2.1 [1]. It's a standalone version of the SHA-3 extension that I merged into CPython's development branch (future 3.4) a couple of hours ago. It provides the Keccak [2] cryptographic hashing algorithm that was officially selected as SHA-3 by NIST a four days ago. The module implements 224, 256, 384 and 512 bits digest size. Arbitrarily-long output is not supported by the module as it's not part of the NIST interface. pysha3 is available for Python 2.6, 2.7, 3.2 and 3.3. It has been tested on Linux (X86, X86_64 with gcc 4.6 and clang), FreeBSD and Windows (X86, X86_64). 32 and 64bit Windows binaries for all supported Python versions are available on PyPI, too. Have fun, Christian [1] http://pypi.python.org/pypi/pysha3 [2] http://keccak.noekeon.org/ -- http://mail.python.org/mailman/listinfo/python-list
Re: fmap(), "inverse" of Python map() function
> Thanks to all who replied. Always good to learn something new. P.S. A reader posted a good comment with Scala as well as Python code for a compose function (basically same functionality as fmap, or more - the compose once, run many times thing). It's the 4th comment on my blog post. - Vasudev -- http://mail.python.org/mailman/listinfo/python-list
Re: Are ABCs an anti-pattern?
On 12-10-05 12:58 PM, Trent Nelson wrote: I like them. In particular, I like that I can enumerate all the subclasses that happen to implement the ABC via the metaclass's __subclasses__() method. As long as you have a common base class (which in your case is a requirement), then __subclasses__ works for introspecting child classes. It doesn't *really* have anything to do with abcs. I also like that I can trust Python not to instantiate a subclass of an ABC unless it meets all the interface criteria I've stipulated. Another way to read this is that you don't trust those using your code to be bright enough to understand what your code is doing and what it requires. In my mind, this seems to somewhat contradict the philosophy of "we're all consenting adults here". Whether you utilize interfaces or not, code should be documented. Your documentation would be responsible for laying out the expected interface (again, whether you're using the interfaces or not). Code would fail at some point if a requirement on an interface hasn't been filled. The *one* nice thing is that it'll error on import rather than execution time, but to me, if your code is unit tested, then all these things should be caught almost immediately anyway. From my experience (again, *only* talking about Python here), it seem to me that less is generally more. Less code means less things to read and tie together, making it easier to grok overall (not to mention less overhead for the interpreter, but that's virtually moot due to the *very* little overhead in 99% of cases of uses features such as abcs). Using abcs not only lends itself to code bloat, but it also leads to over-engineering as once you fall into old OOP habits, you start getting back to un-Pythonic code (pretty subjective statement, I know :)). Again, please don't misunderstand my intentions here. I'm not arguing the need for abstract base classes in a strict OOP world. I'm arguing them as not genuinely being Pythonic. Thanks for your the feedback so far. -- Demian Brecht @demianbrecht http://demianbrecht.github.com -- http://mail.python.org/mailman/listinfo/python-list
Re: [ann] pysha3 0.2.1 released
Christian Heimes wrote: today I've released pysha3 0.2.1 [1]. pysha3 is available for Python 2.6, 2.7, 3.2 and 3.3. It has been tested on Linux (X86, X86_64 with gcc 4.6 and clang), FreeBSD and Windows (X86, X86_64). 32 and 64bit Windows binaries for all supported Python versions are available on PyPI, too. [1] http://pypi.python.org/pypi/pysha3 [2] http://keccak.noekeon.org/ Nice! Thanks! ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is pylaucher in Python 3.3 being installed in Windows folder?
On 5/10/2012 2:40 AM, Oscar Benjamin wrote: Having them on PATH means that you can do: > py script.py and the effect will be analogous to (in a unix shell): $ ./script.py Of course the idea with the launcher is that you just do > script.py Unless you want a specific version - particularly for testing - eg: % py -3.2 script.py Mark -- http://mail.python.org/mailman/listinfo/python-list
Re: Coexistence of Python 2.x and 3.x on same OS
On Sat, Oct 6, 2012 at 1:27 AM, wrote: > Using Python on Windows is a dream. > > Python uses and needs the system, but the system does > not use Python. > > Every Python version is installed in its own isolated > space, site-packages included and without any defined > environment variable. Every Python can be seen as a > different application. > Knowing this, it is a no-problem to use the miscellaneous > versions; can be with the console, with an editor, with > .bat or .cmd files, with the Windows "start menu" launcher, > ... like any application. > > The file extension is a double sword. Do not use it or > unregister it, the msi installer allows to do this. > It is the same task/problem as with any file, .txt, .png, ... > > The new Python launcher is a redondant tool. Yes, because all scripts are only ever run by the person who wrote them. The main benefit here is for distribution of Python scripts with version requirements. Just because you can rely on your Python installation to be in C:\Python27 doesn't mean you can rely on somebody else's installation to be there, or on their file associations to be configured for the correct version. -- http://mail.python.org/mailman/listinfo/python-list
Re: try/except KeyError vs "if name in ..."
On Sunday, 7 October 2012 01:12:56 UTC+5:30, Terry Reedy wrote:
> On 10/6/2012 7:36 AM, Dave Angel wrote:
>
>
>
> > The distinction in performance between the success and failure modes of
>
> > the try/catch isn't nearly as large as one of the other responses might
>
> > lead you to believe. For example, a for loop generally terminates with
>
> > a raise (of StopIteration exception), and that doesn't convince us to
>
> > replace it with a while loop.
>
>
>
> For statement generally loop many times, up to millions of times,
>
> without an exception being raised, whereas while statements test the
>
> condition each time around the loop. So the rule 'if failure is rare
>
> (less than 10-20%) use try', applies here. For if/them versus
>
> try/except, I don't worry too much about it.
>
>
>
> --
>
> Terry Jan Reedy
I use try and except when I need to raise exceptions e.g.:
try:
value = vm.variables[name]
except KeyError:
raise NameError("variable name not defined in VM's variables")
--
http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: > I am currently starting my PhD in software quality assurance and have been > doing a lot of reading round this subject. I am just trying to find out if > there is any relevant/current research in the production of a generic quality > assurance tool i.e. a tool/methodology that can accept many languages for the > following areas: > > • Problems in code/coding errors > > • Compiler bugs > > • Language bugs > > • Users mathematical model > > I would greatly appreciate any input and advice in this area, feel free to > repost on this topic and/or contact me at: owens.darryl@gmail.com > > > > Thank you in advance > > > > Darryl Owens Does this have anything to do with Python? Banned from #python-offtopic till Christmas -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Oct 7, 9:15 am, Ramchandra Apte wrote: > On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: > > I am currently starting my PhD in software quality assurance and have been > > doing a lot of reading round this subject. I am just trying to find out if > > there is any relevant/current research in the production of a generic > > quality assurance tool i.e. a tool/methodology that can accept many > > languages for the following areas: > > > • Problems in code/coding errors > > > • Compiler bugs > > > • Language bugs > > > • Users mathematical model > > > I would greatly appreciate any input and advice in this area, feel free to > > repost on this topic and/or contact me at: owens.darryl@gmail.com > > > Thank you in advance > > > Darryl Owens > > Does this have anything to do with Python? Why not? > > Banned from #python-offtopic till Christmas Did you wait for an answer? -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sunday, 7 October 2012 10:32:45 UTC+5:30, rusi wrote: > On Oct 7, 9:15 am, Ramchandra Apte wrote: > > > On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: > > > > I am currently starting my PhD in software quality assurance and have > > > been doing a lot of reading round this subject. I am just trying to find > > > out if there is any relevant/current research in the production of a > > > generic quality assurance tool i.e. a tool/methodology that can accept > > > many languages for the following areas: > > > > > > > • Problems in code/coding errors > > > > > > > • Compiler bugs > > > > > > > • Language bugs > > > > > > > • Users mathematical model > > > > > > > I would greatly appreciate any input and advice in this area, feel free > > > to repost on this topic and/or contact me at: owens.darryl@gmail.com > > > > > > > Thank you in advance > > > > > > > Darryl Owens > > > > > > Does this have anything to do with Python? > > > > Why not? > > > > > > > > Banned from #python-offtopic till Christmas > > > > Did you wait for an answer? no -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sun, Oct 7, 2012 at 1:02 AM, rusi wrote: > On Oct 7, 9:15 am, Ramchandra Apte wrote: >> On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: >> > I am currently starting my PhD in software quality assurance and have been >> > doing a lot of reading round this subject. I am just trying to find out if >> > there is any relevant/current research in the production of a generic >> > quality assurance tool i.e. a tool/methodology that can accept many >> > languages for the following areas: >> >> > •Problems in code/coding errors >> >> > •Compiler bugs >> >> > •Language bugs >> >> > •Users mathematical model >> The main tests for python is: http://docs.python.org/library/unittest.html For other languages, and even in python, you can roll your own. I'd begin by algorithming each particular language's calls(based on the statistical probabilities of languages that are utilized, and designed in a hierarchical order of the utilization), language bugs, and mathematical models needed performed, then perform the necessary function calls/series of calls. Pass data, and check the returns. CMD errors in some cases, and checking for error logs from URL calls. I'd suggest the bug repositories for the OS, browser, or app framework the language is launched in(version/build #, etc), or some form of url scraping the data from these in order to correct/check known problems. -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sun, Oct 7, 2012 at 1:55 AM, Dwight Hutto wrote: > On Sun, Oct 7, 2012 at 1:02 AM, rusi wrote: >> On Oct 7, 9:15 am, Ramchandra Apte wrote: >>> On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: >>> > I am currently starting my PhD in software quality assurance and have >>> > been doing a lot of reading round this subject. I am just trying to find >>> > out if there is any relevant/current research in the production of a >>> > generic quality assurance tool i.e. a tool/methodology that can accept >>> > many languages for the following areas: >>> >>> > •Problems in code/coding errors >>> >>> > •Compiler bugs >>> >>> > •Language bugs >>> >>> > •Users mathematical model >>> You could also utilize other unittests from other languages, and roll that into wrappers that checked for specific languages utilization, and it's probable errors, by initiating the unittest functions with a python call specific to the language being utilized. -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sun, Oct 7, 2012 at 2:03 AM, Dwight Hutto wrote: > On Sun, Oct 7, 2012 at 1:55 AM, Dwight Hutto wrote: >> On Sun, Oct 7, 2012 at 1:02 AM, rusi wrote: >>> On Oct 7, 9:15 am, Ramchandra Apte wrote: On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: > I am currently starting my PhD in software quality assurance and have > been doing a lot of reading round this subject. I am just trying to find > out if there is any relevant/current research in the production of a > generic quality assurance tool i.e. a tool/methodology that can accept > many languages for the following areas: > •Problems in code/coding errors > •Compiler bugs > •Language bugs > •Users mathematical model > Maybe easier through checking particular error logs. -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list
Re: I am just trying to find out if there is any relevant/current research in the production of a generic quality assurance tool for my PhD.
On Sun, Oct 7, 2012 at 2:16 AM, Dwight Hutto wrote: > On Sun, Oct 7, 2012 at 2:03 AM, Dwight Hutto wrote: >> On Sun, Oct 7, 2012 at 1:55 AM, Dwight Hutto wrote: >>> On Sun, Oct 7, 2012 at 1:02 AM, rusi wrote: On Oct 7, 9:15 am, Ramchandra Apte wrote: > On Sunday, 7 October 2012 00:13:58 UTC+5:30, Darryl Owens wrote: > > I am currently starting my PhD in software quality assurance and have > > been doing a lot of reading round this subject. I am just trying to > > find out if there is any relevant/current research in the production of > > a generic quality assurance tool i.e. a tool/methodology that can > > accept many languages for the following areas: > > > •Problems in code/coding errors > > > •Compiler bugs > > > •Language bugs > > > •Users mathematical model > >> It is about 2:30 A.M. here, and I'm tossing out thoughts that could go deeper. You're looking for: > •Problems in code/coding errors > •Compiler bugs > •Language bugs > •Users mathematical model > The below is the base algorithm I see for every language: There are problems, and ways to test in every language. Some have more advanced tests based on their usage, and those who use them. You have identified the errors needed to be checked for: -Problems in code/coding errors -Compiler bugs -Language bugs -Users mathematical model 1. You have test methods in lots of languages for these, and you need to parse for the file extension, or something in the code that shows it has switched to a new language. I'm assuming classes and functions here 2. It seems like you should have a file/script in each language to check for as much as you can. 3. You could call these scripts via a python command line app, and have an app to display the output, and check for know error calls returned from the command line output(stderr/stdin/,etc), or the browsers output/error logs. 4. You could go to a lower level. 5. You're in python, so pick the best way to wrap and execute the above based on file extensions, and parsing, then run your test on portions of code if the have parameters or error values, or the code as a whole, and deal with each of the problems stated above. This is just to begin to understand your mentality of how you want to implement in python. -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list
