Re: How to make Python run as fast (or faster) than Julia
On Sun, 25 Feb 2018 21:19:19 -0800, Rick Johnson wrote: > I agree with your sarcasm. And that's why these types of auto > conversions should be optional. I agree that most times it's more > practical to let python handle the dirty details. But in some cases, > where you need to squeeze out a few more drops of speed juice, you won't > mind the extra trouble. And that's where you call out to a library like numpy, or write a C extension, or use a tool like Numba or Cython to optimise your Python code to use native ints. (Numba is still a work in progress, but Cython is a mature working product.) Or to put it another way... if you want machine ints in Python, the way you get them is something like: from numba import jit @jit def myfunction(): ... The core language doesn't have to support these things when there is a healthy language ecosystem that can do it. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
help me ?
Define 2 lists. The first one must contain the integer values 1, 2 and 3 and the second one the string values a, b and c. Iterate through both lists to create another list that contains all the combinations of the A and B elements. The final list should look like one of the 2 lists: 1. [1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 3c] 2. [a1, a2. a3, b1, b2, b3, c1, c2, c3] BONUS: Make the final list contain all possible combinations : [1a, a1, 1b, b1, 1c, c1, 2a, a2, 2b, b2, 2c, c2, 3a, a3, 3b, b3, 3c, c3] Help me ! -- https://mail.python.org/mailman/listinfo/python-list
Re: help me ?
Am 26.02.18 um 10:40 schrieb sotaro...@gmail.com: Define 2 lists. The first one must contain the integer values 1, 2 and 3 and the second one the string values a, b and c. Iterate through both lists to create another list that contains all the combinations of the A and B elements. The final list should look like one of the 2 lists: 1. [1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 3c Look up list comprehensions. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Sun, 25 Feb 2018 20:22:17 -0800, Rick Johnson wrote: > So of course, speed is not and should not be the > primary concern, but to say that execution speed is of _no_ concern is > quite absurd indeed. I'm pretty sure that nobody here has said that speed is of no concern. Rather, I would argue that the position we're taking is that *in general* Python is fast enough for the sorts of tasks we use it for (except, of course, when it isn't, in which case you have our sympathy, and if we can suggest some solutions we will). Of course we'd take speed optimizations if they were free, but they're never free: - they often require more memory to run; - they usually require more code, which adds to the maintenance burden and increases the interpreter bloat; - and increases the risk of bugs; - somebody has to write, debug, document and maintain the code, and developer time and effort is in short supply; - or the optimization requires changes to the way Python operates; - even if we wanted to make that change, it will break backwards compatibility; - and often what people imagine is a simple optimization (because they haven't tried it) isn't simple at all; - or simply doesn't work; - and most importantly, just saying "Make it go faster!" doesn't work, we actually need to care about the details of *how* to make it faster. (We tried painting Go Faster stripes on the server, and it didn't work.) There's no point suggesting such major changes to Python that requires going back to the drawing board, to Python 0.1 or earlier, and changing the entire execution and memory model of the language. That would just mean we've swapped from a successful, solid, reliable version 3.6 of the language to an untried, unstable, unproven, bug-ridden version 0.1 of New Python. And like New Coke, it won't attract new users (they're already using Javascript or Go or whatever...) and it will alienate existing users (if they wanted Javascript or Go they'd already be using it). There have been at least two (maybe three) attempts to remove the GIL from CPython. They've all turned out to increase complexity by a huge amount, and not actually provide the hoped-for speed increase. Under many common scenarios, the GIL-less CPython actually ran *slower*. (I say "hoped for", but it was more wishful thinking than a rational expectation that removing the GIL would speed things up. I don't believe any of the core developers were surprised that removing the GIL didn't increase speeds much, if at all, and sometimes slowed things down. The believe that the GIL slows Python down is mostly due to a really simplistic understanding of how threading works.) Besides, if you want Python with no GIL so you can run threaded code, why aren't you using IronPython or Jython? Another example: UnladenSwallow. That (mostly) failed to meet expectations because the compiler tool chain being used wasn't mature enough. If somebody were to re-do the project again now, the results might be different. But it turns out that not that many people care enough to make Python faster to actually invest money in it. Everyone wants to just stamp their foot and get it for free. (If it weren't for the EU government funding it, PyPy would probably be languishing in oblivion. Everyone wants a faster Python so long as they don't have to pay for it, or do the work.) A third example: Victor Stinner's FatPython, which seemed really promising in theory, but turned out to not make enough progress fast enough and he lost interest. I have mentioned FatPython here a number of times. All you people who complain that Python is "too slow" and that the core devs ought to do something about it... did any of you volunteer any time to the FatPython project? -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Detection of ultrasonic side channels in mobile devices with Python?
Hi, I would like to know if its possible to detect or decode ultrasonic signals in mobiles devices like Android with Python? For a introduction to ultrasonic tracking, see: https://www.wired.com/2016/11/block-ultrasonic-signals-didnt-know-tracking/ Thank you, Etienne -- Etienne Robillard tkad...@yandex.com https://www.isotopesoftware.ca/ -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Sun, 25 Feb 2018 19:26:12 -0800, Rick Johnson wrote: > On Friday, February 23, 2018 at 8:48:55 PM UTC-6, Steven D'Aprano wrote: > [...] >> Take the Fibonacci double-recursion benchmark. Okay, it tests how well >> your language does at making millions of function calls. Why? > > Because making "millons of function calls" is what software *DOES*! You're right -- I worded that badly, and ended up saying something I didn't really mean. Mea culpa. Of course over the course of the entire program run, most non-trivial programmers will end up having millions (if not billions) of function calls *in total*. But they're not all happening *together*. What I tried to get across is, how often do we use millions of function calls to get one value? By default, Python will only allow one thousand function calls in a stack before raising RecursionError. py> def bad(): ... return bad() ... py> bad() Traceback (most recent call last): File "", line 1, in File "", line 2, in bad [ previous line repeats 998 times ] RecursionError: maximum recursion depth exceeded There's nothing unique about recursion that will trigger this error. Any call stack of more than 1000 function calls will do it. So forget about code getting a million functions deep. The average is probably more like a dozen, perhaps a hundred tops. [...] >> For most application code, executing the function is far more costly >> than the overhead of calling it, > > What? Your functions are only called _once_ during the lifetime of your > code execution? Enough with the leg pulling. Let me explain what I intended to say. Here is a really poor estimate of the overhead of calling a function on my computer, measured in iPython. In [50]: def call(): : pass : In [51]: %timeit call() 100 loops, best of 3: 717 ns per loop So, on my computer, in iPython, it takes 717ns to call a function that does nothing. How long does it take to do the tiniest smidgen of useful work? In [52]: def do_something(): : a = 1 : b = a+1 : return b : In [53]: %timeit do_something() 100 loops, best of 3: 923 ns per loop Only an extra 200 ns! Wow, function calls are expensive!!! (Well, yes, I knew that.) However, let's do some *real* work, not a Mickey Mouse function like do_something(). In [54]: import pyprimes In [55]: %timeit pyprimes.nth_prime(10) 1 loops, best of 3: 2.6 s per loop I happen to know that calling nth_prime ends up making a couple of hundred function calls. (Because I wrote it.) Let's call it 10,000 function calls, to be on the safe side. So 10,000 x 800 nanoseconds is 0.008 second, let's call it 0.01 second, which is just 0.4% of the total cost of calculating the number I was after. (For the record, that's 1299709.) So just a trivial fraction of the total cost of the function. Now I know that this is not always the case. But I maintain that for *most* (but not all) practical, real-world code that actually does something useful, the overhead of calling the functions is *usually* (but not always) low compared to the cost of actually doing whatever it is that the functions do. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: help me ?
Christian Gollwitzer wrote: > Am 26.02.18 um 10:40 schrieb sotaro...@gmail.com: >> Define 2 lists. The first one must contain the integer values 1, 2 and 3 >> and the second one the string values a, b and c. Iterate through both >> lists to create another list that contains all the combinations of the A >> and B elements. The final list should look like one of the 2 lists: 1. >> [1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 3c > > Look up list comprehensions. Hm, this will teach the OP a concise way to rewrite the solution once he has solved the problem with the conventional nested for loops, list.append(), and str.format()... -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 09:22, Steven D'Aprano wrote:
On Sun, 25 Feb 2018 21:19:19 -0800, Rick Johnson wrote:
I agree with your sarcasm. And that's why these types of auto
conversions should be optional. I agree that most times it's more
practical to let python handle the dirty details. But in some cases,
where you need to squeeze out a few more drops of speed juice, you won't
mind the extra trouble.
And that's where you call out to a library like numpy, or write a C
extension, or use a tool like Numba or Cython to optimise your Python
code to use native ints. (Numba is still a work in progress, but Cython
is a mature working product.)
Or to put it another way... if you want machine ints in Python, the way
you get them is something like:
from numba import jit
@jit
def myfunction(): ...
The core language doesn't have to support these things when there is a
healthy language ecosystem that can do it.
Below is the first draft of a Python port of a program to do with random
numbers. (Ported from my language, which in turned ported it from a C
program by George Marsaglia, the random number guy.)
However, running it quickly exhausts the memory in my machine. The
reason is that Python unhelpfully widens all results to bignums as
needed. The code relies on calculations being modulo 2^64.
Note that restricting integer ops to 64 bits probably still won't work,
as I believe the numbers need to be unsigned.
--
Q=0
carry=36243678541
xcng=12367890123456
xs=521288629546311
indx=20632
def refill():
global Q, carry, indx
for i in range(20632):
h = carry & 1
z = ((Q[i]<<41)>>1)+((Q[i]<<39)>>1)+(carry>>1)
carry = (Q[i]>>23)+(Q[i]>>25)+(z>>63)
Q[i] = ~((z<<1)+h)
indx=1
return Q[0]
def start():
global Q, carry, xcng, xs, indx
Q=[0,]*20632
for i in range(20632):
xcng=6906969069 * xcng + 123
xs ^= (xs<<13)
xs ^= (xs>>17)
xs ^= (xs<<43)
Q[i] = xcng + xs
N = 10
for i in range(N):
if indx<20632:
s = Q[indx]
indx+=1
else:
s = refill()
xcng=6906969069 * xcng + 123
xs ^= (xs<<13)
xs ^= (xs>>17)
xs ^= (xs<<43)
x = s+xcng+xs
print ("Does x= 4013566000157423768")
print (" x=",x)
start()
--
(The code performs N iterations of a random number generator. You get
the result expected, ie. x=401...768, when N is a billion.)
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Mon, Feb 26, 2018 at 8:57 PM, Steven D'Aprano wrote: > On Sun, 25 Feb 2018 20:22:17 -0800, Rick Johnson wrote: > >> So of course, speed is not and should not be the >> primary concern, but to say that execution speed is of _no_ concern is >> quite absurd indeed. > > I'm pretty sure that nobody here has said that speed is of no concern. > > Rather, I would argue that the position we're taking is that *in general* > Python is fast enough for the sorts of tasks we use it for (except, of > course, when it isn't, in which case you have our sympathy, and if we can > suggest some solutions we will). > > Of course we'd take speed optimizations if they were free, but they're > never free: > > - they often require more memory to run; > > - they usually require more code, which adds to the maintenance burden > and increases the interpreter bloat; > > - and increases the risk of bugs; > > - somebody has to write, debug, document and maintain the code, > and developer time and effort is in short supply; > > - or the optimization requires changes to the way Python operates; > > - even if we wanted to make that change, it will break backwards > compatibility; > > - and often what people imagine is a simple optimization (because > they haven't tried it) isn't simple at all; > > - or simply doesn't work; > > - and most importantly, just saying "Make it go faster!" doesn't work, > we actually need to care about the details of *how* to make it faster. Or it reduces memory usage, improves performance, and makes things easier on the programmer... but might place unexpected (or unwanted) demands on other implementations of Python. CPython, by virtue of being the "default" Python interpreter, has to be careful of appearing to mandate something. That's why buy-in from other Python implementation teams (I believe Jython was the most notable here) was important in the discussion about the recent compact dict update. Even what looks like a no-brainer still can have unwanted consequences. > (We tried painting Go Faster stripes on the server, and it didn't work.) Steven Middlename D'Aprano! You should know better than that. "It didn't work" is not good enough. What actually happened? Did the stripes smoulder and smoke until you powered down the server? Did the server raise NotImplementedError when you touched it with the paintbrush? Did your boss walk up behind you and hand you a pink slip? Be specific! > There have been at least two (maybe three) attempts to remove the GIL > from CPython. They've all turned out to increase complexity by a huge > amount, and not actually provide the hoped-for speed increase. Under many > common scenarios, the GIL-less CPython actually ran *slower*. > > (I say "hoped for", but it was more wishful thinking than a rational > expectation that removing the GIL would speed things up. I don't believe > any of the core developers were surprised that removing the GIL didn't > increase speeds much, if at all, and sometimes slowed things down. The > believe that the GIL slows Python down is mostly due to a really > simplistic understanding of how threading works.) Removing the GIL from CPython is not about "speeding up" the language or the interpreter, but about improving parallelism. And I think all the core devs understand this (hence the lack of surprise there), but people who call for it often don't. > (If it weren't for the EU government funding it, PyPy would probably be > languishing in oblivion. Everyone wants a faster Python so long as they > don't have to pay for it, or do the work.) Hm, I didn't know the EU was funding PyPy. Okay. Cool. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Mon, Feb 26, 2018 at 10:13 PM, bartc wrote: > Below is the first draft of a Python port of a program to do with random > numbers. (Ported from my language, which in turned ported it from a C > program by George Marsaglia, the random number guy.) > > However, running it quickly exhausts the memory in my machine. The reason is > that Python unhelpfully widens all results to bignums as needed. The code > relies on calculations being modulo 2^64. > > Note that restricting integer ops to 64 bits probably still won't work, as I > believe the numbers need to be unsigned. No, it's because the original implementation assumed integer wrap-around (at least, I think that's what's happening; I haven't analyzed the code in great detail). That means all your integer operations are doing two jobs: the one they claim to, and then a masking to 64 bits signed. That's two abstract operations that happen, due to the nature of the CPU, to work efficiently together. If you don't implement both halves of that in your Python port, you have failed at porting. What if you were porting a program from a 72-bit chip that assumed Binary Coded Decimal? Would you complain that C's integers are badly behaved? And that's without even asking whether a C program that assumes integer wrap-around counts as portable. At least with Python, you have a guarantee that integer operations are going to behave the same way on all compliant implementations of the language. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 23-02-18 02:27, Steven D'Aprano wrote: > Why do you care about the 50 million calls? That's crazy -- the important > thing is *calculating the Fibonacci numbers as efficiently as possible*. No necessarily. David Beazley in his talks sometimes uses an ineffecient algorithm for calculating fibonacci numbers because he needs something that uses the cpu intensively. calculating the fibonacci numbers in that context as efficiently as possible would defeat that purpose. So in a context of a benchmark it is not unreasonable to assume those 50 million calls are the purpose and not calculating the Fibonacci numbers as efficiently as possible. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 11:40, Chris Angelico wrote:
On Mon, Feb 26, 2018 at 10:13 PM, bartc wrote:
Below is the first draft of a Python port of a program to do with random
numbers. (Ported from my language, which in turned ported it from a C
program by George Marsaglia, the random number guy.)
However, running it quickly exhausts the memory in my machine. The reason is
that Python unhelpfully widens all results to bignums as needed. The code
relies on calculations being modulo 2^64.
Note that restricting integer ops to 64 bits probably still won't work, as I
believe the numbers need to be unsigned.
No, it's because the original implementation assumed integer
wrap-around (at least, I think that's what's happening; I haven't
analyzed the code in great detail). That means all your integer
operations are doing two jobs: the one they claim to, and then a
masking to 64 bits signed. That's two abstract operations that happen,
due to the nature of the CPU, to work efficiently together. If you
don't implement both halves of that in your Python port, you have
failed at porting. What if you were porting a program from a 72-bit
chip that assumed Binary Coded Decimal? Would you complain that C's
integers are badly behaved?
And that's without even asking whether a C program that assumes
integer wrap-around counts as portable. At least with Python, you have
a guarantee that integer operations are going to behave the same way
on all compliant implementations of the language.
A C version is given below. (One I may have messed around with, which
I'm not sure works properly. For an original, google for Marsaglia and
KISS64 or SUPRKISS64.)
Most integers are unsigned, which have well-defined overflow in C (they
just wrap as expected). In C, a mixed signed/unsigned op is performed as
unsigned.
-
/* SUPRKISS64.c, period 5*2^1320480*(2^64-1) */
#include
#include
#include "timer.c"
static signed long long Q[20632],
carry=36243678541LL,
xcng=12367890123456LL,
xs=521288629546311LL,
indx=20632;
#define CNG ( xcng=6906969069LL*xcng+123 )
#define XS ( xs^=xs<<13,xs^=xs>>17,xs^=xs<<43 )
#define SUPR ( indx<20632 ? Q[indx++] : refill() )
#define KISS SUPR+CNG+XS
signed long long refill(void) {
int i; signed long long z,h;
for(i=0;i<20632;i++) {
h = (carry&1);
z = ((Q[i]<<41)>>1)+((Q[i]<<39)>>1)+(carry>>1);
carry = (Q[i]>>23)+(Q[i]>>25)+(z>>63);
Q[i] = ~((z<<1)+h);
}
indx=1;
return (Q[0]);
}
int main() {
int i; signed long long x;
for(i=0;i<20632;i++) Q[i]=CNG+XS;
for(i=0;i<10;i++) x=KISS;
printf("Does x=4013566000157423768\n x=%llu.\n",x);
}
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 24-02-18 06:18, Terry Reedy wrote: > On 2/23/2018 11:32 AM, Python wrote: >> >> Doing that would completely fail to accomplish the task of comparing >> the performance of recursive function calls in the two languages, >> which is what the benchmark was designed to do. > > That is an non goal because *languages* do not have clock-time speeds, > only programs written in concrete implementations run on real hardware. > > Why do you think it fair to compare function call times using the > slowest rather than the fastest implementation of Python function > calls? Real Python programmers who are seriously concerned about time > try to use the fastest implementation appropriate to the situation. If the slowest happens to be the standard implementation I see nothing wrong with it. > > > So, no actually, it shouldn't. > > To me, it is 'not using the fasted Python' that fails to make apples > to apples comparisons. > > It has been said here that comparisons should use the same algorithm > doing the much the same work in both implementation. However, a > Python function call does *much more work* than in most languages, > because the meaning of 'function call' is much more flexible in Python > than most languages. So? I don't see what is wrong when is pointed out that all that extra work, that a lot of people don't care about is costing performance. > The usual comparison is like lemons (other language calls) to oranges > (Python language calls, much more flexible). To be fair, the > comparison should be to a Python implementation that either notices or > accepts a declaration that, for instance, fib(n) only needs to pass an > int by position. > > Comparing int addition to python object addition also compares unlike > operations. To compare using the same addition algorithm, one must > use an implementation that can restrict '+' to just int addition. I don't agree, if the performance loss comes from the standard implementation of the language unable of doing that then people testing languages shouldn't be burdened with going to search for some implementation that can. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 12:06, Antoon Pardon wrote: On 23-02-18 02:27, Steven D'Aprano wrote: Why do you care about the 50 million calls? That's crazy -- the important thing is *calculating the Fibonacci numbers as efficiently as possible*. No necessarily. David Beazley in his talks sometimes uses an ineffecient algorithm for calculating fibonacci numbers because he needs something that uses the cpu intensively. calculating the fibonacci numbers in that context as efficiently as possible would defeat that purpose. So in a context of a benchmark it is not unreasonable to assume those 50 million calls are the purpose and not calculating the Fibonacci numbers as efficiently as possible. I don't think Steven is ever going to concede this point. Because Python performs badly compared to Julia or C, and it's not possible to conveniently offload the task to some fast library because it only uses a handful of primitive byte-codes. (I have the same trouble with my own interpreted language. Although somewhat brisker than CPython, it will always be much slower than a C-like language on such micro-benchmarks. But I accept that; I don't have an army of people working on acceleration projects and tracing JIT compilers. To those people however, such a benchmark can be a useful yardstick of progress.) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Mon, 26 Feb 2018 22:34:00 +1100, Chris Angelico wrote: [...] >> (We tried painting Go Faster stripes on the server, and it didn't >> work.) > > Steven Middlename D'Aprano! You should know better than that. "It didn't > work" is not good enough. What actually happened? A truck crashed into the power pole outside our building and the power went off. Then the UPS ran down and the server crashed. Clearly this is a bug in the go-faster stripes. We tried reporting it: Expected behaviour: server goes faster. Actual behaviour: a truck crashes into the power pole across the street and the lights go out. but the maintainers closed the issue "Works for me". Very disappointed! [...] > Removing the GIL from CPython is not about "speeding up" the language or > the interpreter, but about improving parallelism. It is about speeding up threaded code which is CPU-bound. I call that speeding up the language :-) Hypothetically, it could also speed up even unthreaded code. Some language features could internally launch threads and operate using implicit parallelism. For instance, map(func, seq) could run in parallel in separate threads whenever Python knew that func had no side-effects, say for built-ins. There are also parallel algorithms for bignum arithmetic. If threads were quick enough, I'm sure we could come up with many more examples. Hypothetically speaking. (In practice, the old hope that parallel computing would speed everything up turned out to be flawed. Relatively few tasks are embarrassingly parallel, most have sequential bottlenecks, and many are inherently sequential.) >> (If it weren't for the EU government funding it, PyPy would probably be >> languishing in oblivion. Everyone wants a faster Python so long as they >> don't have to pay for it, or do the work.) > > Hm, I didn't know the EU was funding PyPy. Okay. Cool. I don't know if they still receive funding, but I know that PyPy really only got going in a big way when they got a grant from the EU. I think it paid for at least one developer to work on it full time for a year. DuckDuckGo is probably your friend if you care about the details. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 2/26/18 7:13 AM, bartc wrote:
On 26/02/2018 11:40, Chris Angelico wrote:
On Mon, Feb 26, 2018 at 10:13 PM, bartc wrote:
Below is the first draft of a Python port of a program to do with
random
numbers. (Ported from my language, which in turned ported it from a C
program by George Marsaglia, the random number guy.)
However, running it quickly exhausts the memory in my machine. The
reason is
that Python unhelpfully widens all results to bignums as needed. The
code
relies on calculations being modulo 2^64.
Note that restricting integer ops to 64 bits probably still won't
work, as I
believe the numbers need to be unsigned.
No, it's because the original implementation assumed integer
wrap-around (at least, I think that's what's happening; I haven't
analyzed the code in great detail). That means all your integer
operations are doing two jobs: the one they claim to, and then a
masking to 64 bits signed. That's two abstract operations that happen,
due to the nature of the CPU, to work efficiently together. If you
don't implement both halves of that in your Python port, you have
failed at porting. What if you were porting a program from a 72-bit
chip that assumed Binary Coded Decimal? Would you complain that C's
integers are badly behaved?
And that's without even asking whether a C program that assumes
integer wrap-around counts as portable. At least with Python, you have
a guarantee that integer operations are going to behave the same way
on all compliant implementations of the language.
A C version is given below. (One I may have messed around with, which
I'm not sure works properly. For an original, google for Marsaglia and
KISS64 or SUPRKISS64.)
Most integers are unsigned, which have well-defined overflow in C
(they just wrap as expected). In C, a mixed signed/unsigned op is
performed as unsigned.
-
/* SUPRKISS64.c, period 5*2^1320480*(2^64-1) */
#include
#include
#include "timer.c"
static signed long long Q[20632],
carry=36243678541LL,
xcng=12367890123456LL,
xs=521288629546311LL,
indx=20632;
#define CNG ( xcng=6906969069LL*xcng+123 )
#define XS ( xs^=xs<<13,xs^=xs>>17,xs^=xs<<43 )
#define SUPR ( indx<20632 ? Q[indx++] : refill() )
#define KISS SUPR+CNG+XS
signed long long refill(void) {
int i; signed long long z,h;
for(i=0;i<20632;i++) {
h = (carry&1);
z = ((Q[i]<<41)>>1)+((Q[i]<<39)>>1)+(carry>>1);
carry = (Q[i]>>23)+(Q[i]>>25)+(z>>63);
Q[i] = ~((z<<1)+h);
}
indx=1;
return (Q[0]);
}
int main() {
int i; signed long long x;
for(i=0;i<20632;i++) Q[i]=CNG+XS;
for(i=0;i<10;i++) x=KISS;
printf("Does x=4013566000157423768\n x=%llu.\n",x);
}
With proper 64-bit masking (def only64(x): return x &
0x), the Python version produces the correct answer
using a reasonable amount of memory. Well, once you notice that the
Python code had N=1e5, and the C code had N=1e9 :) If you want to
experiment, with N=1e5, the final number should be 5255210926702073855.
Also, I note that you said, "Most integers are unsigned", but the C code
has them all declared as signed? It doesn't seem to have mattered to
your result, but I'm not an expert on C portability guarantees.
--Ned.
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/18 13:42, Ned Batchelder wrote: Also, I note that you said, "Most integers are unsigned", but the C code has them all declared as signed? It doesn't seem to have mattered to your result, but I'm not an expert on C portability guarantees. C explicitly leaves the behaviour of signed arithmetic overflow undefined, so you have no portability guarantees there. -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 13:42, Ned Batchelder wrote: On 2/26/18 7:13 AM, bartc wrote: A C version is given below. (One I may have messed around with, which I'm not sure works properly. For an original, google for Marsaglia and KISS64 or SUPRKISS64.) Most integers are unsigned, which have well-defined overflow in C With proper 64-bit masking (def only64(x): return x & 0x), the Python version produces the correct answer using a reasonable amount of memory. I did try sometime like that, but I must have missed something because I didn't get quite the same results as a working version. And with interpreted code, you tend not to test using loops of a billion iterations. Well, once you notice that the Python code had N=1e5, and the C code had N=1e9 :) If you want to experiment, with N=1e5, the final number should be 5255210926702073855. OK, I'll try that. Also, I note that you said, "Most integers are unsigned", but the C code has them all declared as signed? It doesn't seem to have mattered to your result, but I'm not an expert on C portability guarantees. The C code I first pasted used 'unsigned', but the main program logic wasn't right, and I found another version that looked better. That one used 'signed' for some reason, which I completely missed. Even if with C it works with either, the signed version might have 'undefined behaviour'. As said, google for the original; the ones I can see have 'unsigned'. But I can also see a Fortran version that just uses 'integer*8', which I believe is signed 64-bit. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, Feb 27, 2018 at 12:03 AM, Steven D'Aprano wrote: > On Mon, 26 Feb 2018 22:34:00 +1100, Chris Angelico wrote: >> Removing the GIL from CPython is not about "speeding up" the language or >> the interpreter, but about improving parallelism. > > It is about speeding up threaded code which is CPU-bound. I call that > speeding up the language :-) > > Hypothetically, it could also speed up even unthreaded code. In its purest sense, no, it cannot speed up unthreaded code. Since many Python programs are single-threaded, this restricts the benefits to those programs which can actually parallelize, but the penalties (mainly overhead) usually apply to all programs. > Some language features could internally launch threads and operate using > implicit parallelism. For instance, map(func, seq) could run in parallel > in separate threads whenever Python knew that func had no side-effects, > say for built-ins. There are also parallel algorithms for bignum > arithmetic. If threads were quick enough, I'm sure we could come up with > many more examples. Hypothetically speaking. These are ways of converting single-threaded code into multi-threaded code, which could then benefit from parallelization, which can reduce wall-time for the process as a whole. > (In practice, the old hope that parallel computing would speed everything > up turned out to be flawed. Relatively few tasks are embarrassingly > parallel, most have sequential bottlenecks, and many are inherently > sequential.) Exactly; and it goes further: many modern programmers do not think in terms of parallelism. Even event-driven code takes a bit of getting your head around. Why are async functions preferred over threads? It's not JUST because today's OSes have thread count limits far below socket limits - if that were the sole justification, we wouldn't have language-level support for async/await - it'd be a niche technique used only in the highest-throughput applications. No, async functions are preferred because they *reduce the points where context switching can occur*, thus making it easier *for the programmer*. If there were a way for the language to automatically run things in parallel, the biggest risk is that the programmer who wrote it can no longer understand it. >>> (If it weren't for the EU government funding it, PyPy would probably be >>> languishing in oblivion. Everyone wants a faster Python so long as they >>> don't have to pay for it, or do the work.) >> >> Hm, I didn't know the EU was funding PyPy. Okay. Cool. > > I don't know if they still receive funding, but I know that PyPy really > only got going in a big way when they got a grant from the EU. I think it > paid for at least one developer to work on it full time for a year. > DuckDuckGo is probably your friend if you care about the details. Meh, I don't care that much about what the EU does or doesn't fund. Was just a mild case of curiosity. I had about as much interest as my savings account accrued last year. I'm glad _someone_ funded PyPy, anyhow. It's a great demonstration of what can be done. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: help me ?
Define 2 lists. ... [...] Help me ! Sounds like homework. Have you tried anything? Does it work? -- https://mail.python.org/mailman/listinfo/python-list
Is there are good DRY fix for this painful design pattern?
I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on There's a lot of tedious boilerplate repetition in this, and to add insult to injury the class is still under active development with an unstable API, so every time I change one of the parameters, or add a new one, I have to change it in over a dozen places. Is there a good fix for this to reduce the amount of boilerplate? Thanks, -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 14:34, Chris Angelico wrote: I'm glad _someone_ funded PyPy, anyhow. It's a great demonstration of what can be done. And it's good that /someone/ at least understands how PyPy works, as I don't think many people do. Apparently it doesn't optimise 'hot paths' within a Python program, but optimises hot paths in the special Python interpreter. One written in [R]Python. Or something... -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 26-02-18 15:41, Steven D'Aprano wrote: > I have a class with a large number of parameters (about ten) assigned in > `__init__`. The class then has a number of methods which accept > *optional* arguments with the same names as the constructor/initialiser > parameters. If those arguments are None, the defaults are taken from the > instance attributes. > > An example might be something like this: > > > class Foo: > def __init__(self, bashful, doc, dopey, grumpy, >happy, sleepy, sneezy): > self.bashful = bashful # etc > > def spam(self, bashful=None, doc=None, dopey=None, >grumpy=None, happy=None, sleepy=None, >sneezy=None): > if bashful is None: > bashful = self.bashful > if doc is None: > doc = self.doc > if dopey is None: > dopey = self.dopey > if grumpy is None: > grumpy = self.grumpy > if happy is None: > happy = self.happy > if sleepy is None: > sleepy = self.sleepy > if sneezy is None: > sneezy = self.sneezy > # now do the real work... Just throwing out an idea, but would it be possible to do what you want with a decorator? You let the wrapper do this kind of control and then pass control to the actual function with all values filled in? I'll try to work something out later. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 14:04, bartc wrote:
On 26/02/2018 13:42, Ned Batchelder wrote:
Well, once you notice that the
Python code had N=1e5, and the C code had N=1e9 :) If you want to
experiment, with N=1e5, the final number should be 5255210926702073855.
OK, I'll try that.
I have that Python version working now. It's necessary to apply that
masking function to wherever numbers can get bigger.
I don't know how long a 1-billion loop will take, but a 10-million loop
took 46 seconds on Python 3.6, and 21 seconds on PyPy 2.7 from a couple
of years ago. (And on Windows, which has a somewhat slower CPython than
Linux.)
Result should be x=11240129907685265998.
By comparison, the C version compiled with -O3 took 0.11 seconds.
(The C version I posted will work, if adjusted to a 1000 loop, but
you have to change 'signed' to 'unsigned'. Apparently they weren't
interchangeable after all. I've no idea why I used 'signed' there.
That version is rather cryptic, but it can be better written and without
the macros, and it will run just as fast. (Marsaglia may have been hot
with random number routines, but his C could have done with some work...)
My interpreter, using 64-bit numbers, managed 4.8 seconds. But unsigned
arithmetic, which is uncommon, is not accelerated.)
---
Q=0
carry=36243678541
xcng=12367890123456
xs=521288629546311
indx=20632
def i64(x): return x & 0x
def refill():
global Q, carry, indx
for i in range(20632):
h = carry & 1
z = i64(( i64((Q[i]<<41))>>1)+(i64((Q[i]<<39))>>1)+(carry>>1))
carry = i64((Q[i]>>23)+(Q[i]>>25)+(z>>63))
Q[i] = i64(~(i64(i64(z<<1)+h)))
indx=1
return Q[0]
def start():
global Q, carry, xcng, xs, indx
Q=[0,]*20632
for i in range(20632):
xcng=i64(6906969069 * xcng + 123)
xs = i64(xs ^ (xs<<13))
xs = i64(xs ^ (xs>>17))
xs = i64(xs ^ (xs<<43))
Q[i] = i64(xcng + xs)
N = 1000
for i in range(N):
if indx<20632:
s = Q[indx]
indx+=1
else:
s = refill()
xcng=i64(6906969069 * xcng + 123)
xs = i64(xs ^ (xs<<13))
xs = i64(xs ^ (xs>>17))
xs = i64(xs ^ (xs<<43))
x = i64(s+xcng+xs)
print ("Does x= 4013566000157423768")
print (" x=",x)
start()
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Tue, Feb 27, 2018 at 1:41 AM, Steven D'Aprano wrote: > I have a class with a large number of parameters (about ten) assigned in > `__init__`. The class then has a number of methods which accept > *optional* arguments with the same names as the constructor/initialiser > parameters. If those arguments are None, the defaults are taken from the > instance attributes. I have a very similar problem in a project of mine, and it's currently an unsolved problem. My current best theory is to basically just accept **kwargs for everything (so there's no None defaults), and then use chain_map for all lookups. Seems inefficient, but at least it'd be functional. Haven't implemented it though. If you get any better ideas, I'll be reading them too. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, Feb 27, 2018 at 2:02 AM, bartc wrote: > On 26/02/2018 14:04, bartc wrote: >> >> On 26/02/2018 13:42, Ned Batchelder wrote: > > >> Well, once you notice that the >>> >>> Python code had N=1e5, and the C code had N=1e9 :) If you want to >>> experiment, with N=1e5, the final number should be 5255210926702073855. >> >> >> OK, I'll try that. > > > I have that Python version working now. It's necessary to apply that masking > function to wherever numbers can get bigger. > > I don't know how long a 1-billion loop will take, but a 10-million loop took > 46 seconds on Python 3.6, and 21 seconds on PyPy 2.7 from a couple of years > ago. (And on Windows, which has a somewhat slower CPython than Linux.) You're still reimplementing the C code in Python, which is inefficient. Have you considered going back to the *actual algorithm* and implementing that idiomatically in Python? I think you'll find that (a) the code is more readable, and (b) the run time is much lower. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
Le 26/02/18 à 15:41, Steven D'Aprano a écrit : I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on There's a lot of tedious boilerplate repetition in this, and to add insult to injury the class is still under active development with an unstable API, so every time I change one of the parameters, or add a new one, I have to change it in over a dozen places. Is there a good fix for this to reduce the amount of boilerplate? Thanks, Maybe something like this: def config(self, **kwargs): for key, value in kwargs.items(): if value is not None: setattr(self, key, value) def spam(self, **kwargs): self.config(kwargs) Vincent -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 26-02-18 15:41, Steven D'Aprano wrote:
> I have a class with a large number of parameters (about ten) assigned in
> `__init__`. The class then has a number of methods which accept
> *optional* arguments with the same names as the constructor/initialiser
> parameters. If those arguments are None, the defaults are taken from the
> instance attributes.
>
This is what I came up with at short notice:
def passthrough(func):
def wrapper(self, **kwds):
for argn in "bashful doc dopey grumpy happy sleepy sneezy".split():
if argn not in kwds:
kwds[argn] = getattr(self, argn)
func(self, **kwds)
return wrapper
class Foo:
def __init__(self, bashful, doc, dopey, grumpy,
happy, sleepy, sneezy):
self.bashful = bashful
self.doc = doc
self.dopey = dopey
self.grumpy = grumpy
self.happy = happy
self.sleepy = sleepy
self.sneezy = sneezy
@passthrough
def spam(self, bashful=None, doc=None, dopey=None,
grumpy=None, happy=None, sleepy=None,
sneezy=None):
print(bashful, doc, dopey, grumpy, happy, sleepy, sneezy)
foo = Foo('a', 'b', 'c', 'd', 'e', 'f', 'g')
foo.spam(happy = 1) # prints => a b c d 1 f g
--
https://mail.python.org/mailman/listinfo/python-list
Re: For Loop Dilema [python-list]
On Sun, Feb 25, 2018 at 8:05 PM, INADA Naoki wrote: > https://docs.python.org/3.6/library/itertools.html#itertools.product I don't see how you would use itertools.product to do what the OP asked for. You could use itertools.chain.from_iterable, though: py> names = ['Jack', 'Susan'] py> list(chain.from_iterable(names)) ['J', 'a', 'c', 'k', 'S', 'u', 's', 'a', 'n'] -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 26/02/18 14:41, Steven D'Aprano wrote: I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on There's a lot of tedious boilerplate repetition in this, and to add insult to injury the class is still under active development with an unstable API, so every time I change one of the parameters, or add a new one, I have to change it in over a dozen places. Is there a good fix for this to reduce the amount of boilerplate? You could use dicts? Untested code: class Foo: def __init__(self, **kwds): self.defaults = kwds def spam(self, **kwds): vars = self.defaults.copy() vars.update(kwds) # Do the work with vars['bashful'] etc -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 2/26/18 10:09 AM, Chris Angelico wrote: On Tue, Feb 27, 2018 at 2:02 AM, bartc wrote: On 26/02/2018 14:04, bartc wrote: On 26/02/2018 13:42, Ned Batchelder wrote: Well, once you notice that the Python code had N=1e5, and the C code had N=1e9 :) If you want to experiment, with N=1e5, the final number should be 5255210926702073855. OK, I'll try that. I have that Python version working now. It's necessary to apply that masking function to wherever numbers can get bigger. I don't know how long a 1-billion loop will take, but a 10-million loop took 46 seconds on Python 3.6, and 21 seconds on PyPy 2.7 from a couple of years ago. (And on Windows, which has a somewhat slower CPython than Linux.) You're still reimplementing the C code in Python, which is inefficient. Have you considered going back to the *actual algorithm* and implementing that idiomatically in Python? I think you'll find that (a) the code is more readable, and (b) the run time is much lower. Don't bother: C will win this kind of race every time. --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 02/26/2018 07:28 AM, Rhodri James wrote: On 26/02/18 14:41, Steven D'Aprano wrote: I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on There's a lot of tedious boilerplate repetition in this, and to add insult to injury the class is still under active development with an unstable API, so every time I change one of the parameters, or add a new one, I have to change it in over a dozen places. Is there a good fix for this to reduce the amount of boilerplate? You could use dicts? Untested code: class Foo: def __init__(self, **kwds): self.defaults = kwds def spam(self, **kwds): vars = self.defaults.copy() vars.update(kwds) # Do the work with vars['bashful'] etc That's what I was about to suggest (and equally fail to test). It at least segregates the set of "instance variables I want to have auto-default behavior" from the rest of them. Yeah, **kwargs is clumsy, but it gets the job done. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 15:09, Chris Angelico wrote: On Tue, Feb 27, 2018 at 2:02 AM, bartc wrote: On 26/02/2018 14:04, bartc wrote: On 26/02/2018 13:42, Ned Batchelder wrote: Well, once you notice that the Python code had N=1e5, and the C code had N=1e9 :) If you want to experiment, with N=1e5, the final number should be 5255210926702073855. OK, I'll try that. I have that Python version working now. It's necessary to apply that masking function to wherever numbers can get bigger. I don't know how long a 1-billion loop will take, but a 10-million loop took 46 seconds on Python 3.6, and 21 seconds on PyPy 2.7 from a couple of years ago. (And on Windows, which has a somewhat slower CPython than Linux.) You're still reimplementing the C code in Python, which is inefficient. Have you considered going back to the *actual algorithm* and implementing that idiomatically in Python? I think you'll find that (a) the code is more readable, and (b) the run time is much lower. Do you really think that? The algorithm seems to require this sequence of calculations to be gone through. Otherwise anything more efficient can also be done in any language. So how would idiomatic Python help, by seeing if such a routine already exists in some library? That wouldn't be playing the game. If it helps, I remember playing with a version in Algol 68 Genie (interpreted). Still buggy as the output is different, it does about the same amount of work. A 10-million loop would take an estimated 1000 seconds, on the same machine that CPython took 46 seconds. So four magnitudes slower than C. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Sun, 25 Feb 2018 18:52:14 -0500, Terry Reedy wrote: [...] > numpy has a matrix multiply function and now the '@' matrix multiply > operator. Yes, but what I was wondering is whether there's a faster way of multiplying each row (1x3) of a matrix by another matrix (3x3), compared to looping through the matrix row by row as shown in the code. Perhaps I'm not understanding how to use the broadcasting features of `matmul`. -- Seb -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
Steven D'Aprano wrote:
> I have a class with a large number of parameters (about ten) assigned in
> `__init__`. The class then has a number of methods which accept
> *optional* arguments with the same names as the constructor/initialiser
> parameters. If those arguments are None, the defaults are taken from the
> instance attributes.
>
> An example might be something like this:
>
>
> class Foo:
> def __init__(self, bashful, doc, dopey, grumpy,
>happy, sleepy, sneezy):
> self.bashful = bashful # etc
>
> def spam(self, bashful=None, doc=None, dopey=None,
>grumpy=None, happy=None, sleepy=None,
>sneezy=None):
> if bashful is None:
> bashful = self.bashful
> if doc is None:
> doc = self.doc
> if dopey is None:
> dopey = self.dopey
> if grumpy is None:
> grumpy = self.grumpy
> if happy is None:
> happy = self.happy
> if sleepy is None:
> sleepy = self.sleepy
> if sneezy is None:
> sneezy = self.sneezy
> # now do the real work...
>
> def eggs(self, bashful=None, # etc...
>):
> if bashful is None:
> bashful = self.bashful
> # and so on
>
>
> There's a lot of tedious boilerplate repetition in this, and to add
> insult to injury the class is still under active development with an
> unstable API, so every time I change one of the parameters, or add a new
> one, I have to change it in over a dozen places.
>
> Is there a good fix for this to reduce the amount of boilerplate?
I have not yet looked into dataclasses. Don't they handle the __init__()
part? Anyway, here's my attempt to make spam() less spammy:
$ cat attrs_to_args_decorator.py
import functools
import inspect
def add_defaults(f):
argnames = inspect.getfullargspec(f).args[1:]
@functools.wraps(f)
def wrapper(self, *args, **kw):
args = [
getattr(self, name) if value is None else value
for name, value in zip(argnames, args)
]
for name in argnames[len(args):]:
if name not in kw or kw[name] is None:
kw[name] = getattr(self, name)
return f(self, *args, **kw)
return wrapper
def update_attrs(kw):
self = kw.pop("self")
for name, value in kw.items():
setattr(self, name, value)
class Foo:
def __init__(self, bashful, doc, dopey, grumpy,
happy, sleepy, sneezy):
update_attrs(locals())
@add_defaults
def spam(self, bashful=None, doc=None, dopey=None,
grumpy=None, happy=None, sleepy=None,
sneezy=None):
return "{}-{}-{}".format(bashful, doc, sneezy)
def __repr__(self):
return "Foo({})".format(
", ".join(
"{}={!r}".format(*pair)
for pair in sorted(self.__dict__.items())
)
)
if __name__ == "__main__":
foo = Foo("bashful", "doc", "dopey", "grumpy", "happy", "sleepy",
"sneezy")
print(foo)
print(foo.spam())
print(foo.spam(bashful="BASHFUL"))
print(foo.spam(bashful="BASHFUL", doc="DOC"))
print(foo.spam("BASHFUL"))
print(foo.spam("BASHFUL", "DOC"))
print(foo.spam("BASHFUL", "DOC", sneezy="SNEEZY"))
$ python3 attrs_to_args_decorator.py
Foo(bashful='bashful', doc='doc', dopey='dopey', grumpy='grumpy',
happy='happy', sleepy='sleepy', sneezy='sneezy')
bashful-doc-sneezy
BASHFUL-doc-sneezy
BASHFUL-DOC-sneezy
BASHFUL-doc-sneezy
BASHFUL-DOC-sneezy
BASHFUL-DOC-SNEEZY
$
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
bartc writes: > A C version is given below. (One I may have messed around with, which > I'm not sure works properly. For an original, google for Marsaglia and > KISS64 or SUPRKISS64.) The version I know uses unsigned integers. Did you change then to signed? For a Python version, go back to the original C and work from there. -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Mon, Feb 26, 2018 at 9:53 AM, Seb wrote: > On Sun, 25 Feb 2018 18:52:14 -0500, > Terry Reedy wrote: > > [...] > >> numpy has a matrix multiply function and now the '@' matrix multiply >> operator. > > Yes, but what I was wondering is whether there's a faster way of > multiplying each row (1x3) of a matrix by another matrix (3x3), compared > to looping through the matrix row by row as shown in the code. Perhaps > I'm not understanding how to use the broadcasting features of `matmul`. >From the matmul documentation: "If either argument is N-D, N > 2, it is treated as a stack of matrices residing in the last two indexes and broadcast accordingly." So you probably want your uvw array to be 1000x1x3 instead of 1000x3. Then the result of multiplying them should be 1000 1x3 matrices. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Monday, February 26, 2018 at 3:59:40 AM UTC-6, Steven D'Aprano wrote: > On Sun, 25 Feb 2018 20:22:17 -0800, Rick Johnson wrote: > (We tried painting Go Faster stripes on the server, and it > didn't work.) Well of course the server won't work after you drip water- based paint all over the circuit boards. Way to go! In the future i would suggest draping some old linens over the electrical bits before throwing your paint around (just be sure to consult with the wife before rummaging through her linen closet, as men have been known to meet an untimely demise for doing much less) . And might i suggest some skulls and faux flames? Hey, it may not make the thing go any faster, but it will surely make the other kids jealous. There's no reason why apple and microsoft should be allowed to suck up every last merchandising dollar. > There's no point suggesting such major changes to Python > that requires going back to the drawing board, to Python > 0.1 or earlier, and changing the entire execution and > memory model of the language. Absolutely not. Probably too late for that now. I'm only suggesting that we, when at all possible, provide a switch to disable these lazy-features in those rare cases when they do create bottlenecks. Lazy iterators and dict-views are great ideas, but i'm sure there are instances which prove them to be PITA and even a bottleneck. -- https://mail.python.org/mailman/listinfo/python-list
Re: For Loop Dilema [python-list]
On Monday, February 26, 2018 at 8:51:35 PM UTC+5:30, Ian wrote:
> On Sun, Feb 25, 2018 at 8:05 PM, INADA Naoki wrote:
> > https://docs.python.org/3.6/library/itertools.html#itertools.product
>
> I don't see how you would use itertools.product to do what the OP
> asked for. You could use itertools.chain.from_iterable, though:
>
> py> names = ['Jack', 'Susan']
> py> list(chain.from_iterable(names))
> ['J', 'a', 'c', 'k', 'S', 'u', 's', 'a', 'n']
if you want to access a dict per say insisde a list or maybe some data i.e deep
nested in some mixture of ds wont it be relevant to use a single for statement
to query the data out.(Just a thought)
Students = [{"name":John,"age":16},{"name":Maria,"age":18}]
for info in student in Students:
print(student[info])
--
https://mail.python.org/mailman/listinfo/python-list
Re: For Loop Dilema [python-list]
On Monday, February 26, 2018 at 6:20:06 AM UTC+5:30, Ian wrote: > On Sun, Feb 25, 2018 at 11:19 AM, wrote: > > Why we don’t use: > > > > for _ in _ in _ > > > > Instead of > > > > for _ in _: > > for _ in _: > > > > Ex: > > > > Names = ["Arya","Pupun"] > > > > for name in Names: > >for c in name: > >print(c) > > > > instead use: > > > > for c in name in Names: > > print(c) > > It doesn't seem very intuitive (doesn't follow proper English > phrasing, for instance) and I don't think it's a common enough > situation to warrant adding a special syntax for it. But if you really > want it, you could use something like this: > > def double_for(iterable): > for outer in iterable: > yield from outer > > for c in double_for(Names): > print(c) > > But I don't think this is any clearer than making the loops explicit. Thank you. -- https://mail.python.org/mailman/listinfo/python-list
Re: For Loop Dilema [python-list]
On Monday, February 26, 2018 at 6:23:24 AM UTC+5:30, Rick Johnson wrote: > On Sunday, February 25, 2018 at 12:19:56 PM UTC-6, arya.ku...@gmail.com wrote: > > > Ex: > > > > Names = ["Arya","Pupun"] > > > > for name in Names: > >for c in name: > >print(c) > > > > instead use: > > > > for c in name in Names: > > print(c) > > Hmm. Why stop there? > > bit = ["kibbles"] > bits = [bit, bit] > bitts = [bits, bits] > for kibbles in bit in bits in bitts: > do_something(kibbles) My thought exactly.. :) -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Mon, Feb 26, 2018 at 8:06 AM, Chris Angelico wrote: > On Tue, Feb 27, 2018 at 1:41 AM, Steven D'Aprano > wrote: >> I have a class with a large number of parameters (about ten) assigned in >> `__init__`. The class then has a number of methods which accept >> *optional* arguments with the same names as the constructor/initialiser >> parameters. If those arguments are None, the defaults are taken from the >> instance attributes. > > I have a very similar problem in a project of mine, and it's currently > an unsolved problem. My current best theory is to basically just > accept **kwargs for everything (so there's no None defaults), and then > use chain_map for all lookups. Seems inefficient, but at least it'd be > functional. Haven't implemented it though. > > If you get any better ideas, I'll be reading them too. I would be tempted to take this and then formalize it into a Context class that would be passed to the various methods and initializers rather than passing all the sundry arguments directly. Similar to what the decimal class uses although perhaps without the idea of a global context (which just leads to trouble down the road). -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 17:05, Ben Bacarisse wrote: bartc writes: A C version is given below. (One I may have messed around with, which I'm not sure works properly. For an original, google for Marsaglia and KISS64 or SUPRKISS64.) The version I know uses unsigned integers. Did you change then to signed? Yeah, I don't know what I was doing with that version. For a Python version, go back to the original C and work from there. The original C makes confusing use of macros and comma operators. A version without macros or comma expressions is here (tweaked from generated C): https://pastebin.com/raw/k4jFK5TN This runs the 1-billion loop in 6 seconds. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Questions about `locals` builtin
Hi,
I am a little bit confused with `locals` builtin in these moments:
1. The documentation says that _free varaibles_ are returned, which seems
incorrect description. In my mind the term free variable refers to
variables used in a function that are not local variables nor parameters of
that function.
In docs: "Update and return a dictionary representing the current local
symbol table. Free variables are returned by `locals()` when it is called
in function blocks, but not in class blocks."
>>> def func():
b = 12
print(locals(), ' : ', id(locals))
c = 13
print(locals(), ' : ', id(locals))
>>> print("Free variables: ", func.__code__.co_names)
>>> print("Local variables: ", func.__code__.co_varnames)
Free variables: ('print', 'locals', 'id')
Local variables: ('b', 'c')
2. The documentation has a note that "The contents of this dictionary
should not be modified". Which implies that it is a read only mapping. So
the question why it is `dict` instead of `types.MappingProxyType`?
3. There is one more moment: local variables had been determined when
function was compiled. But `locals` returns _some_ current runtime copy. I
find this also confusing:
>>> def func1():
loc = locals()
b = 12
return loc
>>> def func2():
b = 12
loc = locals()
return loc
>>> func1()
{ }
>>> func2()
{'b': 12}
With kind regards,
-gdg
--
https://mail.python.org/mailman/listinfo/python-list
Implicit conversion to str in str.join method?
Currently `str.join` raises `TypeError` if there are any non-string values in iterable, including `bytes` objects. Is it an awful idea to implicitly _cast_ elements in iterable to their `str` or `repr` representation? Was this question adressed before? As for me there are two valid points: On one hand "Explicit is beter than implicit" and on the other "Practicality beats purity". May be I'm the only one who finds that it is rather boring to write somehting like: ", ".join(str(i) for i in iterable) when iterable contains non-string values, instead of: ", ".join(iterable) I don't know how much overhead will yield for the case when iterable contains only strings...and if it is possible to optimize this case. This change is backward compatible, but as I wrote above, it can be perceived badly and yield a huge overhead for general case that it is not even worth discussing. What do you think? With kind regards, -gdg -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Monday, February 26, 2018 at 4:39:22 AM UTC-6, Steven D'Aprano wrote: > On Sun, 25 Feb 2018 19:26:12 -0800, Rick Johnson wrote: > > > On Friday, February 23, 2018 at 8:48:55 PM UTC-6, Steven D'Aprano wrote: > > [...] > > > Take the Fibonacci double-recursion benchmark. Okay, it > > > tests how well your language does at making millions of > > > function calls. Why? > > > > Because making "millons of function calls" is what > > software *DOES*! > > You're right -- I worded that badly, and ended up saying > something I didn't really mean. Mea culpa. Of course over > the course of the entire program run, most non-trivial > programmers will end up having millions (if not billions) > of function calls *in total*. But they're not all happening > *together*. So what? Latency is latency. And whether it occurs over the course of one heavily recursive algorithm that constitutes the depth and breadth of an entire program (a la fib()), or it is the incremental cumulative consequence of the entire program execution, the fact remains that function call overhead contributes to a significant portion of the latency inherent in some trivial, and *ALL* non-trivial, modern software. > > What I tried to get across is, how often do we use millions > of function calls to get one value? All the time! For, in the abstract, the fundamental function of software is to extract a "value" from one or more inputs -- is it not? So, whether that "value" be: (1) a document in an office productivity suite, or (2) a flight path for a mars lander or a jumbo jet or child's quad-copter, or (3) a modified image in a image processing program, or (4) a design drawing or technical schematic for a mars rover bot, or (5) an HD quality streaming porno flick, or yes, (6) even a recursive function which unwittingly underscores some of the less attractive aspects of the Python language (aka: function call overhead!) -- all of these examples, and countless others, extract a value from inputs. Of course, much like physical traits i suppose, some "values" are more desirable than others. > > By default, Python will only allow one thousand function > calls in a stack before raising RecursionError. By _default_, yes. > > py> def bad(): > ... return bad() > ... > py> bad() > Traceback (most recent call last): > File "", line 1, in > File "", line 2, in bad > [ previous line repeats 998 times ] > RecursionError: maximum recursion depth exceeded > > > There's nothing unique about recursion that will trigger > this error. Any call stack of more than 1000 function calls > will do it. So forget about code getting a million > functions deep. The average is probably more like a dozen, > perhaps a hundred tops. I'm very glad you brought this up, because it proves my main point that, Python's maximum recursion error is one feature they got _right_. If you don't like the default recursion limit of 1000, or maybe 1000 recursions is breaking a valid recursive algorithm, you can set the recursion limit to whatever you like... ## Python 2 ## >>> import sys >>> sys.setrecursionlimit >>> sys.setrecursionlimit(10) >>> count = 1 >>> def bad(): ... global count ... print count ... count += 1 ... bad() ... >>> bad() 1 2 3 4 5 6 7 8 9 Traceback (most recent call last): File "", line 1, in File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad File "", line 5, in bad RuntimeError: maximum recursion depth exceeded It is however unfortunate that we get slamed with N recursive exception messages :-(, but hey, at least we can modify the maximum recursion limit! It's a step in the right direction. :-) > [...] > > > For most application code, executing the function is far > > > more costly than the overhead of calling it, > > > > What? Your functions are only called _once_ during the > > lifetime of your code execution? Enough with the leg > > pulling. > > Let me explain what I intended to say. Here is a really > poor estimate of the overhead of calling a function on my > computer, measured in iPython. > > [...snip valid examples...] > > Now I know that this is not always the case. But I maintain > that for *most* (but not all) practical, real-world code > that actually does something useful, the overhead of > calling the functions is *usually* (but not always) low > compared to the cost of actually doing whatever it is that > the functions do. True. But again you failed to address my main point that: "Lazy features (when feasible) must be offered _optionally_". Yes, function call overhead may be tiny in relation to the code encapsulated within (at least for non-trivial code), but the fact remains that function calls are inherent in modern software, and as
Re: Implicit conversion to str in str.join method?
On Tue, Feb 27, 2018 at 6:38 AM, Kirill Balunov wrote: > Currently `str.join` raises `TypeError` if there are any non-string values > in iterable, including `bytes` objects. Is it an awful idea to implicitly > _cast_ elements in iterable to their `str` or `repr` representation? Was > this question adressed before? > > As for me there are two valid points: On one hand "Explicit is beter than > implicit" and on the other "Practicality beats purity". May be I'm the only > one who finds that it is rather boring to write somehting like: > > ", ".join(str(i) for i in iterable) > > when iterable contains non-string values, instead of: > > ", ".join(iterable) > > I don't know how much overhead will yield for the case when iterable > contains only strings...and if it is possible to optimize this case. This > change is backward compatible, but as I wrote above, it can be perceived > badly and yield a huge overhead for general case that it is not even worth > discussing. What do you think? This would be a perfect job for map. ", ".join(map(str, iterable)) If this is for display, you could also just print the values directly: print(*iterable, sep=", ") ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, Feb 27, 2018 at 6:37 AM, Rick Johnson wrote: > On Monday, February 26, 2018 at 4:39:22 AM UTC-6, Steven D'Aprano wrote: >> On Sun, 25 Feb 2018 19:26:12 -0800, Rick Johnson wrote: >> >> > On Friday, February 23, 2018 at 8:48:55 PM UTC-6, Steven D'Aprano wrote: >> > [...] >> > > Take the Fibonacci double-recursion benchmark. Okay, it >> > > tests how well your language does at making millions of >> > > function calls. Why? >> > >> > Because making "millons of function calls" is what >> > software *DOES*! >> >> You're right -- I worded that badly, and ended up saying >> something I didn't really mean. Mea culpa. Of course over >> the course of the entire program run, most non-trivial >> programmers will end up having millions (if not billions) >> of function calls *in total*. But they're not all happening >> *together*. > > So what? Latency is latency. And whether it occurs over the > course of one heavily recursive algorithm that constitutes > the depth and breadth of an entire program (a la fib()), or > it is the incremental cumulative consequence of the entire > program execution, the fact remains that function call > overhead contributes to a significant portion of the latency > inherent in some trivial, and *ALL* non-trivial, modern > software. By saying "the fact remains", you're handwaving away the work of actually measuring that function call overhead is "significant". Can you show me those numbers? Steve's point is that it is NOT significant, because non-toy functions have non-trivial bodies. If you wish to disagree, you have to demonstrate that the function call is *itself* costly, even when there's a real body to it. > I'm very glad you brought this up, because it proves my main > point that, Python's maximum recursion error is one feature > they got _right_. If you don't like the default recursion > limit of 1000, or maybe 1000 recursions is breaking a valid > recursive algorithm, you can set the recursion limit to > whatever you like... > > ## Python 2 ## > >>> import sys > >>> sys.setrecursionlimit > > >>> sys.setrecursionlimit(10) > >>> count = 1 > >>> def bad(): > ... global count > ... print count > ... count += 1 > ... bad() > ... > >>> bad() > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > 9 > Traceback (most recent call last): > File "", line 1, in > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > File "", line 5, in bad > RuntimeError: maximum recursion depth exceeded > > It is however unfortunate that we get slamed with N > recursive exception messages :-(, but hey, at least we can > modify the maximum recursion limit! It's a step in the right > direction. :-) Ah, been a while since you tried Python 3, is it? :) [Previous line repeated X more times] ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Implicit conversion to str in str.join method?
> > print(*iterable, sep=", ") Thanks, I apologize :-) and why I always manage to find complicated ways... With kind regards, -gdg 2018-02-26 22:43 GMT+03:00 Chris Angelico : > On Tue, Feb 27, 2018 at 6:38 AM, Kirill Balunov > wrote: > > Currently `str.join` raises `TypeError` if there are any non-string > values > > in iterable, including `bytes` objects. Is it an awful idea to implicitly > > _cast_ elements in iterable to their `str` or `repr` representation? Was > > this question adressed before? > > > > As for me there are two valid points: On one hand "Explicit is beter than > > implicit" and on the other "Practicality beats purity". May be I'm the > only > > one who finds that it is rather boring to write somehting like: > > > > ", ".join(str(i) for i in iterable) > > > > when iterable contains non-string values, instead of: > > > > ", ".join(iterable) > > > > I don't know how much overhead will yield for the case when iterable > > contains only strings...and if it is possible to optimize this case. This > > change is backward compatible, but as I wrote above, it can be perceived > > badly and yield a huge overhead for general case that it is not even > worth > > discussing. What do you think? > > This would be a perfect job for map. > > ", ".join(map(str, iterable)) > > If this is for display, you could also just print the values directly: > > print(*iterable, sep=", ") > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Implicit conversion to str in str.join method?
On Tue, Feb 27, 2018 at 6:54 AM, Kirill Balunov wrote: >> print(*iterable, sep=", ") > > > Thanks, I apologize :-) and why I always manage to find complicated ways... Hey, that's why we have the list :) I call this a success. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Implicit conversion to str in str.join method?
On 2018-02-26, Kirill Balunov wrote: > Currently `str.join` raises `TypeError` if there are any non-string values > in iterable, including `bytes` objects. Is it an awful idea to implicitly > _cast_ elements in iterable to their `str` or `repr` representation? Yes, that's an awful idea. PHP pulls stunts like that behind your back, and it's a huge source of bugs. -- Grant Edwards grant.b.edwardsYow! Let's all show human at CONCERN for REVERAND MOON's gmail.comlegal difficulties!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Monday, February 26, 2018 at 3:44:14 PM UTC+1, Steven D'Aprano wrote:
> I have a class with a large number of parameters (about ten) assigned in
> `__init__`. The class then has a number of methods which accept
> *optional* arguments with the same names as the constructor/initialiser
> parameters. If those arguments are None, the defaults are taken from the
> instance attributes.
>
> An example might be something like this:
>
>
> class Foo:
> def __init__(self, bashful, doc, dopey, grumpy,
>happy, sleepy, sneezy):
> self.bashful = bashful # etc
>
> def spam(self, bashful=None, doc=None, dopey=None,
>grumpy=None, happy=None, sleepy=None,
>sneezy=None):
> if bashful is None:
> bashful = self.bashful
> if doc is None:
> doc = self.doc
> if dopey is None:
> dopey = self.dopey
> if grumpy is None:
> grumpy = self.grumpy
> if happy is None:
> happy = self.happy
> if sleepy is None:
> sleepy = self.sleepy
> if sneezy is None:
> sneezy = self.sneezy
> # now do the real work...
>
> def eggs(self, bashful=None, # etc...
>):
> if bashful is None:
> bashful = self.bashful
> # and so on
>
>
> There's a lot of tedious boilerplate repetition in this, and to add
> insult to injury the class is still under active development with an
> unstable API, so every time I change one of the parameters, or add a new
> one, I have to change it in over a dozen places.
>
> Is there a good fix for this to reduce the amount of boilerplate?
>
>
> Thanks,
>
>
>
> --
> Steve
What about something along these lines:
from inspect import getargspec, getargvalues
class Demo(object):
def __init__(self, a=None, b=10, c='oops'):
spec = getargspec(self.__init__)
for key, value in zip(spec.args[1:], spec.defaults):
setattr(self, key, value)
def foo(self, *args, **kwargs):
assert len(args) == 0
for k, v in kwargs.items():
try:
getattr(self, k)
except AttributeError:
print('*** error: attribute {} does not exist.'.format(k))
setattr(self, k, v)
d = Demo()
print(d.a)
d.foo(a=3.14)
print(d.a)
d.foo(q='this will raise')
So, use the inspect module to detect the valid arguments
from the class initializer. Then use **kwargs in every
class method. It would be nice if the full method signature
can be kept, but I have not yet figured out if this is
possible.
Marco
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 19:50, Chris Angelico wrote: On Tue, Feb 27, 2018 at 6:37 AM, Rick Johnson So what? Latency is latency. And whether it occurs over the course of one heavily recursive algorithm that constitutes the depth and breadth of an entire program (a la fib()), or it is the incremental cumulative consequence of the entire program execution, the fact remains that function call overhead contributes to a significant portion of the latency inherent in some trivial, and *ALL* non-trivial, modern software. Take the last bit of Python I posted, which was that RNG test. It uses this function: def i64(x): return x & 0x This is a small function, but it can't be a toy one as it was suggested by Ned Batchelder. And the test program wasn't at all recursive. Running the program with N=10_000_000 took 46 seconds. Replacing the i64() call with '&m' (where m is a global set to 0x), it took 38 seconds. So putting that fix into a function, which is convenient for coding, maintenance and readability, cost 20% in runtime. Going the other way, having i64() call another function i64a() which does the work, common when using wrapper functions, it took 60 seconds. A 30% slowdown, even though most of the work is doing numeric shifts and adds. So function calls do seem to be expensive operations in CPython, and any benchmarks which highlight that fact should be welcomed. (Note than none of this seems to affect PyPy, which ran at the same speed with i64(), without it, or with both i64() and i64a().) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 26/02/2018 20:27, bartc wrote: On 26/02/2018 19:50, Chris Angelico wrote: On Tue, Feb 27, 2018 at 6:37 AM, Rick Johnson So what? Latency is latency. And whether it occurs over the course of one heavily recursive algorithm that constitutes the depth and breadth of an entire program (a la fib()), or it is the incremental cumulative consequence of the entire program execution, the fact remains that function call overhead contributes to a significant portion of the latency inherent in some trivial, and *ALL* non-trivial, modern software. [Sorry, the bit of your (Chris') post I replied to got chopped by mistake. Here is my post again with the right context:] CA: > By saying "the fact remains", you're handwaving away the work of > actually measuring that function call overhead is "significant". Can > you show me those numbers? Steve's point is that it is NOT > significant, because non-toy functions have non-trivial bodies. If you > wish to disagree, you have to demonstrate that the function call is > *itself* costly, even when there's a real body to it. > Take the last bit of Python I posted, which was that RNG test. It uses this function: def i64(x): return x & 0x This is a small function, but it can't be a toy one as it was suggested by Ned Batchelder. And the test program wasn't at all recursive. Running the program with N=10_000_000 took 46 seconds. Replacing the i64() call with '&m' (where m is a global set to 0x), it took 38 seconds. So putting that fix into a function, which is convenient for coding, maintenance and readability, cost 20% in runtime. Going the other way, having i64() call another function i64a() which does the work, common when using wrapper functions, it took 60 seconds. A 30% slowdown, even though most of the work is doing numeric shifts and adds. So function calls do seem to be expensive operations in CPython, and any benchmarks which highlight that fact should be welcomed. (Note than none of this seems to affect PyPy, which ran at the same speed with i64(), without it, or with both i64() and i64a().) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: For Loop Dilema [python-list]
On Monday, February 26, 2018 at 12:57:25 PM UTC+5:30, Chris Angelico wrote: > On Mon, Feb 26, 2018 at 5:19 AM, wrote: > > Why we don’t use: > > > > for _ in _ in _ > > > > Instead of > > > > for _ in _: > > for _ in _: > > > > Ex: > > > > Names = ["Arya","Pupun"] > > > > for name in Names: > >for c in name: > >print(c) > > > > instead use: > > > > for c in name in Names: > > print(c) > > Because programming is all about building up a program from > primitives. The number of times when we need this kind of nested loop > (with absolutely nothing in between the loops) is way too small to > justify dedicated syntax. > > ChrisA Thank you. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Monday, February 26, 2018 at 8:44:14 AM UTC-6, Steven D'Aprano wrote: > I have a class with a large number of parameters (about > ten) assigned in `__init__`. The class then has a number of > methods which accept *optional* arguments with the same > names as the constructor/initialiser parameters. If those > arguments are None, the defaults are taken from the > instance attributes. > > An example might be something like this: > > class Foo: > def __init__(self, bashful, doc, dopey, grumpy, >happy, sleepy, sneezy): > self.bashful = bashful # etc > > def spam(self, bashful=None, doc=None, dopey=None, >grumpy=None, happy=None, sleepy=None, >sneezy=None): > if bashful is None: > bashful = self.bashful > if doc is None: > doc = self.doc > if dopey is None: > dopey = self.dopey > if grumpy is None: > grumpy = self.grumpy > if happy is None: > happy = self.happy > if sleepy is None: > sleepy = self.sleepy > if sneezy is None: > sneezy = self.sneezy > # now do the real work... > > def eggs(self, bashful=None, # etc... >): > if bashful is None: > bashful = self.bashful > # and so on Steven... even if this example code is absolutely atrocious (because it is!), and even *IF* i have a uneasy suspicion that this "inquiry" masks some dastardly malicious plan (because i do!), I must admit, I am happy to see that you are _finally_ embracing the OOP paradigm. For starters, I would suggest replacing those ugly if- clauses with some short-circuit or'd-logic. But obviously you need to run that "paragraph of arguments" through a single helper function, as the point of "DRY" is "Don't Repeat Yourself". -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Mon, Feb 26, 2018 at 1:09 PM, wrote: > def foo(self, *args, **kwargs): > assert len(args) == 0 Better: def foo(self, **kwargs): > So, use the inspect module to detect the valid arguments > from the class initializer. Then use **kwargs in every > class method. It would be nice if the full method signature > can be kept, but I have not yet figured out if this is > possible. https://pypi.python.org/pypi/decorator/4.2.1 -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Mon, Feb 26, 2018 at 8:53 AM, Seb wrote: > On Sun, 25 Feb 2018 18:52:14 -0500, > Terry Reedy wrote: > > [...] > >> numpy has a matrix multiply function and now the '@' matrix multiply >> operator. > > Yes, but what I was wondering is whether there's a faster way of > multiplying each row (1x3) of a matrix by another matrix (3x3), compared > to looping through the matrix row by row as shown in the code. Perhaps > I'm not understanding how to use the broadcasting features of `matmul`. I'm a bit of a numpy newb, but this sounds like numpy's "broadcasting" feature: http://lmgtfy.com/?q=numpy+broadcasting -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Mon, Feb 26, 2018 at 2:37 PM, Ian Kelly wrote: > On Mon, Feb 26, 2018 at 1:09 PM, wrote: >> def foo(self, *args, **kwargs): >> assert len(args) == 0 > > Better: > > def foo(self, **kwargs): > >> So, use the inspect module to detect the valid arguments >> from the class initializer. Then use **kwargs in every >> class method. It would be nice if the full method signature >> can be kept, but I have not yet figured out if this is >> possible. > > https://pypi.python.org/pypi/decorator/4.2.1 Also, in Python 3.4+ you can just create a Signature object (either from scratch or by calling inspect.signature()) and then assign it to the function's __signature__ attribute. -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Mon, Feb 26, 2018 at 2:40 PM, Dan Stromberg wrote: > On Mon, Feb 26, 2018 at 8:53 AM, Seb wrote: >> On Sun, 25 Feb 2018 18:52:14 -0500, >> Terry Reedy wrote: >> >> [...] >> >>> numpy has a matrix multiply function and now the '@' matrix multiply >>> operator. >> >> Yes, but what I was wondering is whether there's a faster way of >> multiplying each row (1x3) of a matrix by another matrix (3x3), compared >> to looping through the matrix row by row as shown in the code. Perhaps >> I'm not understanding how to use the broadcasting features of `matmul`. > > I'm a bit of a numpy newb, but this sounds like numpy's "broadcasting" > feature: > http://lmgtfy.com/?q=numpy+broadcasting Taking LMGTFY to a whole new level of rudeness by obviously not even bothering to read the entire paragraph before responding. -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Mon, Feb 26, 2018 at 2:07 PM, Ian Kelly wrote: > On Mon, Feb 26, 2018 at 2:40 PM, Dan Stromberg wrote: >> On Mon, Feb 26, 2018 at 8:53 AM, Seb wrote: >>> On Sun, 25 Feb 2018 18:52:14 -0500, >>> Terry Reedy wrote: >>> >>> [...] >>> numpy has a matrix multiply function and now the '@' matrix multiply operator. >>> >>> Yes, but what I was wondering is whether there's a faster way of >>> multiplying each row (1x3) of a matrix by another matrix (3x3), compared >>> to looping through the matrix row by row as shown in the code. Perhaps >>> I'm not understanding how to use the broadcasting features of `matmul`. >> >> I'm a bit of a numpy newb, but this sounds like numpy's "broadcasting" >> feature: >> http://lmgtfy.com/?q=numpy+broadcasting > > Taking LMGTFY to a whole new level of rudeness by obviously not even > bothering to read the entire paragraph before responding. Is LMGTFY rude? I think maybe it was back when it said "Now, was that so hard?", but today there's no snarkiness to be found on the site, AFAIK. Also, like I said, I'm a numpy newbie. If I misunderstood the question, that's probably why. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
Steven D'Aprano writes: > I have a class with a large number of parameters (about ten) assigned > in `__init__`. The class then has a number of methods which accept > *optional* arguments with the same names as the > constructor/initialiser parameters. If those arguments are None, the > defaults are taken from the instance attributes. The catalogue in Refactoring identifies what you need: Introduce Parameter Object You have a group of parameters that naturally go together. Replace them with an object. https://refactoring.com/catalog/introduceParameterObject.html> Applying that to your example:: class Foo: def __init__(self, wibble_param, dwarves, wobble_param): do_stuff_with(wibble_param) self.dwarves = dwarves do_other_stuff_with(wobble_param) def spam(self, dwarves=None): if dwarves is None: dwarves = self.dwarves # … now do the real spam work … def eggs(self, dwarves=None): if dwarves is None: dwarves = self.dwarves # … now do the real eggs work … > […] to add insult to injury the class is still under active > development with an unstable API, so every time I change one of the > parameters, or add a new one, I have to change it in over a dozen > places. By encapsulating those parameters that all belong together, you can change that set of parameters (by changing what ‘dwarves’ gets passed around) and the function signatures don't change. -- \“All opinions are not equal. Some are a very great deal more | `\ robust, sophisticated, and well supported in logic and argument | _o__) than others.” —Douglas Adams | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Mon, Feb 26, 2018 at 3:12 PM, Dan Stromberg wrote: > On Mon, Feb 26, 2018 at 2:07 PM, Ian Kelly wrote: >> Taking LMGTFY to a whole new level of rudeness by obviously not even >> bothering to read the entire paragraph before responding. > > Is LMGTFY rude? I think maybe it was back when it said "Now, was that > so hard?", but today there's no snarkiness to be found on the site, > AFAIK. Maybe it's gotten less condescending since the last time I followed such a link. It is notably disallowed on Stack Overflow for the reason of being considered impolite. -- https://mail.python.org/mailman/listinfo/python-list
Re: help me ?
On Monday, February 26, 2018 at 9:40:54 AM UTC, Negru Popi wrote: > Define 2 lists. The first one must contain the integer values 1, 2 and 3 and > the second one the string values a, b and c. Iterate through both lists to > create another list that contains all the combinations of the A and B > elements. The final list should look like one of the 2 lists: > 1. [1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 3c] > 2. [a1, a2. a3, b1, b2, b3, c1, c2, c3] > BONUS: Make the final list contain all possible combinations : [1a, a1, 1b, > b1, 1c, c1, 2a, a2, 2b, b2, 2c, c2, 3a, a3, 3b, b3, 3c, c3] > > Help me ! Homework? -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
Seb wrote: I was wondering is whether there's a faster way of multiplying each row (1x3) of a matrix by another matrix (3x3), compared to looping through the matrix row by row as shown in the code. Just multiply the two matrices together. If A is an nx3 matrix and B is a 3x3 matrix, then C = A @ B is an nx3 matrix where C[i] = A[i] @ B. (This is a property of matrix multiplication in general, nothing special about numpy.) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 2018-02-26 21:12, Rick Johnson wrote: On Monday, February 26, 2018 at 8:44:14 AM UTC-6, Steven D'Aprano wrote: I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on Steven... even if this example code is absolutely atrocious (because it is!), and even *IF* i have a uneasy suspicion that this "inquiry" masks some dastardly malicious plan (because i do!), I must admit, I am happy to see that you are _finally_ embracing the OOP paradigm. For starters, I would suggest replacing those ugly if- clauses with some short-circuit or'd-logic. But obviously you need to run that "paragraph of arguments" through a single helper function, as the point of "DRY" is "Don't Repeat Yourself". Before using or'd-logic, you need to know whether the value could be falsey, e.g. 0. -- https://mail.python.org/mailman/listinfo/python-list
Re: Questions about `locals` builtin
On 2/26/2018 1:55 PM, Kirill Balunov wrote: Hi, I am a little bit confused with `locals` builtin in these moments: 1. The documentation says that _free varaibles_ are returned, which seems incorrect description. In my mind the term free variable refers to variables used in a function that are not local variables nor parameters of that function. I disagree with the terminology used in this section of the docs, so I won't defend or explain. The important points are that 1. *Python* code is that executed in the context of multiple directly accessible namespaces; 2. there is more than one way to implement a namespace; 3. the Python namespace system has grown in complexity since the beginning; and 4. some details depend on the implementation (CPython versus others) and Python and therefore implementation version. The point of point 3 is that terminology and details would likely be different if Python were freshly designed more or less as it is today, and some things only make more or less sense in historical context. Learn what you need to know to write code that works. The program namespace is called 'builtins'. Treat it as readonly. The namespace, as opposed to its contents, cannot be accessed unless imported. Each module namespace is accessed as globals(). It must be a dict. The names of builtins are accessed as it they were in globals, even though, in CPython, they are not. (I believe 'globals' predates multi-module programs.) The local namespace is the default recipient of name bindings from assignment, class, and def statements. At module scope, the local namespace is the module or global namespace. A binding in module locals == globals can mask a builtin. 'def print(): pass'. Deletion of that binding unmasks the builtin. 'del print'. (If an implementation actually initialized each globals() by copying builtins, then each deletion would have to check for possible unmasking.) Class statements get a new local namespace, which defaults to some sort of mapping. When the class statement finishes, the local names become class attributes. Be default, function local namespaces in CPython are implemented as arrays, with local names compiles as indexes in the array. Calling surrounding function local names collectively 'nonlocals' is the result of months of bikeshedding. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Questions about `locals` builtin
On Mon, 26 Feb 2018 21:55:35 +0300, Kirill Balunov wrote: > Hi, > > I am a little bit confused with `locals` builtin in these moments: > > 1. The documentation says that _free varaibles_ are returned, which > seems incorrect description. I can't answer that, sorry. > 2. The documentation has a note that "The contents of this dictionary > should not be modified". Which implies that it is a read only mapping. > So the question why it is `dict` instead of `types.MappingProxyType`? Mostly because locals() predates MappingProxyType by many years, and also because that restriction doesn't apply to other implementations of Python such as Jython and IronPython. In CPython, the dict returned by locals() is a copy of the local namespace, so modifying the dict doesn't modify the real local variables. (Well, sometimes it does, but not always. The story in Python 2 is really complex.) But in Jython and IronPython, it actually is the namespace, so writing to it works like writing to globals. So writing to locals *sometimes* works, and cannot be prohibited outright, but if you want portable code, you have to avoid it. > 3. There is one more moment: local variables had been determined when > function was compiled. But `locals` returns _some_ current runtime copy. Have you tried it in Python 2 or 3? The behaviour may be less confusing in 3. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Monday, February 26, 2018 at 5:44:18 PM UTC-6, MRAB wrote: [...] > Before using or'd-logic, you need to know whether the value > could be falsey, e.g. 0. True. However. Steven failed to provide any info that might help us determine the types of these parameters, and as such, i was forced to take the liberty of assumption. Now, while it's quite possible someone could assign dwarfy-named- symbols to implicit or explicit boolean values, i find it highly unlikely, and totally unintuitive. But then again, i have my suspicions about ths thread. ;-) -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, 27 Feb 2018 02:09:53 +1100, Chris Angelico wrote: > You're still reimplementing the C code in Python, which is inefficient. > Have you considered going back to the *actual algorithm* and > implementing that idiomatically in Python? I think you'll find that (a) > the code is more readable, and (b) the run time is much lower. Chris, I think this is incredibly optimistic, if not naive. We're talking about a PRNG by Marsaglia, so my guess is that the "original algorithm" *is* the C code. Or possibly Fortran. Even if not, even if there is actually a language-neutral algorithm, its a PRNG which means its going to be filled with bit-twiddling and number- crunching operations. Pure Python without a JIT is never going to be competitive with C or Fortran for code like that. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, Feb 27, 2018 at 11:17 AM, Steven D'Aprano wrote: > On Tue, 27 Feb 2018 02:09:53 +1100, Chris Angelico wrote: > >> You're still reimplementing the C code in Python, which is inefficient. >> Have you considered going back to the *actual algorithm* and >> implementing that idiomatically in Python? I think you'll find that (a) >> the code is more readable, and (b) the run time is much lower. > > Chris, I think this is incredibly optimistic, if not naive. We're talking > about a PRNG by Marsaglia, so my guess is that the "original algorithm" > *is* the C code. Or possibly Fortran. > > Even if not, even if there is actually a language-neutral algorithm, its > a PRNG which means its going to be filled with bit-twiddling and number- > crunching operations. Pure Python without a JIT is never going to be > competitive with C or Fortran for code like that. > I may have been a little unclear. It's highly unlikely that the run time of the properly-implemented Python code will be lower than the original C or Fortran. But it most certainly CAN be more efficient than the Python reimplementation of the C implementation, which would narrow the gap considerably. Implementing Python idiotically instead of idiomatically gives you suboptimal performance. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Questions about `locals` builtin
On Mon, Feb 26, 2018 at 4:25 PM, Steven D'Aprano
wrote:
> On Mon, 26 Feb 2018 21:55:35 +0300, Kirill Balunov wrote:
>> 2. The documentation has a note that "The contents of this dictionary
>> should not be modified". Which implies that it is a read only mapping.
>> So the question why it is `dict` instead of `types.MappingProxyType`?
>
> Mostly because locals() predates MappingProxyType by many years, and also
> because that restriction doesn't apply to other implementations of Python
> such as Jython and IronPython.
>
> In CPython, the dict returned by locals() is a copy of the local
> namespace, so modifying the dict doesn't modify the real local variables.
>
> (Well, sometimes it does, but not always. The story in Python 2 is really
> complex.)
>
> But in Jython and IronPython, it actually is the namespace, so writing to
> it works like writing to globals.
>
> So writing to locals *sometimes* works, and cannot be prohibited
> outright, but if you want portable code, you have to avoid it.
I find this interesting.
So I decided to write a little test program, to see it first hand.
I don't have IronPython handy, but according my (quite possibly
flawed) test program, locals() is a copy on CPython 3, CPython 2,
Pypy3, Pypy, Jython and MicroPython.
I didn't see any interpreters that returned the namespace itself.
What am I missing?
Here's the test program. Perhaps there's something wrong in it?
#!/usr/local/cpython-3.6/bin/python3
"""Test whether locals() is a copy of the local namespace, or a reference."""
import sys
def main():
"""Run the test."""
var = 1
locs = locals()
locs['var'] = 2
if var == 2:
sys.stdout.write('locals not just a copy\n')
else:
sys.stdout.write('locals is just a copy\n')
main()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Tuesday, February 20, 2018 at 5:45:36 PM UTC-6, Steven D'Aprano wrote: > On Tue, 20 Feb 2018 12:42:23 -0800, Rick Johnson wrote: > > > For instance, if the age is queried many times a second, > > it would be a much wiser design to set-up an event that > > will advance the age at the end of the last second of > > every year > > Do you really mean to say that everybody in the world has > their birthday on January 1st? We're not racehorses you > know. No, silly rabbit. I was thinking about the problem from a _relative_ perspective, whereas you were thinking about the problem from a _global_ perspective. Neither perspective is wrong. If you read my exact words again: "a much wiser design to set-up an event that will advance the age at the end of the last second of every year" ...you'll notice that i mentioned no specific date. Therefore, "the last day of the year" (in relativistic terms) is 11:59:59PM on the calendar day which _precedes_ the day of the month for which you were born. So, for instance: if your birthday is January 25th 1969, the last second of the last day of your _first_ year is January 24th 1970 @ 11:59:59PM. And the last second of the last day of your _second_ year is January 24th 1971 @ 11:59:59PM. And so forth... Does this make sense? > > Under your scheme, 99.7% of records will return the wrong > age (off by one) at least once per year. Statistically, 50% > of queries will be wrong. Can you whip-up a small code example which proves your assertion? Hey, and feel free to include the seven dwarfs if you like. ;-) -- https://mail.python.org/mailman/listinfo/python-list
Re: matrix multiplication
On Tue, 27 Feb 2018 12:25:30 +1300, Gregory Ewing wrote: > Seb wrote: >> I was wondering is whether there's a faster way of multiplying each >> row (1x3) of a matrix by another matrix (3x3), compared to looping >> through the matrix row by row as shown in the code. > Just multiply the two matrices together. > If A is an nx3 matrix and B is a 3x3 matrix, then C = A @ B is an nx3 > matrix where C[i] = A[i] @ B. > (This is a property of matrix multiplication in general, nothing > special about numpy.) I think that's only true if B is the same for every row in A. In the code I posted, B varies by row of A. -- Seb -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On 27/02/2018 00:35, Chris Angelico wrote: On Tue, Feb 27, 2018 at 11:17 AM, Steven D'Aprano wrote: On Tue, 27 Feb 2018 02:09:53 +1100, Chris Angelico wrote: You're still reimplementing the C code in Python, which is inefficient. Have you considered going back to the *actual algorithm* and implementing that idiomatically in Python? I think you'll find that (a) the code is more readable, and (b) the run time is much lower. Chris, I think this is incredibly optimistic, if not naive. We're talking about a PRNG by Marsaglia, so my guess is that the "original algorithm" *is* the C code. Or possibly Fortran. Even if not, even if there is actually a language-neutral algorithm, its a PRNG which means its going to be filled with bit-twiddling and number- crunching operations. Pure Python without a JIT is never going to be competitive with C or Fortran for code like that. I may have been a little unclear. It's highly unlikely that the run time of the properly-implemented Python code will be lower than the original C or Fortran. But it most certainly CAN be more efficient than the Python reimplementation of the C implementation, which would narrow the gap considerably. Implementing Python idiotically instead of idiomatically gives you suboptimal performance. Nonsense. You shouldn't need to care about such things. An algorithm is an algorithm. And the better ones should be easily written in any language. This particular one, which was of interest because the calculations tended to overflow into undesirable bignums, is just a bunch of calculations. 'Bit-twiddling'. Anyway, even this pure Python version can deliver pseudo random numbers at some 200,000 per second, while the built-in generator does 450,000 per second, so it's not bad going. Of course, the C version will generate them at over 100 million per second. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: How to make Python run as fast (or faster) than Julia
On Tue, Feb 27, 2018 at 12:57 PM, bartc wrote: > On 27/02/2018 00:35, Chris Angelico wrote: >> >> On Tue, Feb 27, 2018 at 11:17 AM, Steven D'Aprano >> wrote: >>> >>> On Tue, 27 Feb 2018 02:09:53 +1100, Chris Angelico wrote: >>> You're still reimplementing the C code in Python, which is inefficient. Have you considered going back to the *actual algorithm* and implementing that idiomatically in Python? I think you'll find that (a) the code is more readable, and (b) the run time is much lower. >>> >>> >>> Chris, I think this is incredibly optimistic, if not naive. We're talking >>> about a PRNG by Marsaglia, so my guess is that the "original algorithm" >>> *is* the C code. Or possibly Fortran. >>> >>> Even if not, even if there is actually a language-neutral algorithm, its >>> a PRNG which means its going to be filled with bit-twiddling and number- >>> crunching operations. Pure Python without a JIT is never going to be >>> competitive with C or Fortran for code like that. >>> >> >> I may have been a little unclear. It's highly unlikely that the run >> time of the properly-implemented Python code will be lower than the >> original C or Fortran. But it most certainly CAN be more efficient >> than the Python reimplementation of the C implementation, which would >> narrow the gap considerably. Implementing Python idiotically instead >> of idiomatically gives you suboptimal performance. > > > Nonsense. You shouldn't need to care about such things. An algorithm is an > algorithm. And the better ones should be easily written in any language. > > This particular one, which was of interest because the calculations tended > to overflow into undesirable bignums, is just a bunch of calculations. > 'Bit-twiddling'. > > Anyway, even this pure Python version can deliver pseudo random numbers at > some 200,000 per second, while the built-in generator does 450,000 per > second, so it's not bad going. The built-in generator is using a completely different algorithm though, so rate of generation isn't the entire story. How long is the period of the one you're using? (How long before it loops?) If you churn the generator to an arbitrary number and then take the next 100 values it generates, are they randomly distributed? Can you reconstruct the RNG's internal state by watching it generate numbers for a short while? Obviously no PRNG is going to be perfect at this, but there are certainly degrees of quality to be compared. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Mon, 26 Feb 2018 17:18:38 -0800, Rick Johnson wrote: > On Tuesday, February 20, 2018 at 5:45:36 PM UTC-6, Steven D'Aprano > wrote: >> On Tue, 20 Feb 2018 12:42:23 -0800, Rick Johnson wrote: >> >> > For instance, if the age is queried many times a second, it would be >> > a much wiser design to set-up an event that will advance the age at >> > the end of the last second of every year >> >> Do you really mean to say that everybody in the world has their >> birthday on January 1st? We're not racehorses you know. [...] > If you read my exact words again: > > "a much wiser design to set-up an event that will advance the age at > the end of the last second of every year" Oh, you mean the exact same words already quoted above? > ...you'll notice that i mentioned no specific date. Yes you did: "the last second of every year" is always 23:59:59 of 31st December, and it is always the same time and date "every year". Just because you didn't explicitly state the name of a month doesn't mean you didn't implicitly specify one. Perhaps you didn't *intend* to specify the last day of the year, in which case you ought to just accept that your writing wasn't as clear as you intended, instead of blaming others for failing to read your mind. > Therefore, "the last day of the year" (in relativistic terms) is > 11:59:59PM on the calendar day which _precedes_ the day of the month for > which you were born. Relativistic terms? You mean when travelling at 90% of the speed of light or in close orbit around a neutron star? There are those language skills letting you down again :-P I think you mean *relative* terms. [...] >> Under your scheme, 99.7% of records will return the wrong age (off by >> one) at least once per year. Statistically, 50% of queries will be >> wrong. > > > Can you whip-up a small code example which proves your assertion? Hey, > and feel free to include the seven dwarfs if you like. ;-) Under the racehorse scheme, 1/365 of your data will be born on the 1st of Jan, 1/365 on the 2nd Jan, and so on through 31st Dec. (Ignoring leap years.) Half of the data records will be updated early, and half will be updated late. Only those born on New Years Day (about 0.3% of the records) will be updated on the correct day. Hence 99.7% will be wrong for at least one day per year. If queries are made randomly, they'll be off by one 50% of the time. Why 50%? 1/365 of your records will never be wrong, 1/365 will be wrong one day of the year, 1/365 will be wrong two days of the year, 1/365 will be wrong three days of the year, ... 1/365 will be wrong 364 days of the year. So on average, records are wrong 182 days of the year, which is close enough to half of 365. So if you query a random record on a random day, there's close enough to a fifty percent chance it will be off by one. Now that I understand that you didn't intend this racehorse scheme, I'm not going to bother writing a simulation. But it is just simple statistics, assuming that the actual birthday is distributed randomly across the year, and assuming that queries for records come in at random times. (And to be pedantic, I mean a uniform random distribution, and that queries and birthdays are independent.) -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Mon, 26 Feb 2018 17:18:38 -0800, Rick Johnson wrote: [...] > So, for instance: if your birthday is January 25th 1969, the last second > of the last day of your _first_ year is January 24th 1970 @ 11:59:59PM. > And the last second of the last day of your _second_ year is January > 24th 1971 @ 11:59:59PM. And so forth... > > Does this make sense? Indeed it does, and frankly, the Racehorse scheme is better. At least with the Racehorse scheme, you only need to update the database once a year, at midnight on the new year, which hopefully is the quietest time of the year for you. You can lock access to the database, run the update, and hopefully be up and running again before anyone notices anything other than a minor outage. With your scheme, well, I can think of a few ways to do it, none of which are good. A database expert might be able to think of some better ideas, but you might: 1. Run a separate scheduled job for each record, which does nothing but advance the age by one at a certain time, then sleep for a year. If you have ten million records, you need ten million scheduled jobs; I doubt many scheduling systems can cope with that many jobs. (But I welcome correction.) Also, few scheduling systems guarantee that jobs will execute at *precisely* the time you expect. If the system is down at the time the job was scheduled to run, they may never run at all. So there is likely to be a lag between when you want the records updated, and when they actually are updated. No, using scheduled jobs is fragile, and expensive. Plan 2: have a single job that does nothing but scan the database, continuously in a loop, and if a record's birthdate is more than a year in the past, and hasn't been updated in the last year, update the age by one. Actually, I think this sucks worse than the ten-million-scheduled-jobs idea. Hopefully it will be obvious why this idea is so awful. Plan 3: have a trigger that runs whenever a record is queried or accessed. If the birthdate is more than a year in the past, and it has been more than a year since the last access, then update the age. This at least doesn't *entirely* suck. But if you're going to go to the trouble of doing this on *every* access to the record, isn't it simpler to just make the age a computed field that calculates the age when needed? The cost of computing the age is not that expensive, especially if you store the birthdate in seconds. It's just a subtraction, maybe followed by a division if you want the age in years. It hardly seems worthwhile storing the age as a pre-computed integer if you then need a cunning scheme to possibly update that integer on every access to the record. I think that Rick's "optimization" here is a perfect example of pessimisation (making a program slower in the mistaken belief that you're making it faster). To quote W.A. Wulf: "More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason — including blind stupidity." -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Mon, Feb 26, 2018 at 8:09 PM, Steven D'Aprano wrote: > Yes you did: "the last second of every year" is always 23:59:59 of 31st > December, and it is always the same time and date "every year". Except when it's 23:59:60 or 23:59:61 (which hasn't yet happened but could). -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Tue, Feb 27, 2018 at 12:18 PM, Rick Johnson wrote: > On Tuesday, February 20, 2018 at 5:45:36 PM UTC-6, Steven D'Aprano wrote: >> On Tue, 20 Feb 2018 12:42:23 -0800, Rick Johnson wrote: >> >> > For instance, if the age is queried many times a second, >> > it would be a much wiser design to set-up an event that >> > will advance the age at the end of the last second of >> > every year >> >> Do you really mean to say that everybody in the world has >> their birthday on January 1st? We're not racehorses you >> know. > > > No, silly rabbit. I was thinking about the problem from a > _relative_ perspective, whereas you were thinking about the > problem from a _global_ perspective. Neither perspective is > wrong. > > If you read my exact words again: > > "a much wiser design to set-up an event that will advance > the age at the end of the last second of every year" > > ...you'll notice that i mentioned no specific date. > > Therefore, "the last day of the year" (in relativistic > terms) is 11:59:59PM on the calendar day which _precedes_ > the day of the month for which you were born. > > So, for instance: if your birthday is January 25th 1969, the > last second of the last day of your _first_ year is January > 24th 1970 @ 11:59:59PM. And the last second of the last day > of your _second_ year is January 24th 1971 @ 11:59:59PM. And > so forth... > > Does this make sense? Yes! It does. All you have to do is run a batch job in exactly the very last second of *every single day*. Totally not a problem! Have you ever run a database in your life? Tip: When you're in a hole, stop digging. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Tue, Feb 27, 2018 at 2:55 PM, Ian Kelly wrote: > On Mon, Feb 26, 2018 at 8:09 PM, Steven D'Aprano > wrote: >> Yes you did: "the last second of every year" is always 23:59:59 of 31st >> December, and it is always the same time and date "every year". > > Except when it's 23:59:60 or 23:59:61 (which hasn't yet happened but could). Unless you use a leap smear to avoid it. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Mon, Feb 26, 2018 at 8:55 PM, Ian Kelly wrote: > On Mon, Feb 26, 2018 at 8:09 PM, Steven D'Aprano > wrote: >> Yes you did: "the last second of every year" is always 23:59:59 of 31st >> December, and it is always the same time and date "every year". > > Except when it's 23:59:60 or 23:59:61 (which hasn't yet happened but could). Actually, after doing some reading on the subject just now, I learned that double leap seconds don't actually exist! They were accidentally invented by the drafters of the ANSI C standard and later propagated to the POSIX standard and the ANSI SQL standard. https://www.ucolick.org/~sla/leapsecs/timescales.html#UTC So the actual last second of the year must always be one of 23:59:58, 23:59:59 or 23:59:60. (Hint: although we've never had a negative leap second to date, don't schedule your database update for 23:59:59 if you want to ensure that it actually happens.) -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Mon, 26 Feb 2018 17:39:43 +0100, Peter Otten wrote: [...] > I have not yet looked into dataclasses. Don't they handle the __init__() > part? Anyway, here's my attempt to make spam() less spammy: I'm not too concerned about __init__, it's only one method :-) > $ cat attrs_to_args_decorator.py > import functools > import inspect > > def add_defaults(f): > argnames = inspect.getfullargspec(f).args[1:] > > @functools.wraps(f) > def wrapper(self, *args, **kw): > args = [ > getattr(self, name) if value is None else value for name, > value in zip(argnames, args) > ] > for name in argnames[len(args):]: > if name not in kw or kw[name] is None: > kw[name] = getattr(self, name) > return f(self, *args, **kw) > return wrapper [...] Interesting, this does look promising, thanks. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Mon, 26 Feb 2018 16:23:39 -0800, Rick Johnson wrote: > On Monday, February 26, 2018 at 5:44:18 PM UTC-6, MRAB wrote: [...] >> Before using or'd-logic, you need to know whether the value could be >> falsey, e.g. 0. > > True. However. Steven failed to provide any info that might help us > determine the types of these parameters, and as such, i was forced to > take the liberty of assumption. Someone held a gun to your head? > Now, while it's quite possible someone > could assign dwarfy-named- symbols to implicit or explicit boolean > values, i find it highly unlikely, and totally unintuitive. But then > again, i have my suspicions about ths thread. ;-) You mean things like passing 0 as an int argument, or "" as a string argument? Yeah, I see what you mean -- that would be such a weird thing to do... /s -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On Tue, 27 Feb 2018 09:15:12 +1100, Ben Finney wrote: > Steven D'Aprano writes: > >> I have a class with a large number of parameters (about ten) assigned >> in `__init__`. The class then has a number of methods which accept >> *optional* arguments with the same names as the constructor/initialiser >> parameters. If those arguments are None, the defaults are taken from >> the instance attributes. > > The catalogue in Refactoring identifies what you need: > > Introduce Parameter Object > > You have a group of parameters that naturally go together. > > Replace them with an object. > > https://refactoring.com/catalog/introduceParameterObject.html> Interesting, I'll certainly have a good read of that, thanks. But I fear I mislead you: you seem to have understood that either all the parameters are None, or none of them are, whereas what I meant was that they can vary independently of each other. So it isn't enough to check whether the Parameter object is None, I'd need to check each field within the object. On the other hand, at least I could put the logic of that inside the Parameter object, instead of repeating it everywhere it is needed. On the gripping hand, I *think* that means the caller is responsible for creating and populating the Parameter object, instead of just passing keyword arguments... let me think about that one. Either way, thanks for the suggestion. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
Steven D'Aprano writes: > Interesting, I'll certainly have a good read of that, thanks. But I > fear I mislead you: you seem to have understood that either all the > parameters are None, or none of them are, whereas what I meant was > that they can vary independently of each other. I think that issue – independent defaults for the values – is orthogonal. For avoiding the large set of parameters repeated everywhere, you evidently consider that cluster of values a single concept, to be passed around as a unit. That's what the “Introduce Parameter Object” refactor operation is for: Now that you recognise this cluster is an implicit conceptual unit that different methods share, define an explicit type for it, and pass around an instance of that between the functions. > So it isn't enough to check whether the Parameter object is None, I'd > need to check each field within the object. No problem. You can do that within the methods, as before. Alternatively, you can define it as behaviour of the type. The “Introduce Parameter Object” refactor leaves that and other options open, because it's an unrelated issue. > On the gripping hand, I *think* that means the caller is responsible > for creating and populating the Parameter object, instead of just > passing keyword arguments... let me think about that one. One good effect of this refactor is you're forced to explicitly *name* this concept of cluster-of-values-with-some-specific-meaning. Once you have a name – which will confont you with a decision about the meaning of this specific cluster of values – you then get to thinking about how the values should be initialised, etc. All of that is hidden from the methods that just want to pass around the cluster of values to each other. You then get to freely change it while development continues. > Either way, thanks for the suggestion. Happy hacking. -- \“I think it would be a good idea.” —Mohandas K. Gandhi (when | `\asked what he thought of Western civilization) | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Questions about `locals` builtin
Kirill Balunov writes: > I am a little bit confused with `locals` builtin in these moments: > > 1. The documentation says that _free varaibles_ are returned, which seems > incorrect description. In my mind the term free variable refers to > variables used in a function that are not local variables nor parameters of > that function. > > In docs: "Update and return a dictionary representing the current local > symbol table. Free variables are returned by `locals()` when it is called > in function blocks, but not in class blocks." I agree with your definition of "free variable" and I think that the documentation partially speaks of it. In a function body, you have two kinds of "free variables": implicitly used free variables and explicitly declared (via "global", "nonlocal") free variables. I think the documentation wants to make clear, that "locals" does not treat "free variables" in a completely consistent way (there is a difference between class and function blocks). I assume that "locals" may return some (but not necessary all) free variables in a function block -- depending on the actual implementation. -- https://mail.python.org/mailman/listinfo/python-list
