Re: How to benchmark a HTTP connection with requests module?
Hi Dennis, Nice pseudo code! :-) Is it possible benchmark/measure the latency of a HTTP connection for each threads by timing the duration of the requests.get method to complete? I also plan to test with gevent.monkey extension for enabling cooperative multithreading support: from gevent import monkey monkey.patch_all() Etienne Le 2018-02-15 à 11:56, Dennis Lee Bieber a écrit : Keyword: threads Create a number of threads, each handling one request, and use a global flag to start them. And maybe a queue to retrieve Pseudo-code (I haven't checked the manual for correct API): hold = True resultQ = Queue.Queue() def worker(ID): while hold: pass r = requests.get(...) resultQ.put( (ID, r.status_code) ) def benchmark(): requestTasks = [ threading.thread( worker, args=(ID) ) for ID in range(NUMBEROFREQUESTS) ] for rT in requestTasks.start() #or is it .run() #workers are now busy waiting for hold to False #better would be to use threading.Condition and .notifyAll() #having each thread wait on a global condition variable, #rather than spinning on a busy wait hold = False for _ in range(NUMBEROFREQUESTS): (ID, code) = resultQ.get() requestTasks[ID].join() #put any asserts here -- Etienne Robillard tkad...@yandex.com https://www.isotopesoftware.ca/ -- https://mail.python.org/mailman/listinfo/python-list
Best site
which is the best site to learn python for data science. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to benchmark a HTTP connection with requests module?
Hi Dennis, Here's my code so far: import os import requests import threading import time from Queue import Queue from test_support import unittest test_queue = Queue() def worker(ident, url, queue): while True: start = time.clock() r = requests.get(url) end = time.clock() queue.put((ident, r.status_code, end - start)) queue.task_done() return queue class HTTPBenchmarkTestCase(unittest.TestCase): url = 'http://localhost/benchmark/' threads = 5 def setUp(self): self.hold = True def test_benchmark_concurrency(self): for i in range(self.threads): t = threading.Thread(target=worker, args=(i, self.url, test_queue)) t.daemon = True t.start() obj = test_queue.get() print obj test_queue.join() erob@marina:~/src/django-hotsauce-0.9/tests$ pypy ./run.py -C benchmarks DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost (0, 200, 0.1299950349943) DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost (1, 200, 0.05324292899606) DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost (0, 200, 0.1122124810025) DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost (1, 200, 0.1207582250033) DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost DEBUG:http://localhost:80 "GET /benchmark/ HTTP/1.1" 200 76 DEBUG:Starting new HTTP connection (1): localhost (0, 200, 0.1259027660028) . -- Ran 1 test in 0.468s OK What do you think? :-) Etienne Le 2018-02-16 à 10:58, Dennis Lee Bieber a écrit : On Fri, 16 Feb 2018 06:22:04 -0500, Etienne Robillard declaimed the following: Hi Dennis, Nice pseudo code! :-) Is it possible benchmark/measure the latency of a HTTP connection for each threads by timing the duration of the requests.get method to complete? Wrap each .get with calls for start time and end time? (Need to check if time.time or time.clock is more precise on your hardware, docs recommend time.clock for benchmarking) def worker(ID): while hold: pass st = time.clock() r = requests.get(...) en = time.clock() resultQ.put( (ID, r.status_code, en - st) ) #return duration as third parameter for _ in range(NUMBEROFREQUESTS): (ID, code, duration) = resultQ.get() requestTasks[ID].join() -- Etienne Robillard tkad...@yandex.com https://www.isotopesoftware.ca/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Best site
On 16/02/2018 16:04, dsujit2...@gmail.com wrote: which is the best site to learn python for data science. Google. -- https://mail.python.org/mailman/listinfo/python-list
Re: Best site
Besides Google for sure. I liked www.codecademy.com Thank you, Irving Duran On 02/16/2018 11:53 AM, mm0fmf wrote: > On 16/02/2018 16:04, dsujit2...@gmail.com wrote: >> which is the best site to learn python for data science. >> > > Google. -- https://mail.python.org/mailman/listinfo/python-list
How to run script from interpreter?
Yes, it's been covered, but not quite to my satisfaction.
Here's an example simple script:
# Very simple script
bar = 123
I save this as "foo.py" somewhere Python can find it
>>> import foo
>>> bar
Traceback (most recent call last):
File "", line 1, in
NameError: name 'bar' is not defined
# Oops, it's in a different namespace and I have to prefix EVERYTHING with
"foo.". This is inconvenient.
>>> foo.bar
123
Start a new session...
>>> from foo import *
>>> bar
123
Do a little editing:
# Very simple (new) script
bar = 456
Save as foo.py, back to the interpreter:
>>> reload(foo)
Traceback (most recent call last):
File "", line 1, in
NameError: name 'foo' is not defined
# Oops, it's not in the namespace, so there is NO way to use reload
Start a new session...
>>> execfile('foo.py')
Traceback (most recent call last):
File "", line 1, in
IOError: [Errno 2] No such file or directory: 'foo.py'
# Oops, it's on the path, but execfile can't find it
>>> import os,sys
>>> os.chdir('W:/Code')
>>> execfile('foo.py')
>>> bar
456
Do some more editing:
# Very simple (even newer) script
bar = 789
Save it as foo.py, back to the interpreter:
>>> execfile('foo.py')
>>> bar
789
That works, but nothing is very convenient for debugging simple scripts. If I
run the script from a command prompt it works, but I lose all my other stuff
(debugging functions, variables, etc.).
More a comment than a question but seems like sometimes execfile() is the right
tool.
Regards,
Allen
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to run script from interpreter?
On Sat, Feb 17, 2018 at 9:18 AM, windhorn wrote:
> Yes, it's been covered, but not quite to my satisfaction.
>
> Here's an example simple script:
>
> # Very simple script
> bar = 123
>
> I save this as "foo.py" somewhere Python can find it
>
import foo
bar
> Traceback (most recent call last):
> File "", line 1, in
> NameError: name 'bar' is not defined
>
> # Oops, it's in a different namespace and I have to prefix EVERYTHING with
> "foo.". This is inconvenient.
>
foo.bar
> 123
>
> Start a new session...
>
from foo import *
bar
> 123
>
> Do a little editing:
>
> # Very simple (new) script
> bar = 456
>
> Save as foo.py, back to the interpreter:
>
reload(foo)
> Traceback (most recent call last):
> File "", line 1, in
> NameError: name 'foo' is not defined
>
> # Oops, it's not in the namespace, so there is NO way to use reload
>
> Start a new session...
>
execfile('foo.py')
> Traceback (most recent call last):
> File "", line 1, in
> IOError: [Errno 2] No such file or directory: 'foo.py'
>
> # Oops, it's on the path, but execfile can't find it
>
import os,sys
os.chdir('W:/Code')
execfile('foo.py')
bar
> 456
>
> Do some more editing:
>
> # Very simple (even newer) script
> bar = 789
>
> Save it as foo.py, back to the interpreter:
>
execfile('foo.py')
bar
> 789
>
> That works, but nothing is very convenient for debugging simple scripts. If I
> run the script from a command prompt it works, but I lose all my other stuff
> (debugging functions, variables, etc.).
>
> More a comment than a question but seems like sometimes execfile() is the
> right tool.
Or possibly you want to exit completely, and run "python3 -i foo.py",
which will drop you into the interactive interpreter after running the
script. That's safer than trying to re-execute an already-loaded
module.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best site
A. Because it spoils the flow. On 16/02/2018 20:07, Irving Duran wrote: Besides Google for sure. I liked www.codecademy.com Thank you, Irving Duran On 02/16/2018 11:53 AM, mm0fmf wrote: On 16/02/2018 16:04, dsujit2...@gmail.com wrote: which is the best site to learn python for data science. Google. Q. Why shouldn't I top post on a newsgroup? -- https://mail.python.org/mailman/listinfo/python-list
Re: How to run script from interpreter?
On Fri, Feb 16, 2018 at 3:18 PM, windhorn wrote:
> Yes, it's been covered, but not quite to my satisfaction.
>
> Here's an example simple script:
>
> # Very simple script
> bar = 123
>
> I save this as "foo.py" somewhere Python can find it
>
import foo
bar
> Traceback (most recent call last):
> File "", line 1, in
> NameError: name 'bar' is not defined
>
> # Oops, it's in a different namespace and I have to prefix EVERYTHING with
> "foo.". This is inconvenient.
>
foo.bar
> 123
>
> Start a new session...
>
from foo import *
bar
> 123
You didn't need to start a new session here. "from foo import *" works
just as well in the existing session.
> Do a little editing:
>
> # Very simple (new) script
> bar = 456
>
> Save as foo.py, back to the interpreter:
>
reload(foo)
> Traceback (most recent call last):
> File "", line 1, in
> NameError: name 'foo' is not defined
>
> # Oops, it's not in the namespace, so there is NO way to use reload
Sure there is.
py> import foo # Get the module to reload
py> reload(foo) # Reload it
py> from foo import * # Update all the names previously *-imported
> Start a new session...
>
execfile('foo.py')
> Traceback (most recent call last):
> File "", line 1, in
> IOError: [Errno 2] No such file or directory: 'foo.py'
>
> # Oops, it's on the path, but execfile can't find it
execfile doesn't use the path; that's not what it's meant for.
--
https://mail.python.org/mailman/listinfo/python-list
Are the critiques in "All the things I hate about Python" valid?
This article is written by Nathan Murthy, a staff software engineer at Tesla. The article is found at: https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e Apparently he chose his article title as "click bait". Apparently he does not really hate Python (So he says.). His leader paragraph is: "Python is viewed as a ubiquitous programming language; however, its design limits its potential as a reliable and high performance systems language. Unfortunately, not every developer is aware of its limitations." As I currently do not have the necessary technical knowledge to properly evaluate his claims, I thought I would ask those of you who do. I have neither the knowledge or boB-hours to write a large distributed system code base, but I am curious if Python is truly limited for doing these in the ways he claims. BTW, I am not trying to start (another) heated, emotional thread. You guys do sometimes get carried away! I honestly just what to know the truth of the matters out of my continuing to learn Python. I suspect there is probably some truth in his claims, but I am not sure if he is taking things out of their proper application contexts or not. Thanks! -- boB -- https://mail.python.org/mailman/listinfo/python-list
Click bait versus “rational and unbiased demeanor” (was: Are the critiques in "All the things I hate about Python" valid?)
boB Stepp writes: > Apparently he chose his article title as "click bait". Apparently he > does not really hate Python (So he says.). Those may well be true. What I find implausible is his expressed desire: Ok, so “hate” is a strong word, but hopefully this click-baits enough folks into reading this article with a clearer, more rational and unbiased demeanor than what the title suggests. Without descending into hyperbole, what I hope to share with you are all […] At this point, having deliberately chosen a hyperbolic headline, the author is too late to avoid hyperbole, and too late to engender clear, rational unbiased demeanour. I despise the tendency to do all the things designed explicitly to push our irrational buttons, and then attempt feebly to regain the moral high ground by implying that the *reader* is responsible for any lack of “clear, rational, unbiased demeanour”. The author is not writing in good faith, and I do not expect that any arguments found in it are honest either. -- \ “I find the whole business of religion profoundly interesting. | `\ But it does mystify me that otherwise intelligent people take | _o__)it seriously.” —Douglas Adams | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
boB Stepp wrote: This article is written by Nathan Murthy, a staff software engineer at Tesla. The article is found at: https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e Apparently he chose his article title as "click bait". Apparently he does not really hate Python (So he says.). His leader paragraph is: "Python is viewed as a ubiquitous programming language; however, its design limits its potential as a reliable and high performance systems language. Python is simply not a high performance system language, but it has other virtues. You would not likely choose Python for a sophisticated distributed computing application, but not everyone needs to write a sophisticated distributed computing application. I believe those that do, will be conscientious enough to choose an appropriate tool. Unfortunately, not every developer is aware of its limitations." As I currently do not have the necessary technical knowledge to properly evaluate his claims, I thought I would ask those of you who do. I have neither the knowledge or boB-hours to write a large distributed system code base, but I am curious if Python is truly limited for doing these in the ways he claims. BTW, I am not trying to start (another) heated, emotional thread. You guys do sometimes get carried away! I honestly just what to know the truth of the matters out of my continuing to learn Python. I suspect there is probably some truth in his claims, but I am not sure if he is taking things out of their proper application contexts or not. Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
boB Stepp writes: > https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e > As I currently do not have the necessary technical knowledge to > properly evaluate his claims, I thought I would ask those of you who > do. Thank you for asking. The author makes many mistakes. He blithely conflates “weakly typed” (Python objects are not weakly, but very strongly typed) with “dynamically typed” (yes, Python's name binding is dynamically typed). Those are two, orthognal dimensions to describe a language. All the assertions about “strongly typed” should instead be read as “statically typed”. I do not think the author understands the differences, nor the trade-offs involved, in the weak–strong dimension nor the static–dynamic dimension. The author conflates the Python language, which has many different implementations and interpreters on very different technologies, with one particular implementation (CPython). Python does not have fast or strong performance; the performance is entirely a property of a specific implementation. All the arguments about performance should be read as criticisms of whatever implementation the author is using. He doesn't say which, so we can't know how justified those complaints are. A strong hint is in the complaint about the Global Interpreter Lock. This is a property of only one implementation: the CPython implementation. It is a known limitation and many people have worked for many years to ameliorate it; it is unclear whether the author knows of anything done in recent years (the only specific information cited is for 2012; a reminder, that's *six years ago*, a aeon in platform development). The author also uses very old information when complaining about Python 2 versus Python 3. The citation of grave fears is to a 2014 article. The author shows no signs of awareness that the Python 3 ecosystem is very healthy today compared to Python 2. The complaint about abstractions makes another silly conflation; the author seems to think any language that does object-orientation differently from Java, is obviously wrong. The author may benefit from reading http://dirtsimple.org/2004/12/python-is-not-java.html> (advice which is no less relevant than in 2004). But maybe not. The complaint about Python's build system and packaging tools is accurate enough, though pretty parochial (the author has a strong bias to existing Fortran packages). The Python build and packaging ecosystem has been improving steadily for many years now, but yes, it still needs a lot of improvement. The author acknowledges in the conclusion that “nothing I’ve said is new”. I would argue that most of what is said is not even true; the article should not be referenced at all, IMO. -- \ “We jealously reserve the right to be mistaken in our view of | `\ what exists, given that theories often change under pressure | _o__) from further investigation.” —Thomas W. Clark, 2009 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On 2018-02-17 03:22, boB Stepp wrote: This article is written by Nathan Murthy, a staff software engineer at Tesla. The article is found at: https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e Apparently he chose his article title as "click bait". Apparently he does not really hate Python (So he says.). His leader paragraph is: "Python is viewed as a ubiquitous programming language; however, its design limits its potential as a reliable and high performance systems language. Unfortunately, not every developer is aware of its limitations." As I currently do not have the necessary technical knowledge to properly evaluate his claims, I thought I would ask those of you who do. I have neither the knowledge or boB-hours to write a large distributed system code base, but I am curious if Python is truly limited for doing these in the ways he claims. BTW, I am not trying to start (another) heated, emotional thread. You guys do sometimes get carried away! I honestly just what to know the truth of the matters out of my continuing to learn Python. I suspect there is probably some truth in his claims, but I am not sure if he is taking things out of their proper application contexts or not. He's making the assertions that: "dynamically typed" == "weakly typed" and: "strongly typed" == "statically typed" They aren't equivalences. Python is both dynamically typed _and_ strongly typed. And static type checking gets you only so far. If you've added when you should've subtracted, static type checking won't spot it. Another point: a "wheel" is not a "source distribution" that requires a compiler, it can include a pre-compiled binary. At least in the Conclusion he says "It is still a useful and powerful programming language when suited for its intended purposes." Exactly. It's not suited for realtime processing, etc, but, then, it was never intended for that anyway. -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 2:22 PM, boB Stepp wrote: > This article is written by Nathan Murthy, a staff software engineer at > Tesla. The article is found at: > https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e > > Apparently he chose his article title as "click bait". Apparently he > does not really hate Python (So he says.). His leader paragraph is: > > "Python is viewed as a ubiquitous programming language; however, its > design limits its potential as a reliable and high performance systems > language. Unfortunately, not every developer is aware of its > limitations." > > As I currently do not have the necessary technical knowledge to > properly evaluate his claims, I thought I would ask those of you who > do. I have neither the knowledge or boB-hours to write a large > distributed system code base, but I am curious if Python is truly > limited for doing these in the ways he claims. > > BTW, I am not trying to start (another) heated, emotional thread. You > guys do sometimes get carried away! I honestly just what to know the > truth of the matters out of my continuing to learn Python. I suspect > there is probably some truth in his claims, but I am not sure if he is > taking things out of their proper application contexts or not. 1) Type safety. This is often touted as a necessity for industrial-grade software. It isn't. There are many things that a type system, no matter how sophisticated, cannot catch; for some reason, though, we don't hear people saying "C is useless for industrial-grade software because it doesn't have function contracts". Anyway, if you want some system of type checking, you can use static analysis (eg tools like MyPy) to go over your code the same way a compiler might. "The first glaring issue is that I have no guarantee that is_valid() returns a bool type." -- huh? It's being used in a boolean context, and it has a name that starts "is_". How much guarantee are you looking for? *ANY* object can be used in an 'if', so it doesn't even matter. This is a stupidly contrived criticism. > Python markets itself as a dynamically-typed programming language, > similar to Perl, Ruby, or JavaScript. The word “dynamically typed” has > a positive connotation, more so than “weakly typed.” Conversely, the > word “strongly typed” sounds more positive than saying “statically typed.” Those are all different terms, and all of them are misunderstood. They're not synonyms. > Python ships with a threading module; but due to how the Python > interpreter is implemented, it can only ever run one thread at a time. > This is due to the infamous Global Interpreter Lock (GIL). In an age > when the only computers around had a single core, this was > nothing to fuss over. Totally not true. The GIL does not stop other threads from running. Also, Python has existed for multiple CPU systems pretty much since its inception, I believe. (Summoning the D'Aprano for history lesson?) > Python is relatively slow compared to programming languages that > run closer to the operating system. The run time of the countdown > example above is orders of magnitude faster when implemented > in other language runtimes. Yeah, it sometimes runs a bit slower in benchmarking. And it's way WAY faster to code in. Case in point: I reimplemented an app in Python based on a C# (one of the languages the article's author touts as better than Python), and made a single file under a thousand lines long out of what had been two separate projects (a UI and a library), dozens of files, tens of thousands of lines, and tons of boilerplate that looks like this: https://github.com/geel9/SteamAuth/blob/master/SteamAuth/SteamGuardAccount.cs#L15 That's thirty lines of declarations to show how the JSON data translates into this object's properties. In Python, I just decode JSON to a dict, and then subscript that dict with whatever I need. Zero lines of boilerplate. So which one is more expensive? Hours and hours of programmer time, or a few milliseconds of runtime? > If you’re doing development on Mac or Ubuntu, you likely already > have Python 2.7 installed (Windows normally does not come with > Python pre-packaged). There’s a good chance that when you first > started programming in Python, no one told you about this thing > called Python 3 (no one told me at least). Ubuntu ships with Python 3 installed these days (and they're working on removing the need to have Python 2 installed). Macs ship with a lot of outdated software, so the correct thing to do is to install the latest anyway. If "no one told you" about Python 3, you don't know how to ask. And trust me, the Py2/Py3 gap is far less serious than the differences between version X and version X+1 of many JavaScript libraries. I know this for certain, having spent this past week resolving problems with React and related package. > Encapsulation and Packaging > ... > This is unsettling if you’re coming from a Java or C/C++ world and > are expecting ac
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 2:50 PM, Bill wrote: > boB Stepp wrote: >> >> This article is written by Nathan Murthy, a staff software engineer at >> Tesla. The article is found at: >> >> https://medium.com/@natemurthy/all-the-things-i-hate-about-python-5c5ff5fda95e >> >> Apparently he chose his article title as "click bait". Apparently he >> does not really hate Python (So he says.). His leader paragraph is: >> >> "Python is viewed as a ubiquitous programming language; however, its >> design limits its potential as a reliable and high performance systems >> language. > > > Python is simply not a high performance system language, but it has other > virtues. You would not likely choose Python for a sophisticated distributed > computing application, but not everyone needs to write a sophisticated > distributed computing application. I believe those that do, will be > conscientious enough to choose an appropriate tool. > You'd be surprised how rarely that kind of performance even matters. The author of that article cites C# as a superior language, but in the rewrite from C# to Python (the same one I mentioned in the other post), I sped the program up incredibly. Part of that is because C# requires the startup of its framework (in my case that's Mono) just as Python does, and partly it's because the simplicity of Python let me eliminate a number of unnecessary HTTP requests. Trust me, your choice of language doesn't help you if it means you do three (sequential) HTTP requests when one would have done. Clean code has its own advantages. And if the number of HTTP transactions didn't change, the performance of the two languages - or any other languages you choose - would be virtually indistinguishable. Most programs spend most of their time waiting, perhaps on the network, or the disk, or the user, but always waiting. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Fri, 16 Feb 2018 21:22:48 -0600, boB Stepp wrote: > This article is written by Nathan Murthy, a staff software engineer at > Tesla. The article is found at: > https://medium.com/@natemurthy/all-the-things-i-hate-about- python-5c5ff5fda95e > > Apparently he chose his article title as "click bait". Apparently he > does not really hate Python (So he says.). His leader paragraph is: > > "Python is viewed as a ubiquitous programming language; however, its > design limits its potential as a reliable and high performance systems > language. Unfortunately, not every developer is aware of its > limitations." I haven't (yet) read the article, but the above seems reasonable to me. Python's design does put rather harsh limits on how far you can push the compiler to generate high-performance code. I'm not so sure about the claim for "reliable" -- I suspect that's actually *not* a reasonable conclusion to draw (but I shall keep an open mind until I've read the article). If high-performance is important to (generic) you, the right way to use Python is: - use Python for the parts of the application which are not performance critical, e.g. the user interface; - prototype the rest of the application in Python as a first iteration; - profile the application to find the *actual* bottlenecks, not the parts of the code you *imagined* would be bottlenecks; - re-write those parts in a higher performance language; - and use Python as the glue to hold those pieces together. This is why Python is prodding buttock in the field of scientific computing. That's precisely how numpy works: the back-end is written in C and Fortran for speed, and the front-end user-interface and non-critical code is written in Python. Python is really good for gluing together high-performance but user- and programmer-hostile scientific libraries written in C and Fortran. You wouldn't write a serious, industrial-strength neural network in pure Python code and expect to process terabytes of data in any reasonable time. But you might write a serious, industrial-strength neural network in C or Rust, and then use your Python layer as the front-end to it, feeding data in and out of the neural network from easy-to-write, easy-to-debug, easy- to-maintain Python code. And you can write a prototype neural network in Python. It won't process gigabytes of data per second, but it will probably process megabytes. And with tools like cython, you can often turn your Python code into Python- like code that is compiled into efficient machine code, giving you almost C-like speed with almost Python-like simplicity. Remember that Python's original purpose was to act as a minimalist glue language between C and Fortran libraries. Over the years Python has developed enough features and power to be far more than that: its an excellent generalist, a high-performance scripting language and low to medium performance application language[1]. But acting as a glue between components written in other languages is still a very powerful technique for Python programmers. The standard library works like this too: non-critical libraries, like those for manipulating command line arguments, are written in pure Python. Critical components, like dicts and sets, are written in C. And then there are libraries that were prototyped in Python, then re-written in C for speed once they were mature, like the decimal and fractions modules. [1] Pedants will (rightly) observe that computer languages aren't themselves high- or low-performance. Languages are abstract entities that cannot be described as "fast" or "slow", as they run at the speed of imagination. Only specific interpreters/compilers can be described as fast or slow. Here I am talking about the standard CPython interpreter. There are five mainstream, mature Python interpreters: CPython, Jython, IronPython, PyPy and Stackless. They all have their individual strengths and weaknesses. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Fri, Feb 16, 2018 at 10:05 PM, Ben Finney wrote: > He blithely conflates “weakly typed” (Python objects are not weakly, but > very strongly typed) with “dynamically typed” (yes, Python's name > binding is dynamically typed). Those are two, orthognal dimensions to > describe a language. > > All the assertions about “strongly typed” should instead be read as > “statically typed”. I do not think the author understands the > differences, nor the trade-offs involved, in the weak–strong dimension > nor the static–dynamic dimension. I should have caught the "weakly typed" error. But I was too distracted by his "type safety" train of thought. And this is one I am still puzzling over: Are statically-typed languages inherently "safer" than properly implemented dynamically-typed languages? I can see the advantages of catching type errors at compile time versus run time. And with the relatively recent introduction of type hints to python, I was wondering if this was indeed a perceived, need-to-correct weakness by the python devs? But I have seen myself the advantages of being able to rebind identifiers to differently typed objects than what they started out being bound to. So I do not have enough knowledge to properly evaluate software safety or robustness in terms of how a language is typed. > A strong hint is in the complaint about the Global Interpreter Lock. > This is a property of only one implementation: the CPython > implementation. It is a known limitation and many people have worked for > many years to ameliorate it; it is unclear whether the author knows of > anything done in recent years (the only specific information cited is > for 2012; a reminder, that's *six years ago*, a aeon in platform > development). This is another of my big technical weak spots. I currently believe that doing concurrent and asynchronous programming is a difficult chore in any language, but I have yet gotten to the point of where I know enough to start struggling myself with these forms of programming. I do know from some of my books that there are ways despite the GIL to handle these types of programming problems with the current CPython implementation. But I do not know enough to evaluate how well or poorly python can make use of multicore processors. > The author also uses very old information when complaining about Python > 2 versus Python 3. The citation of grave fears is to a 2014 article. The > author shows no signs of awareness that the Python 3 ecosystem is very > healthy today compared to Python 2. I mostly dismissed out of hand this section. The only disturbing part, in my mind, was the claim that individual companies have python 2 vs. 3 rifts within their companies. I hope that is not really true. > The complaint about abstractions makes another silly conflation; the > author seems to think any language that does object-orientation > differently from Java, is obviously wrong. The author may benefit from > reading http://dirtsimple.org/2004/12/python-is-not-java.html> > (advice which is no less relevant than in 2004). But maybe not. Again, my only concern here was software safety issue claims. Does locking down object attribute access by the language necessarily improve software safety or robustness? I actually like python's "we're all consenting adults" mentality. It seems to me that the quality of the code is more reflective of the quality of the programmers than protecting (mediocre?) programmers from themselves by language design. But then again, maybe these sorts of language design restrictions make for fewer errors, even by excellent programmers, than not doing so. Again, I don't believe I know enough to intelligently evaluate the pros and cons. > The author acknowledges in the conclusion that “nothing I’ve said is > new”. I would argue that most of what is said is not even true; the > article should not be referenced at all, IMO. > Ben, I guess I am trying to be more charitable in my interpretation of the author's intent and actual knowledge. I was led to this article via one of the two weekly python news collation email services that I subscribe to, so I am hoping to learn something from the article and everyone's feedback. What's the old saw? Even a blind squirrel finds a nut once in a while? Thanks! -- boB -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 3:36 PM, Steven D'Aprano wrote: > Python is really good for gluing together high-performance but user- and > programmer-hostile scientific libraries written in C and Fortran. You > wouldn't write a serious, industrial-strength neural network in pure > Python code and expect to process terabytes of data in any reasonable > time. > > But you might write a serious, industrial-strength neural network in C or > Rust, and then use your Python layer as the front-end to it, feeding data > in and out of the neural network from easy-to-write, easy-to-debug, easy- > to-maintain Python code. Easy-to-debug. This deserves some expansion. One of the best aspects of Python is its debuggability (from readable code, lots of introspection, etc), so when you convert low level functionality into C, you ideally want to start with the parts that can be most easily unit-tested. User interfaces are notoriously hard to unit test, but pure functions can be more thoroughly tested. So you translate the pure functions first. There's often a strong correlation between the easily-testable functions and the performance bottlenecks. I'm hand-waving away a ton of details and complexities here, and it's certainly not a hard-and-fast rule, but it's a very common correlation. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Fri, Feb 16, 2018 at 10:25 PM, Chris Angelico wrote: > > 1) Type safety. > > This is often touted as a necessity for industrial-grade software. It > isn't... Chris, would you mind expanding on this point? What is necessary for industrial-grade, safe, robust software? Do statically-typed languages give any real advantage over dynamically-typed languages in this context? I know much is achieved by how the software is designed and the practices used to implement the design that are independent of the language used. But what language features/components are essential for industrial-grade software? -- boB -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 3:54 PM, boB Stepp wrote: > On Fri, Feb 16, 2018 at 10:05 PM, Ben Finney > wrote: > >> He blithely conflates “weakly typed” (Python objects are not weakly, but >> very strongly typed) with “dynamically typed” (yes, Python's name >> binding is dynamically typed). Those are two, orthognal dimensions to >> describe a language. >> >> All the assertions about “strongly typed” should instead be read as >> “statically typed”. I do not think the author understands the >> differences, nor the trade-offs involved, in the weak–strong dimension >> nor the static–dynamic dimension. > > I should have caught the "weakly typed" error. But I was too > distracted by his "type safety" train of thought. And this is one I > am still puzzling over: Are statically-typed languages inherently > "safer" than properly implemented dynamically-typed languages? I can > see the advantages of catching type errors at compile time versus run > time. And with the relatively recent introduction of type hints to > python, I was wondering if this was indeed a perceived, > need-to-correct weakness by the python devs? But I have seen myself > the advantages of being able to rebind identifiers to differently > typed objects than what they started out being bound to. So I do not > have enough knowledge to properly evaluate software safety or > robustness in terms of how a language is typed. No, they're not inherently safer. There's no way for the language to guarantee that your code is bug-free. But we all know that all code has bugs in it - and that the sooner you find a bug, the easier it is to fix it. So we have a variety of tools that help us to detect bugs: * Syntax highlighting * Linters (eg "unused local variable") * Warnings of various forms * Type checkers * Naming conventions (eg using plurals for collections) * Clear and consistent indentation * Unit tests * Automated test running as part of CI/CD * Machine-readable documentation (eg 'assert') * Etcetera Each of them helps you to catch bugs before they get out into production. None of them makes one language "safer" than another. They're all tools that can help you, but sometimes one tool's benefit overlaps another's to the extent that it's not worth the effort. That's why most of them are optional. If type checking is of value to you, use it! If you think it's a pointless waste of time, ignore it! It's not hurting you. >> A strong hint is in the complaint about the Global Interpreter Lock. >> This is a property of only one implementation: the CPython >> implementation. It is a known limitation and many people have worked for >> many years to ameliorate it; it is unclear whether the author knows of >> anything done in recent years (the only specific information cited is >> for 2012; a reminder, that's *six years ago*, a aeon in platform >> development). > > This is another of my big technical weak spots. I currently believe > that doing concurrent and asynchronous programming is a difficult > chore in any language, but I have yet gotten to the point of where I > know enough to start struggling myself with these forms of > programming. I do know from some of my books that there are ways > despite the GIL to handle these types of programming problems with the > current CPython implementation. But I do not know enough to evaluate > how well or poorly python can make use of multicore processors. Asynchronicity and concurrency are hard. Getting your head around a program that is simultaneously doing two things is inherently tricky. Anything that simplifies this (such as providing atomic and thread-safe operations on high-level data structures) will make this sort of work FAR easier. That's what Python gives you. CPython's current implementation (with the GIL) is reasonably easy to conceptualize, and requires very few other locks to make everything work sanely; all attempts to remove the GIL from CPython have involved other tradeoffs, usually resulting in significant performance penalties on single-threaded code. GIL or no GIL, CPython *does* support threads, and they are incredibly useful. They just don't automatically allow you to use multiple CPU cores. That's all. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Python 2 to 3 Conversion
I have a bit of code I found on the web that will return
the ip address of the named network interface. The code
is for Python 2 and it runs fine. But, I want to use the
code with Python 3. Below is the code followed by the error
message. Suggestions appreciated.
#!/usr/bin/env python3
import socket
import fcntl
import struct
def get_ip_address(ifname):
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
return socket.inet_ntoa(fcntl.ioctl(
s.fileno(),
0x8915, # SIOCGIFADDR
struct.pack('256s', ifname[:15])
)[20:24])
print(get_ip_address("eth0"))
print(get_ip_address("lo"))
Traceback (most recent call last):
File "./test.py", line 14, in
print(get_ip_address("eth0"))
File "./test.py", line 12, in get_ip_address
struct.pack('256s', ifname[:15])
struct.error: argument for 's' must be a bytes object
--
GNU/Linux user #557453
The cow died so I don't need your bull!
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python 2 to 3 Conversion
On Sat, Feb 17, 2018 at 4:15 PM, Wildman via Python-list
wrote:
> I have a bit of code I found on the web that will return
> the ip address of the named network interface. The code
> is for Python 2 and it runs fine. But, I want to use the
> code with Python 3. Below is the code followed by the error
> message. Suggestions appreciated.
>
> #!/usr/bin/env python3
>
> import socket
> import fcntl
> import struct
>
> def get_ip_address(ifname):
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> return socket.inet_ntoa(fcntl.ioctl(
> s.fileno(),
> 0x8915, # SIOCGIFADDR
> struct.pack('256s', ifname[:15])
> )[20:24])
> print(get_ip_address("eth0"))
> print(get_ip_address("lo"))
>
>
> Traceback (most recent call last):
> File "./test.py", line 14, in
> print(get_ip_address("eth0"))
> File "./test.py", line 12, in get_ip_address
> struct.pack('256s', ifname[:15])
> struct.error: argument for 's' must be a bytes object
The easiest way is to encode it to ASCII:
struct.pack('256s', ifname[:15].encode('ascii'))
But you may want to see if there's a completely different way to do
what you're looking at. I'm not sure what this is doing, so I can't
advise more specifically.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 4:11 PM, boB Stepp wrote:
> On Fri, Feb 16, 2018 at 10:25 PM, Chris Angelico wrote:
>
>>
>> 1) Type safety.
>>
>> This is often touted as a necessity for industrial-grade software. It
>> isn't...
>
> Chris, would you mind expanding on this point? What is necessary for
> industrial-grade, safe, robust software? Do statically-typed
> languages give any real advantage over dynamically-typed languages in
> this context? I know much is achieved by how the software is designed
> and the practices used to implement the design that are independent of
> the language used. But what language features/components are
> essential for industrial-grade software?
>
I've expanded on it a bit in my other post (which crossed in the mail
with this of yours). Basically, type checking ("type safety" is a bit
of a misnomer anyway) is just one of many possible ways to potentially
catch bugs. So it's not a *necessity*, any more than correct
indentation is a necessity. Both of them are valuable; neither of them
is crucial. And code review can do what the language doesn't. Would
you accept this JavaScript function in code review?
function calculateSpam(x) {
if (x>5) {
return "spaam";
}
else {
return "spam";
}}
The indentation doesn't matter, right? Not in JS. But it does to
humans. Same goes for sane use of data types:
def is_valid(x):
if yada_yada:
return "yes"
return [x]
Again, Python doesn't care if a function (a) sometimes returns a
string and sometimes a list, and (b) always returns a truthy value,
despite having a name that suggests a boolean return. But code review
should catch this.
Once you identify a common problem, you can get the computer to catch
it for you, so it doesn't come up in code review all the time. Does
that mean type declarations? Go for it, use them. Does it mean a
pre-commit hook that rejects wonky indentation? Make it so. Does it
mean using a programming language that uses variable declarations? Go
for it. Or maybe it means using a language that DOESN'T use variable
declarations. That's also an option. Ultimately, you use whatever
helps you to produce the least-buggy code you can. No single feature
is 100% necessary; no single feature is utterly useless.
(And yes, the choices you make depend on context. Some tools are
useful in large collaborations but pointless overhead in small
projects, etc. Be smart, don't be rigid.)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
I've just reread everyone's replies and one point you mentioned about the GIL caught my eye ... On Fri, Feb 16, 2018 at 11:16 PM, Chris Angelico wrote: > Asynchronicity and concurrency are hard. Getting your head around a > program that is simultaneously doing two things is inherently tricky. > Anything that simplifies this (such as providing atomic and > thread-safe operations on high-level data structures) will make this > sort of work FAR easier. That's what Python gives you. CPython's > current implementation (with the GIL) is reasonably easy to > conceptualize, and requires very few other locks to make everything > work sanely; all attempts to remove the GIL from CPython have involved > other tradeoffs, usually resulting in significant performance > penalties on single-threaded code. I am curious as to what efforts have been attempted to remove the GIL and what tradeoffs resulted and why? Is there a single article somewhere that collates this information? I fear if I try to Google this I will get a lot of scattered pieces that I will have to wade through to get to what I want to know, where you (or someone else) might be able to point me to a good link. Or kindly summarize yourself the relevant information. Thanks! -- boB -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 5:25 PM, boB Stepp wrote: > I've just reread everyone's replies and one point you mentioned about > the GIL caught my eye ... > > On Fri, Feb 16, 2018 at 11:16 PM, Chris Angelico wrote: > >> Asynchronicity and concurrency are hard. Getting your head around a >> program that is simultaneously doing two things is inherently tricky. >> Anything that simplifies this (such as providing atomic and >> thread-safe operations on high-level data structures) will make this >> sort of work FAR easier. That's what Python gives you. CPython's >> current implementation (with the GIL) is reasonably easy to >> conceptualize, and requires very few other locks to make everything >> work sanely; all attempts to remove the GIL from CPython have involved >> other tradeoffs, usually resulting in significant performance >> penalties on single-threaded code. > > I am curious as to what efforts have been attempted to remove the GIL > and what tradeoffs resulted and why? Is there a single article > somewhere that collates this information? I fear if I try to Google > this I will get a lot of scattered pieces that I will have to wade > through to get to what I want to know, where you (or someone else) > might be able to point me to a good link. Or kindly summarize > yourself the relevant information. > > Thanks! No, there isn't a single article, at least not that I know of. A good word to search for is "gilectomy", which brought me to this talk by Larry Hastings: https://www.youtube.com/watch?v=pLqv11ScGsQ Broadly speaking, what happens is that removing a large-scale lock (the GIL) requires using a whole lot of small-scale locks. That gives finer granularity, but it also adds a whole lot of overhead; the CPU features required for implementing those locks are not fast. With the GIL, you claim it, and you can do what you like. Without the GIL, you have to claim a lock on each object you manipulate, or something along those lines. (Different attempts have gone for different forms of granularity, so I can't generalize too much here.) That means claiming and relinquishing a lot more locks, which in turn means a lot more CPU-level "lock" primitives. That's a lot of overhead. One of the best ways to multi-thread CPU-intensive work is to push a lot of the work into an extension library. Take a function like this: def frobnosticate(stuff): magic magic magic magic magic more magic return result As long as the returned object is a newly-created one and the parameter is not mutated in any way, you can do all the work in the middle without holding the GIL. That can't be done in pure Python, but in a C function, it certainly can. You still have the coarse-grained locking of the GIL, you still have all the protection, but you can now have two threads frobnosticating different stuff at the same time. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 2 to 3 Conversion
Wildman via Python-list writes:
> def get_ip_address(ifname):
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> return socket.inet_ntoa(fcntl.ioctl(
> s.fileno(),
> 0x8915, # SIOCGIFADDR
> struct.pack('256s', ifname[:15])
> )[20:24])
> print(get_ip_address("eth0"))
> print(get_ip_address("lo"))
In Python 3, the literal ‘"eth0"’ creates a text (‘unicode’) object.
Python 3 correctly implements text as Unicode, which has no particular
representation as bytes.
You then pass that object to ‘struct.pack('256s', …)’, which (as the
error says) expects a ‘bytes’ object. A text object is incompatible with
that expectation.
If you want a sequence of bytes, perhaps you want to encode the text, to
some specified encoding. Maybe ASCII:
struct.pack('256s', ifname.encode('ascii'))
That may be different from what you want the code to do, though. It's
not clear from the code what its intention is.
--
\ “What is needed is not the will to believe but the will to find |
`\ out, which is the exact opposite.” —Bertrand Russell, _Free |
_o__) Thought and Official Propaganda_, 1928 |
Ben Finney
--
https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
On Sat, Feb 17, 2018 at 12:54 AM, Chris Angelico wrote: > On Sat, Feb 17, 2018 at 5:25 PM, boB Stepp wrote: >> >> I am curious as to what efforts have been attempted to remove the GIL >> and what tradeoffs resulted and why? Is there a single article >> somewhere that collates this information? I fear if I try to Google >> this I will get a lot of scattered pieces that I will have to wade >> through to get to what I want to know, where you (or someone else) >> might be able to point me to a good link. Or kindly summarize >> yourself the relevant information. >> >> Thanks! > > No, there isn't a single article, at least not that I know of. A good > word to search for is "gilectomy", which brought me to this talk by > Larry Hastings: > > https://www.youtube.com/watch?v=pLqv11ScGsQ Thanks for the link. I'll give it a look tomorrow. Must get some sleep soon! Meanwhile I just finished reading "Efficiently Exploiting Multiple Cores with Python" (from Nick Coghlan's Python Notes) at http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html It answered some of my questions and has a lot of good information. It appears that a "gilectomy" as you put it is a really tough problem. It looks to me that it might require a major backwards-compatibility-breaking change to CPython to implement if I am even close to understanding the issues involved. > Broadly speaking, what happens is that removing a large-scale lock > (the GIL) requires using a whole lot of small-scale locks. That gives > finer granularity, but it also adds a whole lot of overhead; the CPU > features required for implementing those locks are not fast. With the > GIL, you claim it, and you can do what you like. Without the GIL, you > have to claim a lock on each object you manipulate, or something along > those lines. (Different attempts have gone for different forms of > granularity, so I can't generalize too much here.) That means claiming > and relinquishing a lot more locks, which in turn means a lot more > CPU-level "lock" primitives. That's a lot of overhead. Thanks for this explanation. It really helps me a lot! > One of the best ways to multi-thread CPU-intensive work is to push a > lot of the work into an extension library. Take a function like this: > > def frobnosticate(stuff): > magic magic magic > magic magic more magic > return result > > As long as the returned object is a newly-created one and the > parameter is not mutated in any way, you can do all the work in the > middle without holding the GIL. That can't be done in pure Python, but > in a C function, it certainly can. You still have the coarse-grained > locking of the GIL, you still have all the protection, but you can now > have two threads frobnosticating different stuff at the same time. Yeah, Coghlan's article mentioned this. I guess using Cython would be one approach to this. Thanks, Chris! -- boB -- https://mail.python.org/mailman/listinfo/python-list
Re: Are the critiques in "All the things I hate about Python" valid?
Am 17.02.18 um 06:12 schrieb Stefan Ram: Chris Angelico quotes: Python is relatively slow compared to programming languages that run closer to the operating system. The run time of the countdown example above is orders of magnitude faster when implemented in other language runtimes. What seems to be especially slow is printing many long lines (from a loop) into the console of IDLE. The guy in the article has not printed anything, but you are right, printing in IDLE is slow for lonng lines, especially. The reason is, that the Tk text widget is slow if the lines are long due to inferior layout algorithms. Gregor Cramer wrote a new implementation last year: https://core.tcl.tk/tips/doc/trunk/tip/466.md It's fully backwards compatible, orders of magnitude faster in specific circumstances and will appear in Tk 8.7. Testers are most welcome. Sadly, the Tcl/Tk developer community is extremely small. The new text widget was a big fuss last year, but I haven't heard anything about it since almost a year :( Christian -- https://mail.python.org/mailman/listinfo/python-list
