Re: Goto Considered Harmful [was Re: Python and the need for speed]
Steve D'Aprano wrote: If PIC is so great, why do no other languages have it? Something akin to it has turned up in other places, although usually in the guise of an output formatting facility rather than a way of describing how data is stored internally. For example, the PRINT USING found in some BASIC dialects. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Goto Considered Harmful [was Re: Python and the need for speed]
Ian Kelly wrote: Yikes. I never took the time to learn COBOL, but that almost sounds like something that you'd find in an esoteric language like INTERCAL. COBOL has other fun stuff like that, too. For example, the destination of a GOTO statement can be changed from elsewhere in the program: https://www.ibm.com/support/knowledgecenter/SSAE4W_7.1.0/com.ibm.etools.iseries.langref.doc/c0925395347.htm Not quite COMEFROM, but it's getting close! -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Goto Considered Harmful [was Re: Python and the need for speed]
Gregory Ewing : > Steve D'Aprano wrote: > >> If PIC is so great, why do no other languages have it? > > Something akin to it has turned up in other places, although usually > in the guise of an output formatting facility rather than a way of > describing how data is stored internally. For example, the PRINT USING > found in some BASIC dialects. You have the storage thing in SQL: CHAR(4) VARCHAR(256) YEAR(2) DECIMAL(5,2) Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
Christian Gollwitzer : > Am 13.04.17 um 15:20 schrieb Marko Rauhamaa: >> >> Not sure if this is still valid: >> >>Still today Flash RAM cells built in SSDs have a limited lifespan. >>Every write (not read) cycle or better every erasure wears a memory >>cell and at some time it will stop working. >> >>https://askubuntu.com/questions/652337/why-no-swap-partition >>s-on-ssd-drives> >> > > It is true, in general, but the lifetime has gotten MUCH better due to > overprovisioning and intelligent wear leveling. [...] > > For these drives with a capacity of 256GB, the manufacturers > guaranteed ~70 TB written to them, for Samsung 850 Pro and SanDisk > Extreme 150TB were guaranteed. Most drives withstood much more data, > the best one being the Samsung 850 Pro, which did not fail until they > ended the test after writing 4600 TB in half a year. The others failed > at ~1000 TB. None of those numbers sound all that high to me. It's a catch-22: if you rarely use swap, you don't have a problem, but you probably don't need swap space to begin with. However, if you want to lean on swap space heavily and create a Python program that operates on 100 GB of memory, garbage collections are going to create quite a rewrite load on the disk. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
On 14/04/2017 02:44, Steve D'Aprano wrote: On Fri, 14 Apr 2017 12:52 am, bartc wrote: I know this isn't the Python need-for-speed thread, but this is a classic example where the lack of one simple feature leads to using slower, more cumbersome ones. Dear gods, have I fallen back in time to 1975 again? The Goto Wars are over, and the structured programming camp won the war decisively. Are functions/procedures *technically* slower than GOTOs? Yes. So what? You're nano-optimizing the wrong thing: you're saving a microsecond of runtime at the cost of a megasecond of development and maintenance time. Usually, yes. (And when you run CPython, whatever speed you do get, is partly thanks to compiler writers choosing jumps, or in-lining, over function calls. All those nanoseconds add up.) However my comment wasn't specifically about goto. It's about using a slower feature to get around the lack of another feature that could have been implemented efficiently. Function calls, returns and generators are rather more heavyweight in comparison. And enormously better. It depends if your application is speed-sensitive. Much scripting code isn't, or if it is, most of the work is done outside the language. But sometimes you want your dynamic code to do some actual work of its own that involves executing byte-code, rather than calling some compiled code. And performance can matter. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 14/04/2017 03:13, Steve D'Aprano wrote:
On Wed, 12 Apr 2017 07:56 pm, bart4...@gmail.com wrote:
if you care most about program correctness, type-safety and
correctness proofs, choosing C is probably the wrong decision.
CPython is written in C; bad choice?
(** Although I find code full of class definitions, one-liners, decorators
and all the other esoterics, [incomprehensible]... I'm not the only
one, so perhaps readability isn't too much of a priority either.)
Classes and decorators are not esoteric. You sound like an old man who
complains about this new-fangled "telephone", and how things were so much
better when we had messenger boys to deliver messages anywhere in the city.
These days I like code to be as simple and obvious as possible. I guess
no one's going to complain when it is!
I'm not sure what you consider incomprehensible about "one liners", surely
it depends on what the one liner does. These are pretty clear to me:
# The archetypal "Hello World!" program is a one-liner.
print("Hello World!")
# One liner to calculate the sum of the first 50 cubes.
total = sum(i**3 for i in range(50))
Those are fine. I means when lots of things are arranged on one line,
where the output of one method feeds into the next.
Vertical space is free after all.
However, you might argue that having discrete, intermediate steps would
be a little slower than one-lining..
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Fri, Apr 14, 2017 at 8:42 PM, bartc wrote:
>> Classes and decorators are not esoteric. You sound like an old man who
>> complains about this new-fangled "telephone", and how things were so much
>> better when we had messenger boys to deliver messages anywhere in the
>> city.
>
>
> These days I like code to be as simple and obvious as possible. I guess no
> one's going to complain when it is!
Everyone says that, but there's a stark division between two groups of people:
1) Those who think the code should have an obvious concrete meaning -
that you should be able to see exactly what bits get flipped in memory
2) Those who think the code should have an obvious abstract meaning -
that you should be able to see exactly what the programmer intended.
Declarative notations like decorators can beautifully provide the
second, even though they might obscure the first. For example:
@app.route("/hello/")
def greet(name):
return f"A very warm hello to you too, {name}!"
Nice, clean code that might be found in a Flask application. In
concrete terms, there's a lot going on, but in abstract terms it's
simple: when this kind of URL is requested, return this formatted
string.
I'm of the opinion that the concrete meaning is insignificant, and
your code should clearly portray its abstract meaning. So classes,
decorators, etc, etc, etc are all excellent tools for *improving*
readability.
Of course, if you feel the other way, you're most welcome to write
everything in C.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 14/04/2017 11:56, Chris Angelico wrote:
On Fri, Apr 14, 2017 at 8:42 PM, bartc wrote:
Classes and decorators are not esoteric. You sound like an old man who
complains about this new-fangled "telephone", and how things were so much
better when we had messenger boys to deliver messages anywhere in the
city.
These days I like code to be as simple and obvious as possible. I guess no
one's going to complain when it is!
Everyone says that, but there's a stark division between two groups of people:
1) Those who think the code should have an obvious concrete meaning -
that you should be able to see exactly what bits get flipped in memory
2) Those who think the code should have an obvious abstract meaning -
that you should be able to see exactly what the programmer intended.
Declarative notations like decorators can beautifully provide the
second, even though they might obscure the first. For example:
@app.route("/hello/")
def greet(name):
return f"A very warm hello to you too, {name}!"
Nice, clean code that might be found in a Flask application. In
concrete terms, there's a lot going on, but in abstract terms it's
simple: when this kind of URL is requested, return this formatted
string.
I'm of the opinion that the concrete meaning is insignificant, and
your code should clearly portray its abstract meaning. So classes,
decorators, etc, etc, etc are all excellent tools for *improving*
readability.
C++ has classes, templates and so on, all of which can be argued can
improve readability (when you eventually find that code that actually
does something).
But somehow it doesn't really work. Because such features are overused?
I don't know. I do know that if I want to port some program (be it in
Python or C++), or simply try and understand it, if I see it's full of
class definitions or whatever, then I won't bother.
Of course, if you feel the other way, you're most welcome to write
everything in C.
'pseudo-code' is often the language used to describe an algorithm in
terms that are independent of any particular language. It doesn't
usually contain classes or decorators.
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Fri, Apr 14, 2017 at 10:04 PM, bartc wrote: > >> Of course, if you feel the other way, you're most welcome to write >> everything in C. > > > 'pseudo-code' is often the language used to describe an algorithm in terms > that are independent of any particular language. It doesn't usually contain > classes or decorators. Decorators no, classes sometimes. And then you try to transform that pseudocode into executable code as closely as possible, and you need some sort of feature that makes these things functional. For instance, the pseudocode for the example I gave might be: /hello/[name]: return "A very warm hello to you too, [name]!" ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Command Line Arguments
On 2017-04-13, Jason Friedman wrote: >> I have this code which I got from https://www.tutorialspoint. >> com/python/python_command_line_arguments.htm The example works fine but >> when I modify it to what I need, it only half works. The problem is the >> try/except. If you don't specify an input/output, they are blank at the end >> but it shouldn't be. >> >> import getopt >> import sys > > I am guessing you are wanting to parse command-line arguments rather than > particularly wanting to use the getopt module. > If I am correct you might want to spend your time instead learning the > argparse module: > https://docs.python.org/3/library/argparse.html > https://docs.python.org/3/howto/argparse.html He should switch to argparse in any case because getopt is no longer supported and does only receive bugfixes. Bernd -- Die Antisemiten vergeben es den Juden nicht, dass die Juden ‘Geist’ haben – und Geld. Die Antisemiten – ein Name der ‘Schlechtweggekommenenen’ [Friedrich Nietzsche] -- https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
On 2017-04-13, Mikhail V wrote: > On 13 April 2017 at 18:48, Ian Kelly wrote: >> On Thu, Apr 13, 2017 at 10:23 AM, Mikhail V wrote: >>> Now I wonder, have we already collected *all* bells and whistles of Python >>> in these two examples, or is there something else for expressing trivial >>> thing. >> >> Functions and exceptions are considered "bells and whistles"? > > You mean probably classes and exceptions? For me, indeed they are, > but it depends. > > And breaking the code into def() chunks that are not > functions but just chunks... I don't know, looks bad. Organising code in a bunch of small functions is by far better coding style and better readable than put it all together in one chunk. And that holds for all programming languages, not only for Python. Bernd -- Die Antisemiten vergeben es den Juden nicht, dass die Juden ‘Geist’ haben – und Geld. Die Antisemiten – ein Name der ‘Schlechtweggekommenenen’ [Friedrich Nietzsche] -- https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
On 2017-04-13, Grant Edwards wrote:
> On 2017-04-13, Rob Gaddi wrote:
>
>> No, C doesn't support exception handling. As a result, handling error
>> conditions in C is a huge pain for which (forward-only) goto is often,
>> while not the only remedy, the least painful one.
>
> Indeed. That is almost the only place I use 'goto' in C, and the
> almost the only place I see others use it. Very occasionally, you see
> a the error handling goto refactored into a backwards "goto retry":
>
> this code
>
> foo()
> {
> while (1)
> {
>
> if ()
>goto error:
>
> return;
>
> error:
>
> }
> }
foo()
{
int done = 0;
while (! done)
{
if () {
} else {
done = 1;
}
}
Bernd
--
Die Antisemiten vergeben es den Juden nicht, dass die Juden ‘Geist’
haben – und Geld. Die Antisemiten – ein Name der
‘Schlechtweggekommenenen’ [Friedrich Nietzsche]
--
https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
Bernd Nawothnig wrote: > On 2017-04-13, Mikhail V wrote: > > On 13 April 2017 at 18:48, Ian Kelly wrote: > >> On Thu, Apr 13, 2017 at 10:23 AM, Mikhail V wrote: > >>> Now I wonder, have we already collected *all* bells and whistles of Python > >>> in these two examples, or is there something else for expressing trivial > >>> thing. > >> > >> Functions and exceptions are considered "bells and whistles"? > > > > You mean probably classes and exceptions? For me, indeed they are, > > but it depends. > > > > And breaking the code into def() chunks that are not > > functions but just chunks... I don't know, looks bad. > > Organising code in a bunch of small functions is by far better coding > style and better readable than put it all together in one chunk. And > that holds for all programming languages, not only for Python. > The functions need *some* reason for being an entity though. -- Chris Green · -- https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
On Sat, Apr 15, 2017 at 12:13 AM, Dennis Lee Bieber wrote: > On Thu, 13 Apr 2017 18:36:57 -0600, Ian Kelly > declaimed the following: > >> >>Well, you can do it in Assembly. And BASIC, if you count the primitive >>GOSUB-type subroutines, though modern BASICs have real subroutines >>that don't allow it. >> > REXX probably allows it too... (No GOTO, but the SIGNAL statement can > do unconditional jumps to named labels)... Hmmm, if I read the manual > correctly, any use of SIGNAL will terminate loops, even if the SIGNAL and > target are both within the same loop. This is correct. Annoyingly, it also wipes out indentation in the TRACE output, so you really want to use it *only* for error handling (which is its stated purpose). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Hi! i need some help with a program in python on Raspberry pi3.
every time i run the program i get this messeage:
Traceback (most recent call last):
File "smartmirror.py", line 159, in get_weather
temprature2 = "%S%S" % (str(int(weather_obj['currently']['temperature'])),
degree_sign)
KeyError: 'currently'
Error: 'currently'. Cannot get weather.
How do i fix that?
Here is the program:
# smartmirror.py
# requirements
# requests, feedparser, traceback, Pillow
from Tkinter import *
import locale
import threading
import time
import requests
import json
import traceback
import feedparser
from PIL import Image, ImageTk
from contextlib import contextmanager
LOCALE_LOCK = threading.Lock()
ui_locale = '' # e.g. 'fr_FR' fro French, '' as default
time_format = 24 # 12 or 24
date_format = "%b %d, %Y" # check python doc for strftime() for options
news_country_code = 'nb'
weather_api_token = '16dc67b56f94f8083b1afed7e69c5dc1' # create account at
https://darksky.net/dev/
weather_lang = 'nb' # see https://darksky.net/dev/docs/forecast for full list
of language parameters values
weather_unit = 'nb' # see https://darksky.net/dev/docs/forecast for full list
of unit parameters values
latitude = (59.1311800) # Set this if IP location lookup does not work for you
(must be a string)
longitude = (10.2166500) # Set this if IP location lookup does not work for you
(must be a string)
xlarge_text_size = 94
large_text_size = 48
medium_text_size = 28
small_text_size = 18
@contextmanager
def setlocale(name): #thread proof function to work with locale
with LOCALE_LOCK:
saved = locale.setlocale(locale.LC_ALL)
try:
yield locale.setlocale(locale.LC_ALL, name)
finally:
locale.setlocale(locale.LC_ALL, saved)
# maps open weather icons to
# icon reading is not impacted by the 'lang' parameter
icon_lookup = {
'clear-day': "assets/Sun.png", # clear sky day
'wind': "assets/Wind.png", #wind
'cloudy': "assets/Cloud.png", # cloudy day
'partly-cloudy-day': "assets/PartlySunny.png", # partly cloudy day
'rain': "assets/Rain.png", # rain day
'snow': "assets/Snow.png", # snow day
'snow-thin': "assets/Snow.png", # sleet day
'fog': "assets/Haze.png", # fog day
'clear-night': "assets/Moon.png", # clear sky night
'partly-cloudy-night': "assets/PartlyMoon.png", # scattered clouds night
'thunderstorm': "assets/Storm.png", # thunderstorm
'tornado': "assests/Tornado.png",# tornado
'hail': "assests/Hail.png" # hail
}
class Clock(Frame):
def __init__(self, parent, *args, **kwargs):
Frame.__init__(self, parent, bg='black')
# initialize time label
self.time1 = ''
self.timeLbl = Label(self, font=('Helvetica', large_text_size),
fg="white", bg="black")
self.timeLbl.pack(side=TOP, anchor=E)
# initialize day of week
self.day_of_week1 = ''
self.dayOWLbl = Label(self, text=self.day_of_week1, font=('Helvetica',
small_text_size), fg="white", bg="black")
self.dayOWLbl.pack(side=TOP, anchor=E)
# initialize date label
self.date1 = ''
self.dateLbl = Label(self, text=self.date1, font=('Helvetica',
small_text_size), fg="white", bg="black")
self.dateLbl.pack(side=TOP, anchor=E)
self.tick()
def tick(self):
with setlocale(ui_locale):
if time_format == 12:
time2 = time.strftime('%I:%M %p') #hour in 12h format
else:
time2 = time.strftime('%H:%M') #hour in 24h format
day_of_week2 = time.strftime('%A')
date2 = time.strftime(date_format)
# if time string has changed, update it
if time2 != self.time1:
self.time1 = time2
self.timeLbl.config(text=time2)
if day_of_week2 != self.day_of_week1:
self.day_of_week1 = day_of_week2
self.dayOWLbl.config(text=day_of_week2)
if date2 != self.date1:
self.date1 = date2
self.dateLbl.config(text=date2)
# calls itself every 200 milliseconds
# to update the time display as needed
# could use >200 ms, but display gets jerky
self.timeLbl.after(200, self.tick)
class Weather(Frame):
def __init__(self, parent, *args, **kwargs):
Frame.__init__(self, parent, bg='black')
self.temperature = ''
self.forecast = ''
self.location = ''
self.currently = ''
self.icon = ''
self.degreeFrm = Frame(self, bg="black")
self.degreeFrm.pack(side=TOP, anchor=W)
self.temperatureLbl = Label(self.degreeFrm, font=('Helvetica',
xlarge_text_size), fg="white", bg="black")
self.temperatureLbl.pack(side=LEFT, anchor=N)
self.iconLbl = Label(self.degreeFrm, bg="black")
self.iconLbl.pack(side=LEFT, anchor=N, padx=20)
self.currentlyLbl = Label(self, font=('Helvetica', medium_text_size),
fg="white", bg="black")
Re: Hi! i need some help with a program in python on Raspberry pi3.
On Sat, 15 Apr 2017 12:27 am, Kasper wrote: > every time i run the program i get this messeage: > > Traceback (most recent call last): > File "smartmirror.py", line 159, in get_weather > temprature2 = "%S%S" % > (str(int(weather_obj['currently']['temperature'])), > degree_sign) > KeyError: 'currently' > Error: 'currently'. Cannot get weather. > > How do i fix that? We're unpaid volunteers, we're not being paid to debug your code. Don't dump over 300 lines of code in our lap and expect us to debug it. If you can't spend the time simplifying the problem and removing all the irrelevant details, how can you expect us to do it for you? If you want help, make it easy for people to help you. Cut your code down to the simplest, shortest amount of code that demonstrates the problem. If your code has nothing to do with the GUI, cut out all the GUI code. More advice here: http://sscce.org/ -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Regular Expressions, Speed, Python, and NFA
I am running some tests using the site regex101 to figure out the correct regexs to use for a project. I was surprised at how slow it was, constantly needing to increase the timeouts. I went Googling for a reason, and solution, and found Russ Cox’s article from 2007: https://swtch.com/~rsc/regexp/regexp1.html . I couldn’t understand why, if this was even remotely correct, we don’t use NFA in Python, which led me here: https://groups.google.com/forum/#!msg/comp.lang.python/L1ZFI_R2hAo/C12Nf3patWIJ;context-place=forum/comp.lang.python where all of these issues were addressed. Unfortunately, this is also from 2007. BTW, John Machin in one of his replies cites Navarro’s paper, but that link is broken. Navarro’s work can now be found at http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.21.3112&rep=rep1&type=pdf But be forewarned, it is 68 pages of dense reading. I am not a computer science major. I am not new to Python, but I don’t think I’m qualified to take on the idea of creating a new NFA module for Python. >Getting back to the "It would be nice ..." bit: yes, it would be nice >to have even more smarts in re, but who's going to do it? It's not a >"rainy Sunday afternoon" job :-) >Cheers, >John - >Well, just as an idea, there is a portable C library for this at >http://laurikari.net/tre/ released under LGPL. If one is willing to >give up PCRE extensions for speed, it might be worth the work to >wrap this library using SWIG. >Kirk Sluder (BTW, this link is also old. TRE is now at https://github.com/laurikari/tre/ ) I am not a computer science major. I am not new to Python, but I don’t think I’m qualified to take on the idea of creating a new NFA module for Python. Nor am I entirely sure I want to try something new (to me) like TRE. Most threads related to this topic are older than 2007. I did find this https://groups.google.com/forum/#!searchin/comp.lang.python/regex$20speed%7Csort:relevance/comp.lang.python/O7rUwVoD2t0/NYAQM0mUX7sJ from 2011 but I did not do an exhaustive search. The bottom line is I wanted to know if anything has changed since 2007, and if there is a) any hope for improving regex speeds in Python, or b) some 3rd party module/library that is already out there and solves this problem? Or should I just take this advice? >The cheap way in terms of programmer time is to pipe out to grep or >awk on this one. >Kirk Sluder Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Regular Expressions, Speed, Python, and NFA
On Sat, 15 Apr 2017 01:12 am, Malik Rumi wrote: > I couldn’t understand why, if this was even remotely correct, > we don’t use NFA in Python [...] > I don’t think I’m qualified to take on the idea of creating > a new NFA module for Python. If not you, then who should do it? Python is open source and written by volunteers. If you want something done, there are only four possibilities: - you do it yourself; - you wait as long as it takes for somebody else to do it; - you pay somebody to do it; - or it doesn't get done. If you're not willing or able to do the work yourself, does that help you understand why we don't use NFA? -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Hi! i need some help with a program in python on Raspberry pi3.
Kasper wrote: > every time i run the program i get this messeage: > > Traceback (most recent call last): > File "smartmirror.py", line 159, in get_weather > temprature2 = "%S%S" % (str(int(weather_obj['currently'] ['temperature'])), > degree_sign) > KeyError: 'currently' > Error: 'currently'. Cannot get weather. > > How do i fix that? > > Here is the program: > > weather_api_token = '16dc67b56f94f8083b1afed7e69c5dc1' # create account at > https://darksky.net/dev/ Oops, is that your personal token? You are probably not supposed to publish it and now run the risk of having it revoked. > values weather_unit = 'nb' # see https://darksky.net/dev/docs/forecast for > full list of unit parameters values You have to replace nb with one of the allowed values listed on the site. > weather_lang = 'nb' # see > https://darksky.net/dev/docs/forecast for full list of language parameters Again, you have to pick one of the values listed -- unless you want Norsk bokmål that is. -- https://mail.python.org/mailman/listinfo/python-list
RE: Regular Expressions, Speed, Python, and NFA
-Original Message- From: Python-list [mailto:python-list- bounces+jcasale=activenetwerx@python.org] On Behalf Of Malik Rumi Sent: Friday, April 14, 2017 9:12 AM To: python-list@python.org Subject: Regular Expressions, Speed, Python, and NFA > I am running some tests using the site regex101 to figure out the correct > regexs to use for a project. I was surprised at how slow it was, constantly > needing to increase the timeouts. I went Googling for a reason, and solution, > and found Russ Cox’s article from 2007: > https://swtch.com/~rsc/regexp/regexp1.html . I couldn’t understand why, if > this was even remotely correct, we don’t use NFA in Python, which led me > here: Do you have any sample data you can share? -- https://mail.python.org/mailman/listinfo/python-list
Re: Regular Expressions, Speed, Python, and NFA
Malik Rumi wrote: > I am running some tests using the site regex101 to figure out the correct > regexs to use for a project. I was surprised at how slow it was, > constantly needing to increase the timeouts. I went Googling for a reason, > and solution, and found Russ Cox’s article from 2007: > https://swtch.com/~rsc/regexp/regexp1.html . I couldn’t understand why, if > this was even remotely correct, we don’t use NFA in Python, which led me > here: You might try https://en.wikipedia.org/wiki/RE2_(software) for which Python wrappers are available. However, "RE2 does not support back-references, which cannot be implemented efficiently." -- https://mail.python.org/mailman/listinfo/python-list
Re: "Goto" statement in Python
On 04/14/2017 07:19 AM, Dennis Lee Bieber wrote: On Fri, 14 Apr 2017 11:44:59 +1000, Steve D'Aprano declaimed the following: Even that's not enough for some. Donald Knuth, who supports the use of GOTO under some circumstances, maintains that any program using GOTOs should have the invariant that its flow chart can be drawn with all forward branches on the left, all backward branches on the right, and no branches crossing each other. Good thing I never had him for an instructor... My practice, when last flow-charting, favored back-branches on the left. After all, I read left-to-right/top-to-bottom, so forward branches going right and down seem natural. Encountering a branch going left sort of implies "re-reading" part of the chart; going upwards... Yeah, but I love the naked concept there. It pretty much encapsulates what I find to be the only useful use of GOTO in any reasonable language in a simple, easy to visualize way. If your GOTOs are crossing you've done something wrong. If more people had good mechanisms for visualizing their code I'd use far less profanity literally every day of my life. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Regular Expressions, Speed, Python, and NFA
On 04/14/2017 08:12 AM, Malik Rumi wrote: I am running some tests using the site regex101 to figure out the correct regexs to use for a project. I was surprised at how slow it was, constantly needing to increase the timeouts. I went Googling for a reason, and solution, and found Russ Cox’s article from 2007: https://swtch.com/~rsc/regexp/regexp1.html . I couldn’t understand why, if this was even remotely correct, we don’t use NFA in Python, which led me here: https://groups.google.com/forum/#!msg/comp.lang.python/L1ZFI_R2hAo/C12Nf3patWIJ;context-place=forum/comp.lang.python where all of these issues were addressed. Unfortunately, this is also from 2007. BTW, John Machin in one of his replies cites Navarro’s paper, but that link is broken. Navarro’s work can now be found at http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.21.3112&rep=rep1&type=pdf But be forewarned, it is 68 pages of dense reading. I am not a computer science major. I am not new to Python, but I don’t think I’m qualified to take on the idea of creating a new NFA module for Python. Getting back to the "It would be nice ..." bit: yes, it would be nice to have even more smarts in re, but who's going to do it? It's not a "rainy Sunday afternoon" job :-) Cheers, John - Well, just as an idea, there is a portable C library for this at http://laurikari.net/tre/ released under LGPL. If one is willing to give up PCRE extensions for speed, it might be worth the work to wrap this library using SWIG. Kirk Sluder (BTW, this link is also old. TRE is now at https://github.com/laurikari/tre/ ) I am not a computer science major. I am not new to Python, but I don’t think I’m qualified to take on the idea of creating a new NFA module for Python. Nor am I entirely sure I want to try something new (to me) like TRE. Most threads related to this topic are older than 2007. I did find this https://groups.google.com/forum/#!searchin/comp.lang.python/regex$20speed%7Csort:relevance/comp.lang.python/O7rUwVoD2t0/NYAQM0mUX7sJ from 2011 but I did not do an exhaustive search. The bottom line is I wanted to know if anything has changed since 2007, and if there is a) any hope for improving regex speeds in Python, or b) some 3rd party module/library that is already out there and solves this problem? Or should I just take this advice? The cheap way in terms of programmer time is to pipe out to grep or awk on this one. Kirk Sluder Thanks. I'll also throw in the obligatory quote from Jamie Zawinsky, "Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems." It's not that regexes are the wrong tool for any job; I personally use them all the time. But they're, tautologically, the wrong tool for any job that can be done better with a different one. In Python, you've got "in", .startswith, .endswith, and .split to handle simple parsing tasks. On the other end, you've got lxml and the like to handle complex tasks that provably cannot be done with regexes at all, let alone efficiently. This leaves them in a fairly bounded middle ground, where is my task so complex that it warrants something as difficult to read as a regex, but still simple enough to be solved by one. And that ground is definitely occupied. But not vast. So is it worth the time to try to write a more efficient regex parser for Python? Yours if you want it to be, but not mine. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 04/13/2017 08:13 PM, Steve D'Aprano wrote: > On Wed, 12 Apr 2017 07:56 pm, bart4...@gmail.com wrote: > [...] >> (** Although I find code full of class definitions, one-liners, decorators >> and all the other esoterics, incomprehensive. I'm sure I'm not the only >> one, so perhaps readability isn't too much of a priority either.) > > [...] You sound like an old man who > complains about this new-fangled "telephone", and how things were so much > better when we had messenger boys to deliver messages anywhere in the city. This is the second time in two weeks that I have seen an offensive stereotype presented in this list. Last time it was "ugly americans". For those (who've been living under a rock) that are unaware, the stereotype of the "old person" who can't understand or adapt to new technology is a very vile one that many believe is in large part responsible for the high unemployment rate of over-50's in the US, along with their positions near the tops of lay-off lists. This form of bigotry is called "ageism". There are plenty of ways to use the english language to make your point (that people often prefer that which they're familiar with) without employing hateful stereotypes directed at a specific groups of people. Given that this is the second time in two weeks, I publicly ask the list moderators to invoke whatever procedures they use to deal with bigotry on this list. -- https://mail.python.org/mailman/listinfo/python-list
Re: Hi! i need some help with a program in python on Raspberry pi3.
On 4/14/2017 10:27 AM, Kasper wrote: every time i run the program i get this messeage: Traceback (most recent call last): File "smartmirror.py", line 159, in get_weather temprature2 = "%S%S" % (str(int(weather_obj['currently']['temperature'])), degree_sign) KeyError: 'currently' Error: 'currently'. Cannot get weather. How do i fix that? Use a valid key for weather_obj. Check doc or add print(list(weather_obj.keys())) before the failing subscription. Here is the program: I second Steve's comment. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
Peter,
Retracing my steps to rewrite the getattr(row, label) code, this is what
sent me down the rabbit hole in the first place. (I changed your 'rows'
to 'records' just to use the same name everywhere, but all else is the
same as you gave me.) I'd like you to look at it and see if you still
think complete(group, label) should work. Perhaps seeing why it fails
will clarify some of the difficulties I'm having.
I ran into problems with values and has_empty. values has a problem
because
row[label] gets a TypeError. has_empty has a problem because a list of
field values will be shorter with missing values than a full list, but a
namedtuple with missing values will be the same length as a full
namedtuple since missing values have '' placeholders. Two more
unexpected inconveniences.
A short test csv is at the end, for you to read in and attempt to
execute the following code, and I'm still working on reconstructing the
lost getattr(row, label) code.
import csv
from collections import namedtuple, defaultdict
def get_title(row):
return row.title
def complete(group, label):
values = {row[label] for row in group}
# get "TypeError: tuple indices must be integers, not str"
has_empty = not min(values, key=len)
if len(values) - has_empty != 1:
# no value or multiple values; manual intervention needed
return False
elif has_empty:
for row in group:
row[label] = max(values, key=len)
return True
infile = open("E:\\Coding projects\\Pycharm\\Moving\\Moving 2017 in -
test.csv")
rows = csv.reader(infile)
fieldnames = next(rows)
Record = namedtuple("Record", fieldnames)
records = [Record._make(fieldnames)]
records.extend(Record._make(row) for row in rows)
# group rows by title
groups = defaultdict(list)
for row in records:
groups[get_title(row)].append(row)
LABELS = ['Location', 'Kind', 'Notes']
# add missing values
for group in groups.values():
for label in LABELS:
complete(group, label)
Moving 2017 in - test.csv:
(If this doesn't come through the mail system correctly, I've also
uploaded the file to
http://deborahswanson.net/python/Moving%202017%20in%20-%20test.csv.
Permissions should be set correctly, but let me know if you run into
problems downloading the file.)
CLDesc,url,title,Description,Location,ST,co,miles,Kind,Rent,Date,first,b
r,Notes,yesno,mark,arc
Jan 3 1 bedroom/1 bath mobile $700 1br - (Mount
Vernon),http://skagit.craigslist.org/apa/5943737902.html,1 bedroom/1
bath mobile $700 1br - (Mount Vernon) - (Mount Vernon),1 bedroom/1 bath
mobile $700 1br,Mount Vernon,WA,sk,,trailer,700,1/3/2017,1/3/2017,1,no
smoking,,,deleted by its author
Jan 6 1 bedroom/1 bath mobile $700 1br - (Mount
Vernon),http://skagit.craigslist.org/apa/5943737902.html,1 bedroom/1
bath mobile $700 1br - (Mount Vernon) - (Mount Vernon),1 bedroom/1 bath
mobile $700 1br,,WA,,,trailer,700,1/6/2017,1/3/2017,1,no smoking,,,
Jan 10 1 bedroom/1 bath mobile $700 1br - (Mount
Vernon),http://skagit.craigslist.org/apa/5943737902.html,1 bedroom/1
bath mobile $700 1br - (Mount Vernon) - (Mount Vernon),1 bedroom/1 bath
mobile $700 1br,,700,1/10/2017,1/3/2017,1
Jan 17 1 bedroom/1 bath mobile $700 1br - (Mount
Vernon),http://skagit.craigslist.org/apa/5943737902.html,1 bedroom/1
bath mobile $700 1br - (Mount Vernon) - (Mount Vernon),1 bedroom/1 bath
mobile $700 1br,Mount Vernon,WA,,,trailer,700,1/17/2017,1/3/2017,1,no
smoking,,,
Jan 19 1 bedroom/1 bath mobile $700 1br - (Mount
Vernon),http://skagit.craigslist.org/apa/5943737902.html,1 bedroom/1
bath mobile $700 1br - (Mount Vernon) - (Mount Vernon),1 bedroom/1 bath
mobile $700 1br,Mount Vernon,WA,,,trailer,700,1/19/2017,1/3/2017,1,no
smoking,,,
Jan 26 1240 8th Avenue $725 2br - 676ft2 -
(Longview),http://portland.craigslist.org/clk/apa/5976442500.html,1240
8th Avenue $725 2br - 676ft2 - (Longview),1240 8th Avenue $725 2br -
676ft2,,725,1/26/2017,1/16/2017,2
Jan 16 1240 8th Avenue $725 2br - 676ft2 -
(Longview),http://portland.craigslist.org/clk/apa/5961794305.html,1240
8th Avenue $725 2br - 676ft2 - (Longview) - (Longview),1240 8th Avenue
$725 2br -
676ft2,Longview,WA,,,house,725,1/16/2017,1/16/2017,2,"detached garage,
w/d hookups",,,
Jan 6 1424 California Avenue $750 2br - 1113ft2 - (Klamath
Falls),http://klamath.craigslist.org/apa/5947977083.html,1424 California
Avenue $750 2br - 1113ft2 - (Klamath Falls) - (Klamath Falls),1424
California Avenue $750 2br - 1113ft2,Klamath
Falls,OR,kl,,house,750,1/6/2017,1/6/2017,2,no smoking,,,
Jan 11 1424 California Avenue $750 2br - 1113ft2 - (Klamath
Falls),http://klamath.craigslist.org/apa/5947977083.html,1424 California
Avenue $750 2br - 1113ft2 - (Klamath Falls) - (Klamath Falls),1424
California Avenue $750 2br -
1113ft2,,OR,kl,,house,750,1/11/2017,1/6/2017,2,no smoking,,,
"Jan 3 1838 Alma Drive Kelso, WA 98626 $550 1br - 600ft2 - (1838 Alma
Drive Kelso,
WA)",http://portland.craigslist.org/clk/apa/5937961608.html,"1838 Alma
Drive Kelso, WA 98626 $550 1br - 600ft2
Python3 C extension how to support dir()
I have an python3 C++ extension that works. But dir(obj) does not return the list of member variables. How should I support dir() from the C API? I cannot use tp_members, not appropiate to the code I'm writing. If I provide __dir__ I then hit the problem I need the C API version all_attr = super( mytype, obj ).__dir__(); all_attr.extend( mytype_variable_names ); I'm not getting inspiration from the python 3.6 sources for this problem. I did find the object_dir function in typeobject.c, but that has not helped get me forward. What am I missing? Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Goto Considered Harmful [was Re: Python and the need for speed]
r...@zedat.fu-berlin.de (Stefan Ram):
> struct example
> { PIC<5,X<15>> last_name;
> PIC<88,VALUE<1,3,5,7,9>> odd_numbers; }
>
> . The above assumes appropriate definitions for »VALUE« (as a
> variadic template) »PIC«, and »X«. Only a C++ expert would be able
> to provide these definitions, but then they could be used by average
> C++ programmers.
I don't like your expert/average dichotomy. Similar dichotomies plague
other frameworks: Eclipse, Jenkins, Maven, Xt, MS Word etc.
If templates are considered a good thing, creating them should be
considered routine for *all* C++ programmers.
> . C++ has a very expressive user-definable type system including
> user-definable literals.
I don't like said system. Analogously, I'm not too keen on formal
schemata. Even more generally, I'm not big on *rule languages*, which
are ad-hoc and incomplete. Instead, I prefer *programming languages*,
which are compact and expressive, or even plain English.
> It should then also be possible to generate compile-time errors for
> assignments such as
>
> example_instance.odd_numbers = 4;
Every programming language has its big selling points. C++'s gospel is
its utmost compile-time checking. I'm a long time user of C++, but I'm
not a convert to that ideology.
> . I know too little about Python to tell whether something like this
> would be possible with Python, too.
Unfortunately, Python is taking baby steps in that direction.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
Deborah Swanson wrote:
> Peter,
>
> Retracing my steps to rewrite the getattr(row, label) code, this is what
> sent me down the rabbit hole in the first place. (I changed your 'rows'
> to 'records' just to use the same name everywhere, but all else is the
> same as you gave me.) I'd like you to look at it and see if you still
> think complete(group, label) should work. Perhaps seeing why it fails
> will clarify some of the difficulties I'm having.
>
> I ran into problems with values and has_empty. values has a problem
> because
> row[label] gets a TypeError. has_empty has a problem because a list of
> field values will be shorter with missing values than a full list, but a
> namedtuple with missing values will be the same length as a full
> namedtuple since missing values have '' placeholders. Two more
> unexpected inconveniences.
>
> A short test csv is at the end, for you to read in and attempt to
> execute the following code, and I'm still working on reconstructing the
> lost getattr(row, label) code.
>
> import csv
> from collections import namedtuple, defaultdict
>
> def get_title(row):
> return row.title
>
> def complete(group, label):
> values = {row[label] for row in group}
> # get "TypeError: tuple indices must be integers, not str"
Yes, the function expects row to be dict-like. However when you change
row[label]
to
getattr(row, label)
this part of the code will work...
> has_empty = not min(values, key=len)
> if len(values) - has_empty != 1:
> # no value or multiple values; manual intervention needed
> return False
> elif has_empty:
> for row in group:
> row[label] = max(values, key=len)
but here you'll get an error. I made the experiment to change everything
necessary to make it work with namedtuples, but you'll probably find the
result a bit hard to follow:
import csv
from collections import namedtuple, defaultdict
INFILE = "E:\\Coding projects\\Pycharm\\Moving\\Moving 2017 in - test.csv"
OUTFILE = "tmp.csv"
def get_title(row):
return row.title
def complete(group, label):
values = {getattr(row, label) for row in group}
has_empty = not min(values, key=len)
if len(values) - has_empty != 1:
# no value or multiple values; manual intervention needed
return False
elif has_empty:
# replace namedtuples in the group. Yes, it's ugly
fix = {label: max(values, key=len)}
group[:] = [record._replace(**fix) for record in group]
return True
with open(INFILE) as infile:
rows = csv.reader(infile)
fieldnames = next(rows)
Record = namedtuple("Record", fieldnames)
groups = defaultdict(list)
for row in rows:
record = Record._make(row)
groups[get_title(record)].append(record)
LABELS = ['Location', 'Kind', 'Notes']
# add missing values
for group in groups.values():
for label in LABELS:
complete(group, label)
# dump data (as a demo that you do not need the list of all records)
with open(OUTFILE, "w") as outfile:
writer = csv.writer(outfile)
writer.writerow(fieldnames)
writer.writerows(
record for group in groups.values() for record in group
)
One alternative is to keep the original and try to replace the namedtuple
with the class suggested by Gregory Ewing. Then it should suffice to also
change
> elif has_empty:
> for row in group:
> row[label] = max(values, key=len)
to
> elif has_empty:
> for row in group:
setattr(row, label, max(values, key=len))
PS: Personally I would probably take the opposite direction and use dicts
throughout...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Namedtuples: some unexpected inconveniences
On 2017-04-14 20:34, Deborah Swanson wrote:
Peter,
Retracing my steps to rewrite the getattr(row, label) code, this is what
sent me down the rabbit hole in the first place. (I changed your 'rows'
to 'records' just to use the same name everywhere, but all else is the
same as you gave me.) I'd like you to look at it and see if you still
think complete(group, label) should work. Perhaps seeing why it fails
will clarify some of the difficulties I'm having.
I ran into problems with values and has_empty. values has a problem
because
row[label] gets a TypeError. has_empty has a problem because a list of
field values will be shorter with missing values than a full list, but a
namedtuple with missing values will be the same length as a full
namedtuple since missing values have '' placeholders. Two more
unexpected inconveniences.
In the line:
values = {row[label] for row in group}
'group' is a list of records; row is a record (namedtuple).
You can get the members of a namedtuple (also 'normal' tuple) by numeric
index, e.g. row[0], but the point of a namedtuple is that you can get
them by name, as an attribute, e.g. row.Location.
As the name of the attribute isn't fixed, but passed by name, use
getattr(row, label) instead:
values = {getattr(row, label) for row in group}
As for the values:
# Remove the missing value, if present.
values.discard('')
# There's only 1 value left, so fill in the empty places.
if len(values) == 1:
...
The next point is that namedtuples, like normal tuples, are immutable.
You can't change the value of an attribute.
A short test csv is at the end, for you to read in and attempt to
execute the following code, and I'm still working on reconstructing the
lost getattr(row, label) code.
import csv
from collections import namedtuple, defaultdict
def get_title(row):
return row.title
def complete(group, label):
values = {row[label] for row in group}
# get "TypeError: tuple indices must be integers, not str"
has_empty = not min(values, key=len)
if len(values) - has_empty != 1:
# no value or multiple values; manual intervention needed
return False
elif has_empty:
for row in group:
row[label] = max(values, key=len)
return True
infile = open("E:\\Coding projects\\Pycharm\\Moving\\Moving 2017 in -
test.csv")
rows = csv.reader(infile)
fieldnames = next(rows)
Record = namedtuple("Record", fieldnames)
records = [Record._make(fieldnames)]
records.extend(Record._make(row) for row in rows)
# group rows by title
groups = defaultdict(list)
for row in records:
groups[get_title(row)].append(row)
LABELS = ['Location', 'Kind', 'Notes']
# add missing values
for group in groups.values():
for label in LABELS:
complete(group, label)
Moving 2017 in - test.csv:
(If this doesn't come through the mail system correctly, I've also
uploaded the file to
http://deborahswanson.net/python/Moving%202017%20in%20-%20test.csv.
Permissions should be set correctly, but let me know if you run into
problems downloading the file.)
[snip]
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
bartc writes: > I do know that if I want to port some program (be it in Python or > C++), or simply try and understand it, if I see it's full of class > definitions or whatever, then I won't bother. There was a time in the evolution of OOP when inheritance was thought of as a cool and enabling thing for code re-use, and lots of architecture astronautics involved designing deeply nested and complex inheritance hierarchies in programs. These days I think there's more accepted that inheritance is confusing and obscures the control flow in programs, that it was often mis-used, and that while there are still legitimate uses cases for it, it should be used sparingly. That helps quite a lot. C++ templates let you write generics that are often cleaner than using subclasses, though the design of templates can lead to awful code and notoriously bloated and useless error messages. The long-awaited Concepts extension should help some with that. Again though, like anything else, templates work best when used tastefully rather than willy-nilly. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 1:48:57 AM UTC-5, Steven D'Aprano wrote: > On Tue, 11 Apr 2017 21:10:56 -0700, Rick Johnson wrote: > > > high level languages like Python should make it difficult, > > if not impossible, to write sub- optimal code (at least in > > the blatantly obvious cases). > > You mean that Python should make it impossible to write: > > near_limit = [] > near_limit.append(1) > near_limit = len(near_limit) > > instead of: > > near_limit = 1 Yes. > ? I look forward to seeing your version of RickPython that > can enforce that rule. Oh, in case you think I'm making > this up, this example came from some real-life code: > [snip: link] My guess about that obviously superfluous code is that it is an artifact of an early algorithm, but since the code executes without error, the programmer forgot about it. That's my two cents anyways... > Here's another example: > > answer = 0 > for i in range(10): > answer += 1 > > instead of > > answer = 10 > > So... how exactly does the compiler prohibit stupid code? Easy. The same way _any_ optimizer optimizes code, by analysing the code! In this example, the integer named 'answer` is simply being incremented in a superfluous loop. Nothing here is "dynamic". For example: * `range(10)` will always produce a list of the _same_ 10 integers. * `for i in range(10)` will aways iterate over the _same_ 10 integers * `answer += 1` will always increment the "current integer value" by 1 Therefore, since `answer` will always be `10`, the code should be optimized to `answer = 10`. What's so difficult to understand here? Hell, if we have to live with the syntactic burden of type- hints, we should at least be rewarded with an optimizer. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 4:30:16 AM UTC-5, alister wrote: > On Tue, 11 Apr 2017 16:31:16 -0700, Rick Johnson wrote: > > On Tuesday, April 11, 2017 at 9:56:45 AM UTC-5, Steve D'Aprano wrote: > >> On Tue, 11 Apr 2017 07:56 pm, Brecht Machiels wrote: > >> > On 2017-04-11 08:19:31 +, Steven D'Aprano said: > > > > I understand that high performance was never a goal in > > > > CPython development (and Python language design!), but > > > > recent events (DropBox, Google) might help to reconsider > > > > that standpoint. > > > *shrug* It isn't as if high-performance is a > > > requirement for all code. > > But given the choice, no prospective "language shopper" > > is going to choose the slower language over a faster > > language -- at least not from a pool of similar languages > > with similar features (no comparing Python to C, please!). > > So even if you don't need the speed _today_, you may need > > it _tomorrow_. And once you've written a few hundred > > thousand lines of code, well, you're committed to the > > language you chose yesterday. > > but cost is also a factor , not just cost of the tools but > cost in time writing & debugging he software the write. if > they can produce a finished product in half the time with a > "slower" language the speed may not be important > (especially when the application is IO bound & spends 90% > of its time idling anyway). Yes, and that is why many people have, at least historically, picked Python for their project. An interpreted language with a clean syntax and an avoidance of TIM-TOWTDI results not only in a Rapid Development, but also less headaches during the maintenance lifetime of the software. But we cannot _always_ know what the final destination of our software may be. Many times a simple project can grow into an enormous beast, and by the time we realize that Python is too slow, we're faced with only bad choices and worse choices. > if this were not the case then we would all be writing > Assembler Only a few, poor, masochistic souls choose to write assembly, the rest of us try to avoid pain as much as we can. ;-) -- https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
fix = {label: max(values, key=len)}
group[:] = [record._replace(**fix) for record in group]
Peter Otten wrote, on Friday, April 14, 2017 2:16 PM
> > def complete(group, label):
> > values = {row[label] for row in group}
> > # get "TypeError: tuple indices must be integers, not str"
>
> Yes, the function expects row to be dict-like. However when
> you change
>
> row[label]
>
> to
>
> getattr(row, label)
>
> this part of the code will work...
>
> > has_empty = not min(values, key=len)
> > if len(values) - has_empty != 1:
> > # no value or multiple values; manual intervention needed
> > return False
> > elif has_empty:
> > for row in group:
> > row[label] = max(values, key=len)
>
> but here you'll get an error. I made the experiment to change
> everything
> necessary to make it work with namedtuples, but you'll
> probably find the
> result a bit hard to follow:
>
> import csv
> from collections import namedtuple, defaultdict
>
> INFILE = "E:\\Coding projects\\Pycharm\\Moving\\Moving 2017
> in - test.csv" OUTFILE = "tmp.csv"
>
> def get_title(row):
> return row.title
>
> def complete(group, label):
> values = {getattr(row, label) for row in group}
> has_empty = not min(values, key=len)
> if len(values) - has_empty != 1:
> # no value or multiple values; manual intervention needed
> return False
> elif has_empty:
> # replace namedtuples in the group. Yes, it's ugly
> fix = {label: max(values, key=len)}
> group[:] = [record._replace(**fix) for record in group]
> return True
>
> with open(INFILE) as infile:
> rows = csv.reader(infile)
> fieldnames = next(rows)
> Record = namedtuple("Record", fieldnames)
> groups = defaultdict(list)
> for row in rows:
> record = Record._make(row)
> groups[get_title(record)].append(record)
>
> LABELS = ['Location', 'Kind', 'Notes']
>
> # add missing values
> for group in groups.values():
> for label in LABELS:
> complete(group, label)
>
> # dump data (as a demo that you do not need the list of all
> records) with open(OUTFILE, "w") as outfile:
> writer = csv.writer(outfile)
> writer.writerow(fieldnames)
> writer.writerows(
> record for group in groups.values() for record in group
> )
>
> One alternative is to keep the original and try to replace
> the namedtuple
> with the class suggested by Gregory Ewing. Then it should
> suffice to also
> change
>
> > elif has_empty:
> > for row in group:
> > row[label] = max(values, key=len)
>
> to
>
> > elif has_empty:
> > for row in group:
> setattr(row, label, max(values, key=len))
>
> PS: Personally I would probably take the opposite direction
> and use dicts
> throughout...
Ok, thank you. I haven't run it on a real input file yet, but this seems
to work with the test file.
Because the earlier incarnation defined 'values' as
values = {row[label] for row in group}
I'd incorrectly guessed what was going on in
has_empty = not min(values, key=len).
Now that
values = {getattr(row, label) for row in group}
works properly as you intended it to, I see you get the set of unique
values for that label in that group, which makes the rest of it make
sense.
I know it's your "ugly" answer, but can I ask what the '**' in
fix = {label: max(values, key=len)}
group[:] = [record._replace(**fix) for record in group]
means?
I haven't seen it before, and I imagine it's one of the possible
'kwargs' in 'somenamedtuple._replace(kwargs)', but I have no idea where
to look up the possible 'kwargs'. (probably short for keyword args)
Also, I don't see how you get a set for values with the notation you
used. Looks like if anything you've got a comprehension that should give
you a dict. (But I haven't worked a lot with sets either.)
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
MRAB wrote, on Friday, April 14, 2017 2:19 PM
>
> In the line:
>
> values = {row[label] for row in group}
>
> 'group' is a list of records; row is a record (namedtuple).
>
> You can get the members of a namedtuple (also 'normal' tuple) by
numeric
> index, e.g. row[0], but the point of a namedtuple is that you can get
> them by name, as an attribute, e.g. row.Location.
>
> As the name of the attribute isn't fixed, but passed by name, use
> getattr(row, label) instead:
>
> values = {getattr(row, label) for row in group}
>
> As for the values:
>
> # Remove the missing value, if present.
> values.discard('')
>
> # There's only 1 value left, so fill in the empty places.
> if len(values) == 1:
> ...
Thanks for this, but honestly, I'm namedtupled-out at the moment and I
have several other projects I'd like to be working on. But I saved your
suggestion with ones that others have made, so I'll revisit yours again
when I come back for another look at namedtuples.
> The next point is that namedtuples, like normal tuples, are immutable.
> You can't change the value of an attribute.
No you can't, but you can use
somenamedtuple._replace(kwargs)
to replace the value. Works just as well.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 4:34:00 AM UTC-5, Brecht Machiels wrote: > On 2017-04-11 14:56:33 +, Steve D'Aprano said: > > On Tue, 11 Apr 2017 07:56 pm, Brecht Machiels wrote: > [...] > However, we don't really know how Python's performance is > affecting its popularity. It would be good to have some > kind of information about user's motivations for choosing > Python or moving away from it. Have any polls been > organized in the past to try to find out? Not to my knowledge. But a very good idea. Because when Python fails to meet the demands of most users, those users will not bother to complain, they will simply move quietly to another langauge that _does_ meet their needs. So, it would behoove us to anticipate the tends before they become _the tend_. Judging Python's popularity by gawking at Python's position in an internet popularity list is like judging if you have black paint on the bottom side of your shoe by walking across white carpet and looking behind you for footprints -- by the time you realize you're leaving tracks, it's too late! > I realize that work on CPython (and thus the language > itself) is done by unpaid volunteers. But I assume that > they also want the language to thrive, and therefore want > to cater to the wishes of the userbase at some level. I would hope so, for, without the large userbase, what would be the point? > So perhaps the conclusion to this discussion is that we > should first try to find out whether performance is an > issue for a large part of the community (or possible > newcomers). +1 -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 15/04/2017 00:40, Rick Johnson wrote: On Wednesday, April 12, 2017 at 1:48:57 AM UTC-5, Steven D'Aprano wrote: answer = 0 for i in range(10): answer += 1 instead of answer = 10 So... how exactly does the compiler prohibit stupid code? Easy. The same way _any_ optimizer optimizes code, by analysing the code! In Python, the optimising would need to be done every time you run from source. You can't spend too long at it. (I don't know how much it relies on caching pre-compiled byte-code.) In this example, the integer named 'answer` is simply being incremented in a superfluous loop. Nothing here is "dynamic". In Python, nearly everything is. For example: * `range(10)` will always produce a list of the _same_ 10 integers. You don't know if 'range' is still a range. If this has been executed first, then the answer will be 20: oldrange=range def double(x): return oldrange(x*2) range=double * `for i in range(10)` will aways iterate over the _same_ 10 integers And range could be set to something else each time the loop is executed. * `answer += 1` will always increment the "current integer value" by 1 I don't know if it's possible to override __iadd__ for integers so that +=1 does something different here. But if there is any code between answer=0 and the start of the loop, then there is the possibility that answer could be something different. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
Gregory Ewing wrote, on Thursday, April 13, 2017 12:17 AM > > Deborah Swanson wrote: > > But I think you got it right in your last sentence below. defaultdict > > copied them because they were immutable, > > No, definitely not. A defaultdict will never take it upon > itself to copy an object you give it, either as a key or a value. > > The copying, if any, must have occurred somewhere else, in > code that you didn't show us. > > Can you show us the actual code you used to attempt to > update the namedtuples? > > -- > Greg I think you've heard my sob story of how the actual code was lost (PyCharm ate it). I've made some attempts to recover that code, but honestly, at this point Peter Otten has showed my enough examples of getattr() that work with namedtuples with variable names, that I'd rather just accept that probably some tranformation of the structure I did caused the copying of values only. I remember looking at it in the debugger, but that code was convoluted and I don't think it's worth teasing out exactly what went wrong. (or figuring out how I did it in the first place) Deborah -- https://mail.python.org/mailman/listinfo/python-list
Re: Namedtuples: some unexpected inconveniences
Peter Otten wrote: PS: Personally I would probably take the opposite direction and use dicts throughout... Yes, my suggestion to used namedtuples in the first place was based on the assumption that you would mostly be referring to fields using fixed names. If that's not true, then using namedtuples (or a mutable equivalent) might just be making things harder. If the names are sometimes fixed and sometimes not, then you have a tradeoff to make. In the code you posted most recently, the only fixed field reference seems to be row.title, and that only appears once. So as long as that's all you want to do with the rows, storing them as dicts would appear to be a clear winner. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
RE: Namedtuples: some unexpected inconveniences
Roel Schroeven wrote, on Thursday, April 13, 2017 5:26 PM
>
> Gregory Ewing schreef op 13/04/2017 9:34:
> > Deborah Swanson wrote:
> >> Peter Otten wrote, on Wednesday, April 12, 2017 1:45 PM
> >>
> >>> Personally I would immediately discard the header row
> once and for
> >>> all, not again and again on every operation.
> >> Well, perhaps, but I need the header row to stay in place to write
> >> the list to a csv when I'm done
> >
> > That's no problem, just write the header row separately.
> >
> > Do this at the beginning:
> >
> >header = [Record._make(fieldnames)]
> >records = [Record._make(row) for row in rows]
> >
> > and then to write out the file:
> >
> >writer = csv.writer(outputfile)
> >writer.writerow(header)
> >writer.writerows(records)
>
> I don't even think there's any need to store the field names anywhere
> else than in fieldnames. So unless I'm missing something,
> just do this
> at the beginning:
>
> fieldnames = next(rows)
> Record = namedtuple("Record", fieldnames)
> records = [Record._make(row) for row in rows]
>
> and this at the end:
>
> writer = csv.writer(outputfile)
> writer.writerow(fieldnames) # or writer.writerow(Record._fields)
> writer.writerows(records)
>
>
> --
> The saddest aspect of life right now is that science gathers
> knowledge faster than society gathers wisdom.
>-- Isaac Asimov
>
> Roel Schroeven
This is essentially what Peter Otten most recently recommended. I know
you got there first, but it is better to only get the header row to name
the fields as you and Greg Ewing suggested, and then use just the
records in processing the field data, using the field names only for the
output.
Thanks,
Deborah
--
https://mail.python.org/mailman/listinfo/python-list
Re: Goto Considered Harmful [was Re: Python and the need for speed]
Dennis Lee Bieber wrote: And since the original COBOL standard numeric format was BCD, PIC not only defined output layout, but also internal storage needed by numerics and string data types. Unless you said USAGE IS COMPUTATIONAL, which left the compiler free to pick a more efficient storage format internally. BTW, one thing I think COBOL did right was not limiting you to some arbitrary compiler or language defined maximum size for numbers. You could declare something as PIC 9(100) and do arithmetic on 100-digit numbers if you wanted. (At least potentially -- not sure if all compilers would really support something that big.) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 4:57:10 AM UTC-5, bart...@gmail.com wrote: > On Wednesday, 12 April 2017 07:48:57 UTC+1, Steven D'Aprano wrote: > > On Tue, 11 Apr 2017 21:10:56 -0700, Rick Johnson wrote: > > > > > > high level languages like Python should make it > > > difficult, if not impossible, to write sub- optimal code > > > (at least in the blatantly obvious cases). > > > > Here's another example: > > > > answer = 0 > > for i in range(10): > > answer += 1 > > > > instead of > > > > answer = 10 > > > > So... how exactly does the compiler prohibit stupid code? > > Actually, an optimising C compiler (not one of mine!) > probably could reduce that to answer=10. And eliminate even > that if 'answer' was never used. > > But that's not really the issue here. Assume that such a > loop /is/ doing something more useful. The problems with > Python (last time I looked anyway) were these: > > (1) If answer was a global variable, then it needs a symbol > table lookup to find out what it is. That would not happen > in a static language, or a less dynamic one, as you already > have the address. Indeed. I have argued for syntax that will resolve variable scope many times. but apparently, the py-devs believe we only deserve type declarations that do _nothing_ to speed up code execution (aka: type-hints), instead of type declarations that could actually speed up the code. Go figure! I'm not a fan of forced static typing, but i am a fan of optional static typing. The global keyword is one idiosyncrasy of Python that causes a lot of confusion , especially to noobs, but also from a standpoint of "general intuitiviness" for old hats. This keyword does not define "true globals" (aka: a symbol that can be accessed or reassigned from anywhere), no, but rather, a symbol that is global only with respect to the "current module scope" (aka: module level variable). You can only get "true globals" in Python by going rogue and injecting symbols into sys.modules like some mad scientist -- which is not officially supported. Go figure! ;-) > And this [global] lookup happens for every loop iteration. I sure hope not. I have not taken the time to inspect the inner workings of Python, but if the lookup happens every time, that would be an awfully inefficient design. > (2) There is also 'range', which could have been redefined > to mean something else, so /that/ needs a lookup. The byte- > code compiler can't just assume this loop is to be executed > 10 times. Yep, and more evidence that Python has taken dynamics to such a fundamentalist extreme, that even ISIS is jealous! # START INTERACTIVE SESSION (Py2.x) ## >>> def range(*args): ... return "Death to static infidels!" >>> for i in range(10): ... print i D e a t h t o s t a t i c i n f i d e l s ! ## END INTERACTIVE SESSION ## > (3) This was fixed long ago but at one time, even when > 'range' had been established to be a range, it involved > constructing a list of items (10 here, but it could be a > million), and then iterating over the list. First, it was "range", then "xrange", and now, "range" again. Third time's a charm i suppose. The last iteration of the range function design removed the underlying inefficiency, however, it created a semantical nightmare (more on this below). > This might seem crazy, but it might have been exceptable > for a script language at one time. Not for a general > purpose one however. Yeah, this is one of those screw-ups that even GvR's famous time machine could not fix. Must have been in the shop for a tune-up... > (4) Python's integer types being immutable, the += > operation means evaluating the result, then creating a new > integer object and binding 'a' to that new value. (Correct > me if I'm wrong.) > > These are all things the language could point a finger at > before blaming the user for writing inefficient code. > > The problem is also the language encouraging people to use > high-level but inefficient methods, as the emphasis is on > productivity and readability** rather than performance. In > fact most users probably have no idea on what is efficient > and what isn't. And that's the point i was trying to make earlier. Most Python programmers are not CS majors, they are just "technical folk" who need to write a logic program to do this or that task. We shouldn't expect them to know about things like memory management, neither should we expect/burden them to know that calling range(100), just to do one million iterations, is inefficient. Instead, we should provide them with the proper "efficient tools" to get the job done. So instead of: for x in range(100): # Legacy python was slow here! doSomething(x) It should have _always_ been... 100.times: doSomething() or, if you need a counter: 100.times as x: doSomething(x) Or som
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote: > On Wednesday, 12 April 2017 12:56:32 UTC+1, Jussi Piitulainen wrote: > > bartc writes: > > > > > > These are straightforward language enhancements. > > > > FYI, the question is not how to optimize the code but how > > to prevent the programmer from writing stupid code in the > > first place. Someone suggested that a language should do > > that. > The 'stupid code' thing is a red herring. I assume the > code people write is there for a reason. Yeah, but you have to admit, sometimes there is no obvious good reason -- as Steven's example code has proven. ;-) Although, i'm convinved that any one of us could find examples like that in our own repos. And if you cannot find code that makes you feel embarassed, then you're not evolving anymore. > But the language can also play a part in not allowing > certain things to be expressed naturally. Exactly! > So the for-loop in the example has to have a control- > variable even if it's not referenced. Yes, one of the design flaws of Python's "for loop" is that the control variable is injected whether you need it or not (Nightmarish images of petulant children being force-fed castor oil comes to mind...). In fact, in Python, there are only two loop forms avialable -- the "for" and the "while". -- however, many other languages have relized that these two forms are insufficent to cover the various types of loops that are needed. At a minimum, every language should offer the following four loop-forms (using Python semantics): while CONDITION: doSomething() for VALUE in COLLECTION: doSomething(value) loop(N): doSomething() loop(N) as i: doSomething(i) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Thursday, April 13, 2017 at 1:32:29 AM UTC-5, Gregory Ewing wrote: > Jussi Piitulainen wrote: > > Traceback (most recent call last): > > File "/dev/fd/63", line 37, in > > SanityClauseException: code is blatantly sub-optimal > > > > As far as I know, no language does that. Because reasons? > > Because the problem of making a compiler do that is > probably AI-complete! What? GvR's time machine only goes backwards? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Thursday, April 13, 2017 at 1:32:28 AM UTC-5, Steven D'Aprano wrote: > On Wed, 12 Apr 2017 14:38:52 +0100, Ben Bacarisse wrote: > > Steve D'Aprano writes: > >> On Wed, 12 Apr 2017 03:39 am, Paul Rubin wrote: > > [...] Indeed, and this is a very common phenomenon: > features which "ordinary" programmers imagine are "trendy" > and new end up having their roots in some functional > language dating back to the 1980s, or 70s, or in extreme > cases the 1950s and Lisp. Be advised that functional programmers are a fanatical bunch, and saying something like that could be akin to blowing a "functional purist dogwhistle". If this thread becomes overrun with functional trolls, i'm blaming you. ;-) -- https://mail.python.org/mailman/listinfo/python-list
Re: Swiss Ephemeris
On Thursday, April 13, 2017 at 7:39:38 AM UTC-5, Rustom Mody wrote: > Quote from Peter Landin, one of the precursors of modern > functional programming: Most papers in computer science > describe how their author learned what someone else already > knew [And this dates from 60s/70s] Applies beyond just > CS… There are very few truely orignal ideas. Most of what we say, do and know are derivative works. For example; the kernel of Python was formed in the mind of Guido from his observations of another language called "ABC" -- he saw many good ideas in ABC, but also, many design flaws (kinda coming full circle now, eh?). The key to success in evolution is to propagate the good traits whilst discarding the bad ones. And like most other languages, Python is simply an evolution of many likable traits with a few new features sprinkled on top to "sweeten the pot". Much is the same in any field of study. There are countless examples of theorists and experimentors who knew the answers long before the "named discoverer" was given credit for the supposed discovery. In the reams of the historical record, many times, good communication skills and friends in academic/political circles can be the difference between a Nobel laureate and a nobody. -- https://mail.python.org/mailman/listinfo/python-list
