Re: Logging from different python scripts to different output files
James McMahon wrote: [Please keep the discussion on the list] > Thank you Peter. Is it necessary to employ a close() on the handlers and a > shutdown() on the loggers themselves? -Jim Not unless you run into problems with the default mechanism -- logging.shutdown() is scheduled via atexit.register() and invoked when the application terminates normally. -- https://mail.python.org/mailman/listinfo/python-list
K&L graph partitioning code offer
Hi I am planning to tweak the Kernighan Lin algorithm a bit use coercing of certain vertices .I was wondering if u would be kind enough to share the python code with me so that i can include my idea in it. -- https://mail.python.org/mailman/listinfo/python-list
K&L graph partitioning code offer
Hi I am planning to tweak the Kernighan Lin algorithm a bit use coercing of certain vertices .I was wondering if u would be kind enough to share the python code with me so that i can include my idea in it. -- https://mail.python.org/mailman/listinfo/python-list
Re: K&L graph partitioning code offer
On Wed, 29 Mar 2017 07:29 pm, arpitamishra...@gmail.com wrote: > Hi > I am planning to tweak the Kernighan Lin algorithm a bit use coercing of > certain vertices .I was wondering if u would be kind enough to share the > python code with me so that i can include my idea in it. https://www.google.com.au/search?q=Kernighan+Lin+algorithm+python https://duckduckgo.com/?q=python+Kernighan+Lin+algorithm -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
logging.LogRecord asctime aattribute - easily override somehow?
Short of going into some Formatter creation exercise (which seems like overkill to me, and somewhat obscure), is there some way to easily change the format of the logging.LogRecord's asctime attribute? Thx, Skip -- https://mail.python.org/mailman/listinfo/python-list
Python under PowerShell adds characters
I wrote a Python script, which executed as intended on Linux and from cmd.exe on Windows. Then, I ran it from the PowerShell command line, all print statements added ^@ after every character. Have you seen this? Do you know how to prevent this? Thank you, Jay -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
lyngw...@gmail.com writes: > I wrote a Python script, which executed as intended on Linux and from > cmd.exe on Windows. Then, I ran it from the PowerShell command line, > all print statements added ^@ after every character. > > Have you seen this? Do you know how to prevent this? Script is printing UTF-16 or something, viewer is expecting ASCII or some eight bit code and making null bytes visible as ^@. Python gets some default encoding from its environment. There are ways to set the default, and ways to override the default in the script. For example, you can specify an encoding when you open a file. -- https://mail.python.org/mailman/listinfo/python-list
Re: logging.LogRecord asctime aattribute - easily override somehow?
Skip Montanaro wrote:
> Short of going into some Formatter creation exercise (which seems like
> overkill to me, and somewhat obscure), is there some way to easily
> change the format of the logging.LogRecord's asctime attribute?
>
> Thx,
>
> Skip
Like
>>> import logging
>>> logging.basicConfig(format="%(asctime)s" + logging.BASIC_FORMAT,
datefmt="***%A***")
>>> logging.warn("foo")
***Wednesday***WARNING:root:foo
?
--
https://mail.python.org/mailman/listinfo/python-list
Re: logging.LogRecord asctime aattribute - easily override somehow?
>>> import logging
>>> logging.basicConfig(format="%(asctime)s" + logging.BASIC_FORMAT,
>>> datefmt="***%A***")
>>> logging.warn("foo")
***Wednesday***WARNING:root:foo
Thanks, Peter. I suppose a bit more detail of my environment would
have helped. I'm actually using the logging package indirectly via
Flask. After a quick skim of Flask's code, it didn't look like the
Flask author considered that users might want to do their own thing
with logging. I think I could override the Flask.logger property and
chuck in something like your code. I'll just suffer for the time being
with my "two word" timestamps. Whoever thought you'd want to break up
timestamps into two words by default, and hard-code a comma as the
separator between seconds and milliseconds?
Just for the record, for my own standalone code I never use the
logging module. I just have a simple Logger class which does all I
need. The logging module (and log4j) have always seemed to me to be an
overly general solution in search of a problem.
Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: K&L graph partitioning code offer
On Wednesday, March 29, 2017 at 1:23:48 AM UTC-7, arpitam...@gmail.com wrote: > Hi > I am planning to tweak the Kernighan Lin algorithm a bit use coercing of > certain vertices .I was wondering if u would be kind enough to share the > python code with me so that i can include my idea in it. Good luck getting the code you want from Dean Hall, his post is from 1999! (Is 18 years a record for thread necromancy?) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Wed, Mar 29, 2017 at 4:06 PM, wrote: > I wrote a Python script, which executed as intended on Linux and > from cmd.exe on Windows. Then, I ran it from the PowerShell >command line, all print statements added ^@ after every character. ISE is the only command-line environment that's specific to PowerShell. Surely you wouldn't be running Python scripts in ISE. If powershell.exe is run normally, then it's a console application. python.exe would inherit the console handle, and that's the end of its interaction with PowerShell. At most PowerShell (or any process that's attached to the console) may have set the console to a different output codepage via SetConsoleOutputCP or set the mode on the screen buffer via SetConsoleMode. As far as I know, neither of these can make the console print "^@" as a representation of NUL. It only shows "^@" in the input buffer when you type Ctrl+2, which is what most terminals do. For example: >>> s = sys.stdin.read(6) spam^@ >>> s 'spam\x00\n' >>> print(s) spam -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Wednesday, March 29, 2017 at 10:28:58 AM UTC-7, eryk sun wrote: > On Wed, Mar 29, 2017 at 4:06 PM, wrote: > > I wrote a Python script, which executed as intended on Linux and > > from cmd.exe on Windows. Then, I ran it from the PowerShell > >command line, all print statements added ^@ after every character. > > ISE is the only command-line environment that's specific to > PowerShell. Surely you wouldn't be running Python scripts in ISE. > > If powershell.exe is run normally, then it's a console application. > python.exe would inherit the console handle, and that's the end of its > interaction with PowerShell. At most PowerShell (or any process that's > attached to the console) may have set the console to a different > output codepage via SetConsoleOutputCP or set the mode on the screen > buffer via SetConsoleMode. As far as I know, neither of these can make > the console print "^@" as a representation of NUL. It only shows "^@" > in the input buffer when you type Ctrl+2, which is what most terminals > do. For example: > > >>> s = sys.stdin.read(6) > spam^@ > >>> s > 'spam\x00\n' > >>> print(s) > spam I'm not using ISE. I'm using a pre-edited script, and running it with the python command. Consider the following simple script named hello.py (Python 2.7): print "Hello" If I enter: python hello.py > out.txt from cmd.exe I get a 6-character file (characters plus new-line). from PowerShell I get an extract ^@ character after every character j -- https://mail.python.org/mailman/listinfo/python-list
Re: K&L graph partitioning code offer
On 03/29/2017 11:17 AM, jlada...@itu.edu wrote: > On Wednesday, March 29, 2017 at 1:23:48 AM UTC-7, arpitam...@gmail.com wrote: >> Hi >> I am planning to tweak the Kernighan Lin algorithm a bit use coercing of >> certain vertices .I was wondering if u would be kind enough to share the >> python code with me so that i can include my idea in it. > > Good luck getting the code you want from Dean Hall, his post is from 1999! > > (Is 18 years a record for thread necromancy?) I don't see any reply to a post from 1998 here... What post are you talking about? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 4:42 AM, Jay Braun wrote: > Consider the following simple script named hello.py (Python 2.7): > > print "Hello" > > If I enter: > > python hello.py > out.txt > > from cmd.exe I get a 6-character file (characters plus new-line). > > from PowerShell I get an extract ^@ character after every character Sounds like cmd and PS are setting the output encodings differently. Does the same thing occur with Python 3.6? Try adding this to your script: import sys print(sys.stdout.encoding) and run it in the same two environments (no redirection needed). Do they give you the same thing? My suspicion is that you're running cmd.exe in the default Windows console, and PowerShell in some different console. But it's hard to be sure. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Wed, Mar 29, 2017 at 5:42 PM, Jay Braun wrote:
>
> I'm not using ISE. I'm using a pre-edited script, and running it with the
> python command.
>
> Consider the following simple script named hello.py (Python 2.7):
>
> print "Hello"
>
> If I enter:
> python hello.py > out.txt
>
> from cmd.exe I get a 6-character file (characters plus new-line).
> from PowerShell I get an extract ^@ character after every character
You didn't say you were redirecting the output to a file. That's a
completely different story for PowerShell -- and far more frustrating.
cmd.exe implements redirecting a program's output to a file by
temporarily changing its own StandardOutput to the file; spawing the
process, which inherits the StandardOutput handle; and then changing
back to its original StandardOutput (typically a console screen
buffer). The program can write whatever it wants to the file, and cmd
isn't involved in any way.
PowerShell is far more invasive. Instead of giving the child process a
handle for the file, it gives it a handle for a *pipe*. PowerShell
reads from the pipe, and like an annoying busybody that no asked for,
decodes the output as text, processes it (e.g. replacing newlines),
and writes the processed data to the file. For example:
PS C:\Temp> $script = "import sys; sys.stdout.buffer.write(b'\n')"
PS C:\Temp> python -c $script > test.txt
PS C:\Temp> python -c "print(open('test.txt', 'rb').read())"
b'\xff\xfe\r\x00\n\x00'
I wrote a single byte, b'\n', but PowerShell decoded it, replaced "\n"
with "\r\n", and wrote it as UTF-16 with a BOM.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 5:19 AM, eryk sun wrote:
> PowerShell is far more invasive. Instead of giving the child process a
> handle for the file, it gives it a handle for a *pipe*. PowerShell
> reads from the pipe, and like an annoying busybody that no asked for,
> decodes the output as text, processes it (e.g. replacing newlines),
> and writes the processed data to the file. For example:
>
> PS C:\Temp> $script = "import sys; sys.stdout.buffer.write(b'\n')"
> PS C:\Temp> python -c $script > test.txt
> PS C:\Temp> python -c "print(open('test.txt', 'rb').read())"
> b'\xff\xfe\r\x00\n\x00'
>
> I wrote a single byte, b'\n', but PowerShell decoded it, replaced "\n"
> with "\r\n", and wrote it as UTF-16 with a BOM.
Lolwut?
So PS can't handle binary redirection whatsoever. Fascinating.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On 03/29/2017 11:23 AM, Chris Angelico wrote:
On Thu, Mar 30, 2017 at 5:19 AM, eryk sun wrote:
PowerShell is far more invasive. Instead of giving the child process a
handle for the file, it gives it a handle for a *pipe*. PowerShell
reads from the pipe, and like an annoying busybody that no asked for,
decodes the output as text, processes it (e.g. replacing newlines),
and writes the processed data to the file. For example:
PS C:\Temp> $script = "import sys; sys.stdout.buffer.write(b'\n')"
PS C:\Temp> python -c $script > test.txt
PS C:\Temp> python -c "print(open('test.txt', 'rb').read())"
b'\xff\xfe\r\x00\n\x00'
I wrote a single byte, b'\n', but PowerShell decoded it, replaced "\n"
with "\r\n", and wrote it as UTF-16 with a BOM.
Lolwut?
So PS can't handle binary redirection whatsoever. Fascinating.
ChrisA
Engineer 1: Man, that old DOS shell we keep emulating is just getting
older and clunkier.
Engineer 2: I know, we should rewrite it. You know, whole new thing,
really modernize it.
E1: So, like, bring in bash like everyone else?
E2: No, better. How about something that integrates with no preexisting
workflow in the world.
E1: Wait, but what commands would it use?
E2: New ones.
E1: But, then how would it behave?
E2: Totally new. Never before seen. New commands, new semantics, whole
9 yards. If anyone's ever used it, I don't want to.
E1: I love this plan.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Wednesday, March 29, 2017 at 11:20:45 AM UTC-7, eryk sun wrote:
> On Wed, Mar 29, 2017 at 5:42 PM, Jay Braun wrote:
> >
> > I'm not using ISE. I'm using a pre-edited script, and running it with the
> > python command.
> >
> > Consider the following simple script named hello.py (Python 2.7):
> >
> > print "Hello"
> >
> > If I enter:
> > python hello.py > out.txt
> >
> > from cmd.exe I get a 6-character file (characters plus new-line).
> > from PowerShell I get an extract ^@ character after every character
>
> You didn't say you were redirecting the output to a file. That's a
> completely different story for PowerShell -- and far more frustrating.
>
> cmd.exe implements redirecting a program's output to a file by
> temporarily changing its own StandardOutput to the file; spawing the
> process, which inherits the StandardOutput handle; and then changing
> back to its original StandardOutput (typically a console screen
> buffer). The program can write whatever it wants to the file, and cmd
> isn't involved in any way.
>
> PowerShell is far more invasive. Instead of giving the child process a
> handle for the file, it gives it a handle for a *pipe*. PowerShell
> reads from the pipe, and like an annoying busybody that no asked for,
> decodes the output as text, processes it (e.g. replacing newlines),
> and writes the processed data to the file. For example:
>
> PS C:\Temp> $script = "import sys; sys.stdout.buffer.write(b'\n')"
> PS C:\Temp> python -c $script > test.txt
> PS C:\Temp> python -c "print(open('test.txt', 'rb').read())"
> b'\xff\xfe\r\x00\n\x00'
>
> I wrote a single byte, b'\n', but PowerShell decoded it, replaced "\n"
> with "\r\n", and wrote it as UTF-16 with a BOM.
You are correct. Sorry I omitted that in my first post. Thank you for your
help.
j
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 5:29 AM, Rob Gaddi wrote: > Engineer 1: Man, that old DOS shell we keep emulating is just getting older > and clunkier. > > Engineer 2: I know, we should rewrite it. You know, whole new thing, really > modernize it. > > E1: So, like, bring in bash like everyone else? > > E2: No, better. How about something that integrates with no preexisting > workflow in the world. > > E1: Wait, but what commands would it use? > > E2: New ones. My understanding of PowerShell is more like this: E1: Man, batch files are so clunky. It's a ridiculous hodge-podge. E2: Yeah, every shell ever written is clunky. Let's do our own thing and make it more like a scripting language. E1: Cool! Only, we won't use an existing language, we'll make our own, because it'll be better. E2: I love this plan. We've had discussions on this list about using Python as a job control shell, and the usual response is: Python sucks as a command executor. It's just not designed for that. The clunkiness of bash is precisely BECAUSE it's designed to be convenient and comfortable for a sysadmin. All those weird splitting and escaping rules are because (a) the easiest way to do piping, command sequencing, etc is with symbols, (b) you can't stop people from using those symbols in file names or arguments, and (c) it's a pain to have to quote every single string. AIUI PowerShell is somewhat like VBScript, only it isn't quite that either. But it's definitely more like a scripting language than a shell language. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
eryk sun : > PowerShell is far more invasive. Instead of giving the child process a > handle for the file, it gives it a handle for a *pipe*. PowerShell > reads from the pipe, and like an annoying busybody that no asked for, > decodes the output as text, You mean, a bit like Python3 does? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 6:13 AM, Marko Rauhamaa wrote: > eryk sun : >> PowerShell is far more invasive. Instead of giving the child process a >> handle for the file, it gives it a handle for a *pipe*. PowerShell >> reads from the pipe, and like an annoying busybody that no asked for, >> decodes the output as text, > > You mean, a bit like Python3 does? If you open a file in Python 3, you can choose whether to open it as text or binary. When you print text to stdout, well, it's text, so of course it has to be encoded appropriately; if you're doing something unusual (like a CGI script creating an image file), you can override the default and change stdout to be binary. But normally, the standard streams are connected ultimately to a human, so they're text. The problem is that PS is decoding and then re-encoding instead of simply telling the process what encoding to use. That's just wrong. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: logging.LogRecord asctime aattribute - easily override somehow?
Skip Montanaro wrote:
import logging
logging.basicConfig(format="%(asctime)s" + logging.BASIC_FORMAT,
datefmt="***%A***") logging.warn("foo")
> ***Wednesday***WARNING:root:foo
>
> Thanks, Peter. I suppose a bit more detail of my environment would
> have helped. I'm actually using the logging package indirectly via
> Flask. After a quick skim of Flask's code, it didn't look like the
> Flask author considered that users might want to do their own thing
> with logging. I think I could override the Flask.logger property and
> chuck in something like your code. I'll just suffer for the time being
> with my "two word" timestamps.
$ cat hello.py
from flask import Flask
import functools
import flask.logging
flask.logging.Formatter = functools.partial(
flask.logging.Formatter, datefmt="%A"
)
app = Flask(__name__)
@app.route('/')
def hello_world():
app.logger.critical("test")
return 'Hello, World!'
$ FLASK_APP=hello.py python -m flask run &
[1] 6028
$ * Serving Flask app "hello"
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
$ curl http://127.0.0.1:5000/
[Wednesday] CRITICAL in hello: test
127.0.0.1 - - [29/Mar/2017 21:26:38] "GET / HTTP/1.1" 200 -
Hello, World!$
> Whoever thought you'd want to break up
> timestamps into two words by default, and hard-code a comma as the
> separator between seconds and milliseconds?
In Germany the comma is the decimal separator, so this doesn't look like two
words to my eye. However, the culprit is
https://en.wikipedia.org/wiki/ISO_8601
"""
A decimal mark, either a comma or a dot (without any preference as stated in
resolution 10 of the 22nd General Conference CGPM in 2003,[16] but with a
preference for a comma according to ISO 8601:2004)[17] is used as a
separator between the time element and its fraction
"""
>
> Just for the record, for my own standalone code I never use the
> logging module. I just have a simple Logger class which does all I
> need. The logging module (and log4j) have always seemed to me to be an
> overly general solution in search of a problem.
>
> Skip
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
Chris Angelico : > But normally, the standard streams are connected ultimately to a > human, so they're text. Huh? The standard input is the workload and the standard output is the result of the computation. Arguably, the standard error stream is meant for humans. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Wed, Mar 29, 2017 at 7:13 PM, Marko Rauhamaa wrote: > eryk sun : >> PowerShell is far more invasive. Instead of giving the child process a >> handle for the file, it gives it a handle for a *pipe*. PowerShell >> reads from the pipe, and like an annoying busybody that no asked for, >> decodes the output as text, > > You mean, a bit like Python3 does? The closest to what we're talking about here would be using subprocess.Popen and friends to create pipelines and redirect output to files. Opening a file defaults to text mode in Python for how Python access the file, but if you pass a file descriptor as Popen's stdout argument, Python isn't acting as a middle man. The child process writes directly to the file. PowerShell makes itself a middle man in the cases of file redirection and pipelines. It does this to enable all of the capabilities of its object pipeline. That's fine. But, IMO, there should be a simple way to get the plain-old redirection and piping in which the shell is not a middle man. The simplest way I know to do that in PowerShell is to run the command line via `cmd /c`. But I'm no PowerShell expert. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 28 Mar 2017, at 14:21, INADA Naoki wrote: On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann wrote: I suppose smaller and faster benchmark is better to others looking for it. I already stopped the azure instance. [...] There are no maxrss difference in "smaller existing examples"? [...] I want to investigate RAM usage, without any swapping. Running further trials indicate that the problem actually is related to swapping. If I reduce the model size in the benchmark slightly so that everything fits into the main memory, the problem disappears. Only when the memory usage exceeds the 32GB that I have, Python 3.6 will acquire way more memory (from the swap) than Python 3.5. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: logging.LogRecord asctime aattribute - easily override somehow?
Skip> Whoever thought you'd want to break up Skip> timestamps into two words by default, and hard-code a comma as the Skip> separator between seconds and milliseconds? Peter> In Germany the comma is the decimal separator, so this doesn't look like two Peter> words to my eye. The "two words" reference I made was to the space separating the date and time. (I'd prefer a "T" separating date and time, as downstream log processing tools can be slightly simpler, since they don't have to collapse two fields into one.) The comma separator in the seconds field is fine if that's appropriate for your environment, though it appears to be hard-coded, not locale-specific: default_time_format = '%Y-%m-%d %H:%M:%S' default_msec_format = '%s,%03d' If I parse such times in a straightforward way in my locale (C or en_US.utf8), extracting the seconds as a floating point number is, once again, more complex than it ought to be, as a comma is not the proper decimal point in my locale(s). Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: K&L graph partitioning code offer
On Wednesday, March 29, 2017 at 10:46:56 AM UTC-7, Michael Torrie wrote:
> On 03/29/2017 11:17 AM, j...@itu.edu wrote:
> > On Wednesday, March 29, 2017 at 1:23:48 AM UTC-7, arpitam...@gmail.com
> > wrote:
> >> Hi
> >> I am planning to tweak the Kernighan Lin algorithm a bit use coercing of
> >> certain vertices .I was wondering if u would be kind enough to share the
> >> python code with me so that i can include my idea in it.
> >
> > Good luck getting the code you want from Dean Hall, his post is from 1999!
> >
> > (Is 18 years a record for thread necromancy?)
>
> I don't see any reply to a post from 1998 here... What post are you
> talking about?
The Google Groups interface shows that the first post in this thread is as
follows:
> From: dwh...@ksu.edu (Dean Hall)
> Subject: K&L graph partitioning code offer
> Date: 1999/03/01
> Message-ID: <7bftca$l...@abc.ksu.ksu.edu>#1/1
> X-Deja-AN: 450257007
> Summary: K&L two-way partitioning algorithm coded in Python
> X-Complaints-To: ab...@ksu.edu
> X-Trace: cnn.ksu.ksu.edu 920352972 10308 129.130.12.3 (2 Mar 1999 05:36:12
> GMT)
> Organization: Kansas State University
> Keywords: graph theory partition Kernighan Lin K&L
> NNTP-Posting-Date: 2 Mar 1999 05:36:12 GMT
> Newsgroups: comp.lang.python
>
> Hello,
>
> If anyone is interested in the Kernighan & Lin partitioning algorithm,
> I've coded it in Python.
> For those that don't know, the K&L algorithm finds a
> pair of partitions of a set that has the locally minimum
> number of edges between the two partitions (cutsize).
>
> I implemented a dumb Graph class (vertices and egdes),
> then a KLPartitioner class.
> Then, for fun ('cuz that's what Python is),
> I wrote a class that uses Tkinter to display the resulting graph.
>
> I don't read the newsgroups much,
> so email me if you wanna see the code.
>
>
> !!Dean
> dwh...@ksu.edu
--
https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Thu, 30 Mar 2017 07:19 am, Jan Gosmann wrote: > Running further trials indicate that the problem actually is related to > swapping. If I reduce the model size in the benchmark slightly so that > everything fits into the main memory, the problem disappears. Only when > the memory usage exceeds the 32GB that I have, Python 3.6 will acquire > way more memory (from the swap) than Python 3.5. If you can demonstrate this effect using simple example code without the external dependencies (using nothing but the standard library) and people can replicate it, I think it should be reported as a bug. If you are right, it does sound like a performance regression. Maybe there's a way to fix that. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, 30 Mar 2017 06:47 am, Marko Rauhamaa wrote: > Chris Angelico : >> But normally, the standard streams are connected ultimately to a >> human, so they're text. > > Huh? The standard input is the workload Which is usually typed by a human, read from a file containing human-readable text, a file-name intended to be read by a human, or some other data in human-readable form. > and the standard output is the result of the computation. Which is generally intended to be read by a human. > Arguably, the standard error stream is meant for humans. Just like the rest of the computation. Relatively few command-line computations are performed by machines, for machines, using machine-friendly human-hostile formats. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 11:14 AM, Steve D'Aprano wrote: > Just like the rest of the computation. Relatively few command-line > computations are performed by machines, for machines, using > machine-friendly human-hostile formats. And even most computations performed by machines for machines are done using human-friendly transmission formats. Trying to brain-storm actually human-hostile formats... let's see. There's extremely low-level protocols like IP, TCP, DNS, and X11. There are (de)compression tools like gzip, where byte size is the entire point of the utility. There's XML, of course, which isn't exactly machine-friendly, but is pretty human-hostile. And most crypto is done with byte streams rather than text. Beyond that, pretty much everything is text. Ever since I started git-managing my /etc directory, I've been seeing changes made by various programs being reflected there - in text files. Internet protocols are almost exclusively text. Unless you say otherwise, most Unix utilities consume and emit text, even when they're in "machine readable" mode - for example, a number of git commands support a "--porcelain" option that removes color codes and pretty formatting, and also uses a format that's guaranteed to be stable, but it's still text. Despite fundamentally working with bit operations and numbers, computers today are still heavily text-aligned. Might have something to do with the fact that they're programmed by humans... ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: If you can demonstrate this effect using simple example code without the external dependencies (using nothing but the standard library) and people can replicate it, I think it should be reported as a bug. I probably won't be able to demonstrate the effect with simple example code because I haven't the slightest idea what is causing it. Also, I'm not sure how much more time I want to invest in this. After all it is a problem that might never get noticed by any users of the software. Jan -- https://mail.python.org/mailman/listinfo/python-list
Re: newbie question re classes and self
On Tuesday, March 28, 2017 at 3:09:45 AM UTC-5, loial wrote: > Can I pass self(or all its variables) to a class? > Basically, how do I make all the variables defined in self > in the calling python script available to the python class > I want to call? Your question, as presented, is difficult to understand, and the phrase "variables defined in self", is quite absurd. I'm making a wild assumption here, but perhaps you want to "bulk-set" or "bulk-query" all the attributes of a class instance externally? If so, depending on how the object was defined, there are a few ways to achieve this. However, my advanced powers of perception tell me that you might be using Python in an incorrect manner, but i cannot be sure until you explain this problem in more detail. So if you can provide us a simple code example, or even psuedo code, that would be very helpful. -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote: > On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: > > > If you can demonstrate this effect using simple example > > code without the external dependencies (using nothing but > > the standard library) and people can replicate it, I think > > it should be reported as a bug. > > I probably won't be able to demonstrate the effect with > simple example code because I haven't the slightest idea > what is causing it. Also, I'm not sure how much more time I > want to invest in this. After all it is a problem that > might never get noticed by any users of the software. Really? How could your clients not notice 60 GB of memory usage unless they are running some kind of mad-dog insane top-of-the-line hardware? (Got Benjamins???) Of course, in the case they are not DARPA scientist supported by a viturally unlimited supply of tax dollars provided by the endless toils of American slave-bots, how could they ignore the thrashing? o_O -- https://mail.python.org/mailman/listinfo/python-list
Re: Program uses twice as much memory in Python 3.6 than in Python 3.5
On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnson wrote: > On Wednesday, March 29, 2017 at 8:17:01 PM UTC-5, Jan Gosmann wrote: >> On 29 Mar 2017, at 20:12, Steve D'Aprano wrote: >> >> > If you can demonstrate this effect using simple example >> > code without the external dependencies (using nothing but >> > the standard library) and people can replicate it, I think >> > it should be reported as a bug. >> >> I probably won't be able to demonstrate the effect with >> simple example code because I haven't the slightest idea >> what is causing it. Also, I'm not sure how much more time I >> want to invest in this. After all it is a problem that >> might never get noticed by any users of the software. > > Really? How could your clients not notice 60 GB of memory > usage unless they are running some kind of mad-dog insane > top-of-the-line hardware? (Got Benjamins???) Of course, in > the case they are not DARPA scientist supported by a > viturally unlimited supply of tax dollars provided by the > endless toils of American slave-bots, how could they ignore > the thrashing? o_O Did you read the project's README? This is a dramatic reduction from the normal memory usage of this kind of job. So, yes, they *are* going to have top-of-the-line hardware. If you're doing a job that normally would require 256GB of RAM, and instead of being able to run in 40GB, it needs 60GB, are you going to notice? You probably have 64GB or 128GB. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Text-mode apps (Was :Who are the "spacists"?)
On Sunday, March 26, 2017 at 1:21:18 PM UTC-5, Chris Angelico wrote: > On Mon, Mar 27, 2017 at 4:43 AM, Steve D'Aprano > wrote: > [...] So, for instance, Eryk Sun commented that my rounded > box example didn't render correctly in all fonts - but in > the future, a new version of those fonts could be released, > adding support for those characters. We *know* that the > code points I used are permanently and irrevocably > allocated to the purposes I used them for, so we can all be > confident that they'll be used correctly if at all. You place a lot of faith in the supposed "immutability of Unicode code points", but a lot people have placed a lot of faith in many past and current encoding systems, only to be bamboozled by the gatekeepers some time later. > > That's not quite right. Unicode includes 137000 or so > > Private Use Characters, which anyone can define for their > > own purposes, "by private agreement". There's an > > unofficial registry of such private use characters here: And so, although Unicode was created to solve the endless compatibility problems between multiple archaic encoding systems, the designers thought it necessary to add a "custom space" that will keep the incompatibilities on life support. Smart. *REALLY* smart. And people _wonder_ why Dylan was a disgruntled laureate... -- https://mail.python.org/mailman/listinfo/python-list
Re: Text-mode apps (Was :Who are the "spacists"?)
On Sunday, March 26, 2017 at 2:53:49 PM UTC-5, Chris Angelico wrote: > On Mon, Mar 27, 2017 at 6:25 AM, Mikhail V wrote: > > On 26 March 2017 at 20:10, Steve D'Aprano > > wrote: > >> On Mon, 27 Mar 2017 03:57 am, Mikhail V wrote: > I generally find that when people say that Unicode doesn't > solve their problems and they need to roll their own, it's > usually one of two possibilities: 1) "Their problems" are > all about simplicity. They don't want to have to deal with > all the complexities of real-world text, so they > arbitrarily restrict things. There are only so many hours in the day Chris. Not every progammer has the time to cater to every selfish desire of every potential client. You try to create the best product you can, but at the end of the process, there will always be someone (or a group of someones) who are unhappy with the result. > 2) Unicode _would_ solve their problems, they just don't > realize it. So if you're rolling your own text display > system, you should start by making a signed declaration: > > """ > I, the undersigned, acknowledge that my program is > intentionally excluding everyone who does not fit the > following requirements: [choose all applicable] > > [ ] Speaks English exclusively Of course, your comment presupposing that every programmer is fluent in every natural language. Which is not only impractical, it's impossible. > [ ] Uses no diacritical marks Why is it my responsibiliy to encode my text with pronuciation tutorials? Are we adults here or what? > [ ] Writes all text top-to-bottom, left-to-right Not my problem. Learn the King's English or go wait for extinction to arrive. > [ ] Uses only characters from the Basic Multilingual Plane > [ ] Uses only characters from this codepage: > [ ] Uses a monospaced font > [ ] Uses a proportionally-spaced font > [ ] Uses this font: _ > [ ] Uses a mouse > [ ] Uses a keyboard > [ ] Other: ___ > (use additional pages if required) What don't you add these: [ ] Has the ability to read and comprehend at a high school level. [ ] Has functioning visual receptors. [ ] Has a functioning brain. [ ] Is not currently in a vegetative state > Sure, there are good reasons to place restrictions on > people's text. But those restrictions are part of your > public API and UI. Acknowledge them. Own them. And justify > them. The only justifaction required is the bottom line. If your products generate profit, then you're doing something right. Besides, you don't need _everyone_ on the planet to buy your product to be a success. Unlike the business practices of Apple, we would be wise to leave plenty of room for others to enter the market. Competition is good for everyone. Monopolies are evil. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
Steve D'Aprano : > On Thu, 30 Mar 2017 06:47 am, Marko Rauhamaa wrote: >> Huh? The standard input is the workload > > Which is usually typed by a human, read from a file containing > human-readable text, a file-name intended to be read by a human, or > some other data in human-readable form. The main point is that it is supposed to be processed programmatically. It is somewhat rare that you type in the standard input. In particular, you want to be able to form useful pipelines from commands. Of course, in grand UNIX tradition, you strive to design your interchange formats to be also marginally applicable for human interaction (XML, base64 etc). >> and the standard output is the result of the computation. > Which is generally intended to be read by a human. That is more often the case. However, you want the format to be rigorous so it can be easily parsed programmatically. > Relatively few command-line computations are performed by machines, > for machines, using machine-friendly human-hostile formats. Didn't count them. Still, I'd expect not having to deal with Unicode decoding exceptions with arbitrary input. There recently was a related debate on the Guile mailing list. Like Python3, Guile2 is sensitive to illegal UTF-8 on the command line and in the standard streams. An emacs developer was urging Guile developers to follow emacs's example and support a superset of UTF-8 and Unicode where all byte strings can be bijectively mapped into text. Python3 partially does a similar thing, but only when dealing with pathnames. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Text-mode apps (Was :Who are the "spacists"?)
On Sunday, March 26, 2017 at 6:42:36 PM UTC-5, Mikhail V wrote: > And all text I currently read on my monitor are prerendered > bitmaps, refined manually for frequently used sizes, and > that is how it should be made. IOW there are much more > important aspects than the ability to scale a text line > from 600 px to 601 px wide, if you know what I mean. LOL! -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, 30 Mar 2017 07:29:48 +0300, Marko Rauhamaa wrote: [...] > I'd expect not having to deal with Unicode > decoding exceptions with arbitrary input. That's just silly. If you have *arbitrary* bytes, not all byte-sequences are valid Unicode, so you have to expect decoding exceptions, if you're processing text. Coming back to your complaint: Python 3 might default to automatically decoding stdin to Unicode, but you can choose to read stdin as bytes if you so wish. > There recently was a related debate on the Guile mailing list. Like > Python3, Guile2 is sensitive to illegal UTF-8 on the command line and in > the standard streams. An emacs developer was urging Guile developers to > follow emacs's example and support a superset of UTF-8 and Unicode where > all byte strings can be bijectively mapped into text. I'd like to read that. Got a link? -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Text-mode apps (Was :Who are the "spacists"?)
On Thu, Mar 30, 2017 at 3:21 PM, Rick Johnson wrote: > On Sunday, March 26, 2017 at 2:53:49 PM UTC-5, Chris Angelico wrote: >> On Mon, Mar 27, 2017 at 6:25 AM, Mikhail V wrote: >> > On 26 March 2017 at 20:10, Steve D'Aprano >> > wrote: >> >> On Mon, 27 Mar 2017 03:57 am, Mikhail V wrote: > >> I generally find that when people say that Unicode doesn't >> solve their problems and they need to roll their own, it's >> usually one of two possibilities: 1) "Their problems" are >> all about simplicity. They don't want to have to deal with >> all the complexities of real-world text, so they >> arbitrarily restrict things. > > There are only so many hours in the day Chris. Not every > progammer has the time to cater to every selfish desire of > every potential client. You try to create the best product > you can, but at the end of the process, there will always be > someone (or a group of someones) who are unhappy with the > result. Except that it doesn't actually take very much work to call on someone else's library, which is what you get when you use Unicode properly. (At least, assuming you're using a decent language like Python, which comes with Unicode libraries, and a decent GUI toolkit if you're going that way.) >> """ >> I, the undersigned, acknowledge that my program is >> intentionally excluding everyone who does not fit the >> following requirements: [choose all applicable] >> >> [ ] Speaks English exclusively > > Of course, your comment presupposing that every programmer > is fluent in every natural language. Which is not only > impractical, it's impossible. Nope. I can't speak Mandarin, but I can make absolutely sure that all my programs can accept Chinese characters. A friend of mine sent me an audio file with a name that included some Chinese, and I was able to handle it no problem. >> [ ] Uses no diacritical marks > > Why is it my responsibiliy to encode my text with > pronuciation tutorials? Are we adults here or what? > >> [ ] Writes all text top-to-bottom, left-to-right > > Not my problem. Learn the King's English or go wait for > extinction to arrive. And these two cement your parochialism thoroughly in everyone's minds. "Pronunciation tutorials", eh? Sure. Tell that to everyone who speaks Spanish, Turkish, Norwegian, German, or Vietnamese, all of which use diacritical marks to distinguish between letters. English is the weird language in that it uses letter pairs instead of adorned letters (eg "ch" and "sh" instead of "ç" and "ş"). Also, which king are you referring to, exactly? Whose English do you speak? > What don't you add these: > > [ ] Has the ability to read and comprehend at a high > school level. > [ ] Has functioning visual receptors. > [ ] Has a functioning brain. > [ ] Is not currently in a vegetative state Nah. If I did, I'd have to say "[ ] Is not trolling python-list" as well. >> Sure, there are good reasons to place restrictions on >> people's text. But those restrictions are part of your >> public API and UI. Acknowledge them. Own them. And justify >> them. > > The only justifaction required is the bottom line. If your > products generate profit, then you're doing something right. > Besides, you don't need _everyone_ on the planet to buy your > product to be a success. Unlike the business practices of > Apple, we would be wise to leave plenty of room for others > to enter the market. Competition is good for everyone. > Monopolies are evil. Riight. What's the "bottom line" for open source software? How do you measure whether your product is generating a profit or not? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
Steven D'Aprano : > On Thu, 30 Mar 2017 07:29:48 +0300, Marko Rauhamaa wrote: >> I'd expect not having to deal with Unicode decoding exceptions with >> arbitrary input. > > That's just silly. If you have *arbitrary* bytes, not all > byte-sequences are valid Unicode, so you have to expect decoding > exceptions, if you're processing text. The input is not in my control, and bailing out may not be an option: $ echo $'aa\n\xdd\naa' | grep aa aa aa $ echo $'\xdd' | python2 -c 'import sys; sys.stdin.read(1)' $ echo $'\xdd' | python3 -c 'import sys; sys.stdin.read(1)' Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.5/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 0: invalid continuation byte Note that "grep" is also locale-aware. >> There recently was a related debate on the Guile mailing list. Like >> Python3, Guile2 is sensitive to illegal UTF-8 on the command line and >> in the standard streams. An emacs developer was urging Guile >> developers to follow emacs's example and support a superset of UTF-8 >> and Unicode where all byte strings can be bijectively mapped into >> text. > > I'd like to read that. Got a link? http://lists.gnu.org/archive/html/guile-user/2017-02/msg00054.html> Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 4:43 PM, Marko Rauhamaa wrote: > The input is not in my control, and bailing out may not be an option: > >$ echo > aa\n\xdd\naa' | grep aa >aa >aa >$ echo \xdd' | python2 -c 'import sys; sys.stdin.read(1)' >$ echo \xdd' | python3 -c 'import sys; sys.stdin.read(1)' >Traceback (most recent call last): > File "", line 1, in > File "/usr/lib64/python3.5/codecs.py", line 321, in decode >(result, consumed) = self._buffer_decode(data, self.errors, final) >UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 0: > invalid continuation byte > > Note that "grep" is also locale-aware. So what exactly does byte value 0xDD mean in your stream? And if you say "it doesn't matter", then why are you assigning meaning to byte value 0x0A in your first example? Truly binary data doesn't give any meaning to 0x0A. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
Chris Angelico : > On Thu, Mar 30, 2017 at 4:43 PM, Marko Rauhamaa wrote: >> The input is not in my control, and bailing out may not be an option: >> >>$ echo >> aa\n\xdd\naa' | grep aa >>aa >>aa >>$ echo \xdd' | python2 -c 'import sys; sys.stdin.read(1)' >>$ echo \xdd' | python3 -c 'import sys; sys.stdin.read(1)' >>Traceback (most recent call last): >> File "", line 1, in >> File "/usr/lib64/python3.5/codecs.py", line 321, in decode >>(result, consumed) = self._buffer_decode(data, self.errors, final) >>UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 0: >> invalid continuation byte >> >> Note that "grep" is also locale-aware. > > So what exactly does byte value 0xDD mean in your stream? > > And if you say "it doesn't matter", then why are you assigning meaning > to byte value 0x0A in your first example? Truly binary data doesn't > give any meaning to 0x0A. What I'm saying is that every program must behave in a minimally controlled manner regardless of its inputs (which are not in its control). With UTF-8, it is dangerously easy to write programs that explode surprisingly. What's more, resyncing after such exceptions is not at all easy. I would venture to guess that few Python programs even try to do that. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python under PowerShell adds characters
On Thu, Mar 30, 2017 at 4:57 PM, Marko Rauhamaa wrote: > What I'm saying is that every program must behave in a minimally > controlled manner regardless of its inputs (which are not in its > control). With UTF-8, it is dangerously easy to write programs that > explode surprisingly. What's more, resyncing after such exceptions is > not at all easy. I would venture to guess that few Python programs even > try to do that. If you expect to get a series of decimal integers, and you find a "Q" in the middle, is it dangerously easy for your program blow up? How do you resync after that? Do these questions even make sense? Not in my opinion; you got invalid data, so you throw an exception and stop reading data. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
