What should go to stdout/stderr and why Python logging write everything to stderr?

[c.buhtz]

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.

Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.


I do not have any special insight into the developers' minds, but here 
is my own reaction.  Logging output is normally not the primary output 
of a program.  For example, you might want to capture the program's 
output and do some processing on it.  In that case you don't want to 
have to parse out the logging text so as to avoid processing it. If not 
to stdout, then where should logging output go?  The only two 
possibilities seem to be stderr, or a known file.


Also, I think it would be confusing to sometimes have logging output go 
to stdout and other times go to stderr.


In the case of argparse, the messages are the intended program output, 
so to me it makes sense that error announcements should go to stderr and 
requests for help information should go to stdout.


There is probably no single right way to do it, but the way it's being 
done makes sense to me, FWIW.




Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[MRAB]

On 2023-01-03 15:35, c.bu...@posteo.jp wrote:

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.

Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


The purpose of stderr is to display status messages, logging and error 
messages, even user prompts, and not mess up the program's actual 
output. This is important on a *nix system where you might be piping the 
output of one program into the input of another.

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Richard Damon]



> On Jan 3, 2023, at 10:38 AM, c.bu...@posteo.jp wrote:
> Hello,
> 
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
> 
> The logging module write everything to stderr no matter which logging
> level is used.
> 
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
> 
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
> 
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
> 
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

My thought are, that a “Log” should be a COMPLETE output of what is logged. 
Logged messages are NOT intended to be a message to the current user, but for a 
post operation analysis. Remember, a message sent by logging.info may never be 
seen, as that level of logging may be turned off. (You are providing ways to 
control how much is being set to the log, especially if going to the “screen”).

As such, stderr is a often better spot to send the log than stdout, as the 
program may well be using stdout to interact with the user, and you don’t want 
the interaction cluttered with the logging.

Also, I am fairly sure you can configure the logger to do what you want by just 
configuring two different logging handlers, one for stderr that does just 
errors, and one for stdout that shows all messages. If you don’t want the 
errors on stderr and stdout, just log the errors to a different logger which is 
on an error.* hierarchy and don’t let that hierarchy propagate up to the root 
logger.

 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

Really doesn’t have much to do with Python and very much with standard streams, 
which go back as far as the 1950s. 
https://en.wikipedia.org/wiki/Standard_streams

When a user types -h / --help to a Python argparse script, the output of the 
script is the help message. The standard behavior is to print that and exit, 
regardless of any other arguments that are passed. So its behavior is 
consistent with the concept of standard out.

From: Python-list  on 
behalf of MRAB 
Date: Tuesday, January 3, 2023 at 11:41 AM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 2023-01-03 15:35, c.bu...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.

The purpose of stderr is to display status messages, logging and error
messages, even user prompts, and not mess up the program's actual
output. This is important on a *nix system where you might be piping the
output of one program into the input of another.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, c.bu...@posteo.jp  wrote:
>
> If the user request the usage info via "-h" it goes to stdout.

The standard file for application output is sys.stdout. Note that
sys.stdout may be buffered, particularly if it isn't a tty. When a
file is buffered, writes are aggregated and only written to the OS
file when the buffer fills up or is manually flushed.

> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.

The standard file for error messages and other diagnostic information
is sys.stderr. This file should never be buffered.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Keith Thompson]

MRAB  writes:
[...]
> The purpose of stderr is to display status messages, logging and error
> messages, even user prompts, and not mess up the program's actual 
> output. This is important on a *nix system where you might be piping
> the output of one program into the input of another.

I would expect user prompts to be written to stdout, or perhaps to some
system-specific stream like the current tty, not to stderr.  If a
program has user prompts, it probably doesn't make sense to pipe its
output to the input of another.

-- 
Keith Thompson (The_Other_Keith) keith.s.thompso...@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 11:51 AM, Stefan Ram wrote:

Thomas Passin  writes:

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:
Also, I think it would be confusing to sometimes have logging output go
to stdout and other times go to stderr.


   In UNIX, the output of a program can be redirected,
   so error messages written to the output might get appended
   to some result file and might not be seen by the user.


Yes, and in Windows, too.  Note that stderr messages can be written or 
redirected to a file, too, and then would not be seen on the terminal.



   Therefore, UNIX and C provide the unbuffered "stderr" stream
   that can be redirected separately.

   The following example shows how logging might write to stdout.

import logging
import sys
logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
logging.error( 'example' )


This is starting to look too much like java.  Yuck!

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

If sys.stdout is a tty, it typically flushes on newline. e. g.

!/usr/bin/env python3
import time
import sys
print("No flush",end='',file=sys.stdout)
time.sleep(2)
print("flushed",file=sys.stdout)
time.sleep(5)

will print the “flushed” 5 seconds before the script ends

From: Python-list  on 
behalf of Eryk Sun 
Date: Tuesday, January 3, 2023 at 1:33 PM
To: c.bu...@posteo.jp 
Cc: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 1/3/23, c.bu...@posteo.jp  wrote:
>
> If the user request the usage info via "-h" it goes to stdout.

The standard file for application output is sys.stdout. Note that
sys.stdout may be buffered, particularly if it isn't a tty. When a
file is buffered, writes are aggregated and only written to the OS
file when the buffer fills up or is manually flushed.

> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.

The standard file for error messages and other diagnostic information
is sys.stderr. This file should never be buffered.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Weatherby,Gerard  wrote:
> If sys.stdout is a tty, it typically flushes on newline. e. g.

Sorry if I wasn't clear. Yes, a tty file will be buffered, but it's
line buffering, which isn't an issue as long as lines are written to
stdout. I was referring to full buffering of pipe and disk files. This
can be a problem when printing diagnostics. We went the information
written to the file immediately, not whenever the buffer happens to
fill up. Thus sys.stderr is unbuffered.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[c . buhtz]

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs go 
to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Cameron Simpson]

On Jan 3, 2023, at 10:38 AM, c.bu...@posteo.jp wrote:
this posting isn't about asking for a technical solution. My 
intention is to understand the design decision Python's core developers made in

context of that topic.

The logging module write everything to stderr no matter which logging
level is used.


Sure. They're logs.


The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.


Error messages to stderr. _Requested output_ to stdout. Good with me.


Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.


I entirely disagree.

Consider a shell pipeline:

cmd1 | cmd2 | cmd3

where `cmd2` and `cmd3` parse data coming down the pipes. Letting 
logging info into those data streams would produce garbage. This 
separation between core programme output and error/logging output is the 
basic reason that the two streams exist, and the reason logging goes to 
stderr by default.


You're conflating "normal output" with INFO, which is not a substitute 
for eg `print()`.



Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.


Basicly, logging data is not the normal output. So it does not get sent 
to stdout. The log level doesn't change that characteristic.



My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


I think you should, or you should fiddle the handlers as you previous 
resist doing if it is your goal.


But keep in mind that if you do that globally (eg in the root logger) it 
will affect _all_ logging calls, not merely your own, and other 
libraries will assume they can log at whatever level and not pollute 
stdout with their logs.


Basicly, logging isn't "output".

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 08:25,  wrote:
>
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.

But why are DEBUG logs part of stdout? That doesn't make any sense.

> The main question here is why does Python deciecded to make all logs go
> to stderr?
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
>
> If not, what messages should go into logging.info()? Can you name me
> some examples?

Reports of what the program is doing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Michael Torrie]

On 1/3/23 11:45, Keith Thompson wrote:
> MRAB  writes:
> [...]
>> The purpose of stderr is to display status messages, logging and error
>> messages, even user prompts, and not mess up the program's actual 
>> output. This is important on a *nix system where you might be piping
>> the output of one program into the input of another.
> 
> I would expect user prompts to be written to stdout, or perhaps to some
> system-specific stream like the current tty, not to stderr.  If a
> program has user prompts, it probably doesn't make sense to pipe its
> output to the input of another.

I can't think of a specific example, but I know I have piped the output
of a program while at the same time interacting with a prompt on stderr.
 A rare thing, though.

Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams.  Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[2QdxY4RzWzUUiLuE]

On 2023-01-03 at 21:24:20 +,
c.bu...@posteo.jp wrote:

> The main question here is why does Python deciecded to make all logs
> go to stderr?

It makes sense to send all logging to one destination, and stderr is
that place.

> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?

Aha.  This seems to be the misconception.  "Usual" in the sense of "the
application is running as usual," but not in the sense of "the output
the users usually see."

Logging is the process where applications write details that ordinarily
aren't visible to the user.  Logs are for forensics, after the fact.
There is an entire universe of software for managing (writing, reading,
archiving, retrieving, searching, etc.) log files.

At my last job, we wrote gigabytes of compressed logs every hour, and
read roughly two or three out of a million entries to track down
questions from our users.  Users wouldn't have understood 99% of what
was in the logs anyway.

I also worked a lot in embedded systems, like telephone switches.  Users
(i.e., the people with the telephones) didn't have access to the
switches, but when things went wrong, I could look at the switches' logs
to get the their view of what happened.

> If not, what messages should go into logging.info()? Can you name me some
> examples?

INFO level messages usually track "regular" happy-path progress through
your application:  reading config files, calculating results, writing
report.  Again, not meant for users (who would have their own progress
meters, textual or graphical), but for tracking down problems when there
are problems.  For one example, the name of the config file, even if you
only tell the users that you found a config file.

HTH
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 09:17, Michael Torrie  wrote:
>
> On 1/3/23 11:45, Keith Thompson wrote:
> > MRAB  writes:
> > [...]
> >> The purpose of stderr is to display status messages, logging and error
> >> messages, even user prompts, and not mess up the program's actual
> >> output. This is important on a *nix system where you might be piping
> >> the output of one program into the input of another.
> >
> > I would expect user prompts to be written to stdout, or perhaps to some
> > system-specific stream like the current tty, not to stderr.  If a
> > program has user prompts, it probably doesn't make sense to pipe its
> > output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
>  A rare thing, though.

That can certainly happen, but you may more often see things that go
directly to the TTY rather than using the standard streams at all. For
example, ssh and sudo will prompt on the terminal if they need a
password, to avoid interfering with the standard streams (which can
and often will be redirected).

> Maybe some day an interface and shell syntax will be developed to
> interact with an arbitrary number of standard streams.  Current piping
> syntax really only works well with one stream and even trying to use
> stderr and stdout with pipes and filters in a shell is awkward.

Yeah, maybe some day. And then maybe someone will use a time machine
to backport it into bash. You have at least ten available, with more
being possible but potentially causing conflicts:

https://www.gnu.org/software/bash/manual/html_node/Redirections.html

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-03, Michael Torrie  wrote:
> On 1/3/23 11:45, Keith Thompson wrote:
>> MRAB  writes:
>> [...]
>>> The purpose of stderr is to display status messages, logging and error
>>> messages, even user prompts, and not mess up the program's actual 
>>> output. This is important on a *nix system where you might be piping
>>> the output of one program into the input of another.
>> 
>> I would expect user prompts to be written to stdout, or perhaps to some
>> system-specific stream like the current tty, not to stderr.  If a
>> program has user prompts, it probably doesn't make sense to pipe its
>> output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.

Programs that ask for passwords often do it by reading/writing to fd 0
or /dev/tty so that stdout is unaffected.

--
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 09:52, Grant Edwards  wrote:
>
> On 2023-01-03, Michael Torrie  wrote:
> > On 1/3/23 11:45, Keith Thompson wrote:
> >> MRAB  writes:
> >> [...]
> >>> The purpose of stderr is to display status messages, logging and error
> >>> messages, even user prompts, and not mess up the program's actual
> >>> output. This is important on a *nix system where you might be piping
> >>> the output of one program into the input of another.
> >>
> >> I would expect user prompts to be written to stdout, or perhaps to some
> >> system-specific stream like the current tty, not to stderr.  If a
> >> program has user prompts, it probably doesn't make sense to pipe its
> >> output to the input of another.
> >
> > I can't think of a specific example, but I know I have piped the output
> > of a program while at the same time interacting with a prompt on stderr.
> > A rare thing, though.
>
> Programs that ask for passwords often do it by reading/writing to fd 0
> or /dev/tty so that stdout is unaffected.
>

Reading/writing the FD is the same as using stdout (technically you'd
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 10:28, Stefan Ram  wrote:
>
> c.bu...@posteo.jp writes:
> >But I don't want to make all log levels go to stdout. Just DEBUG and
> >INFO.
>
>   The following was stripped down from something some guy
>   posted on a web site, maybe it was "rkachach".

Yet another shining example of bad code that does what someone asks,
but will make life harder for the next person to come along and try to
work with that program. It's like if someone asked for a way to make
True equal to -1 instead of 1, for compatibility with BASIC. Bad idea.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Chris Angelico  wrote:
>
> writing the FD is the same as using stdout

Except stdout may be buffered. One should probably flush the buffer
before each raw write to the file descriptor.

> use /dev/tty to reach for the console directly.

On Windows, open "CON" (read or write), "CONIN$" (read-write), or
"CONOUT$" (read-write). Or use the more explicit device paths
"\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
console, one can call WinAPI AllocConsole() or
AttachConsole(process_id) to obtain one.
-- 
https://mail.python.org/mailman/listinfo/python-list

hello can I be in your group?

[September Skeen]

I was wondering if I could be in your group
-- 
https://mail.python.org/mailman/listinfo/python-list

Python - working with xml/lxml/objectify/schemas, datatypes, and assignments

[aapost]

I am trying to wrap my head around how one goes about working with and 
editing xml elements since it feels more complicated than it seems it 
should be.. Just to get some feedback on how others might approach it 
and see if I am missing anything obvious that I haven't discovered yet, 
since maybe I am wandering off in a wrong way of thinking..


I am looking to interact with elements directly, loaded from a template, 
editing them, then ultimately submitting them to an API as a modified 
xml document.


Consider the following:

from lxml import objectify, etree
schema = etree.XMLSchema(file="path_to_my_xsd_schema_file")
parser = objectify.makeparser(schema=schema, encoding="UTF-8")
xml_obj = objectify.parse("path_to_my_xml_file", parser=parser)
xml_root = xml_obj.getroot()

let's say I have a Version element, that is defined simply as a string 
in a 3rd party provided xsd schema




and is set to a number 2342 in my document

The xml file loads with the above code successfully against the schema

But lxml objectify decides the element type is Int, and the pytype is int..

Version 
Version.pyval 

Let's say I want this loaded into a UI with a variety of dynamically 
loaded entry widgets so I can edit a large number of values like this 
and of many other different types.


I can assign in one of two ways (both resulting the same)
xml_root.Version =
xml_root['Version'] =
(if there is some other more kosher way of assignment, let me know)

I can assign "2342" and the element suddenly becomes a 'lxml.objectify.StringElement'>


I can assign 1.4 and the element suddenly becomes a 'lxml.objectify.FloatElement'>


The schema does not check during this assignment, it could be invalid, 
like assigning "abc" to a xs:dateTime and it does so any way.
The original value is lost. The only way I see to verify against the 
schema again is to do so explicitly against the whole root.


schema.validate(xml_root)

This returns False because of the added xmlns:py, py:pytype stuff, I can 
strip those with:
objectify.deannotate(xml_root[etree.QName(xml_root.Version.tag).localname], 
cleanup_namespaces=True)


and get back to schema.validate(xml_root) validating True. BUT, it 
validates True whether the element is a String, Int, Float, etc (so long 
as it 'could' potentially be a string or something..).. So let's say a 
Version is 322.1121000, should be a string, validates against the schema 
as string, but is now 322.1121 (much more relevant for something like a 
product identification number)


If it is a case where the validate remains False, I then have to 
manually look at the error log via schema.error_log for something like this:


api_files/Basic:0:0:ERROR:SCHEMASV:SCHEMAV_CVC_DATATYPE_VALID_1_2_1: 
Element '{nsstuff}StartTime': 'asdfasdfa' is not a valid value of the 
atomic type 'xs:dateTime'.


Then I have to consider how I should reject the users input.. From a UI 
design standpoint it just seems like a lot of added steps, and redundant 
work on top of a object layer that doesn't really do anything other than 
give me a thumbs up on the way in and a thumbs up on a way out. Rather 
than interacting with an object that can say your change is schema 
approved or not from the get-go, I instead seem to have to parse 10+ 
lines of xsd and design UI interaction much more situationally and 
granularly to assert types and corner cases and preserve original values 
in duplicate structures, etc..


My original assumptions when hearing about xml features doesn't seem to 
exist from what I have found so far. Where schema should be the law, if 
my schema says something should be loaded as a string, it should be a 
string (or something close enough, definitely not an int or float), then 
attempting to assign something to it that doesn't match schema should be 
denied or throw an error. I am sure under the hood it would probably 
have performance draw backs or something.. Oh well.. Back to 
contemplating and tinkering..

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-04, Chris Angelico  wrote:
> On Wed, 4 Jan 2023 at 09:52, Grant Edwards  wrote:
>>
>>> I can't think of a specific example, but I know I have piped the output
>>> of a program while at the same time interacting with a prompt on stderr.
>>> A rare thing, though.
>>
>> Programs that ask for passwords often do it by reading/writing to
>> fd 0 or /dev/tty so that stdout is unaffected.
>
> Reading/writing the FD is the same as using stdout.

No, stdout is fd 1.

> (technically you'd write to fd 1 and read from fd 0),

No, the whole point is _not_ to write to stdout (which would corrupt
the program's output). The way I remember it was that you wrote the
prompt to fd 0, and then read the password from fd 0. That way it
worked even when fd 1 (stdout) was redirected. It still failed if
stdin was redirected...

IIRC, you can't usually write to the stdin stream in C (or Python), so
you had to write(0,promptstr,strlen(promptstr)) instead of
printf("%s",promptstr) or fputs(promptstr,stdin).

> but yes, can use /dev/tty to reach for the console directly.

That's definitely a better option, but I'm pretty sure I've seen
multiple examples over the decades where fd 0 was used instead. Was
/dev/tty a "recent" enough invention that I would have seen
application code that was written before it existed? Or maybe I was
just looking at code written by people who weren't aware of /dev/tty?

--
Grant
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 15:11, Eryk Sun  wrote:
>
> On 1/3/23, Chris Angelico  wrote:
> >
> > writing the FD is the same as using stdout
>
> Except stdout may be buffered. One should probably flush the buffer
> before each raw write to the file descriptor.

FDs can also be buffered. If it's buffering you want to avoid, don't
mess around with exactly which one you're writing to, just flush.

> > use /dev/tty to reach for the console directly.
>
> On Windows, open "CON" (read or write), "CONIN$" (read-write), or
> "CONOUT$" (read-write). Or use the more explicit device paths
> "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
> console, one can call WinAPI AllocConsole() or
> AttachConsole(process_id) to obtain one.

Yeah, /dev/tty is the Unix option, Windows has its own way, but
whatever platform you're on, there's a way to directly reach the
console.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 16:21, Grant Edwards  wrote:
>
> On 2023-01-04, Chris Angelico  wrote:
> > On Wed, 4 Jan 2023 at 09:52, Grant Edwards  
> > wrote:
> >>
> >>> I can't think of a specific example, but I know I have piped the output
> >>> of a program while at the same time interacting with a prompt on stderr.
> >>> A rare thing, though.
> >>
> >> Programs that ask for passwords often do it by reading/writing to
> >> fd 0 or /dev/tty so that stdout is unaffected.
> >
> > Reading/writing the FD is the same as using stdout.
>
> No, stdout is fd 1.
>
> > (technically you'd write to fd 1 and read from fd 0),

I assumed that you were using a shorthand. My bad.

> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...

Writing to FD 0 assumes that (a) it was originally connected to the
TTY, and (b) it hasn't been redirected. Not a reliable assumption.

rosuav@sikorsky:~/tmp$ cat testfds.py
import os
for fd in range(3):
os.write(fd, b"This is on FD %d\n" % fd)
rosuav@sikorsky:~/tmp$ python3 testfds.py
This is on FD 0
This is on FD 1
This is on FD 2
rosuav@sikorsky:~/tmp$ echo | python3 testfds.py
Traceback (most recent call last):
  File "/home/rosuav/tmp/testfds.py", line 3, in 
os.write(fd, b"This is on FD %d\n" % fd)
OSError: [Errno 9] Bad file descriptor

Writing to fd 0 might work if stdout is redirected, but it won't work
if stdin is redirected, so honestly, I don't think this hack is at all
reliable.

> > but yes, can use /dev/tty to reach for the console directly.
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?

No idea. Probably. It's definitely possible that someone tries it with
stdout redirected and goes "oh hey, writing to fd 0 still works", and
never tries it with stdin redirected. It's very common for programmers
to see that something works without knowing WHY it works :)

By the way, I can reinstate the behaviour of "FD 0 writes to the TTY"
in bash by redirecting it:

rosuav@sikorsky:~/tmp$ python3 testfds.py >/dev/null 0<>/dev/tty
This is on FD 0
This is on FD 2

which is another neat demonstration that bash has more power than you
will ever need...

(though redirecting FDs above 2 in read/write mode can be *incredibly* useful.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Grant Edwards  wrote:
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?

POSIX has required "/dev/tty" for 20 years, or longer. It's in the
2004 edition of the spec, which was a minor update to the 2001
edition.

https://pubs.opengroup.org/onlinepubs/95399/basedefs/xbd_chap10.html

> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...

FYI, that wouldn't work on Windows. The "\\.\CONIN$" and "\\.\CONOUT$"
files can be opened with read-write access, but it's for a different
purpose.

A console input buffer can be read from via read() or WinAPI
ReadFile(). It can also be read from using IOCTLs that work with
16-bit wide-character strings, which is the basis of WinAPI
ReadConsoleW() and ReadConsoleInputW(). A console input buffer can be
*written to* via the IOCTL-based function WriteConsoleInputW().

A console screen buffer can be written to via write() or WinAPI
WriteFile(). It can also be written to using IOCTLs that work with
16-bit wide-character strings, which is the basis of WriteConsoleW()
and WriteConsoleOutputW(). A console screen buffer can be *read from*
via the IOCTL-based function ReadConsoleOutputW().
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Chris Angelico  wrote:
>
> FDs can also be buffered. If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.

I meant to flush a C FILE stream or Python file before writing
directly to the file descriptor, in order to avoid out-of-sequence and
interlaced writes that make no sense. Any OS buffering on the file
doesn't matter in this regard.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 17:26, Eryk Sun  wrote:
>
> On 1/3/23, Chris Angelico  wrote:
> >
> > FDs can also be buffered. If it's buffering you want to avoid, don't
> > mess around with exactly which one you're writing to, just flush.
>
> I meant to flush a C FILE stream or Python file before writing
> directly to the file descriptor, in order to avoid out-of-sequence and
> interlaced writes that make no sense. Any OS buffering on the file
> doesn't matter in this regard.

True, that can help. If you're about to prompt for a password, though,
OS buffering can most certainly make a difference. And it doesn't hurt
to explicitly flush when you need it flushed.

I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1", but that's not going to work when you use certain
keyboard-input routines (ISTR readline was one of the few things that
*didn't* have that problem, but I may be misremembering), so the
general strategy was: print your prompt, flush stdout, THEN request
input.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list