Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Michael Torrie via Python-list]

On 11/14/24 12:03 AM, Left Right wrote:
>> On any Unix system this is untrue.  Rotating a log file is quite simple:
> 
> I realized I posted this without cc'ing the list:
> http://jdebp.info/FGA/do-not-use-logrotate.html .
> 
> The link above gives a more detailed description of why log rotation
> on the Unix system is not only not simple, but is, in fact,
> unreliable.

Nothing in that article contradicts what I said about renaming log
files.  His argument is that renaming log files messes with tail -F, and
therefore broken and unreliable.  Which a pretty strange argument.  tail
-F might not see some data during the rotation, but the log files
themselves don't miss anything, which was my contention.  In all my
years of sysadmin-ing I have never once worried about problems GNU tail
might have with a file that gets rotated out from under you.  Not sure
why the author is so fixated on it.

There are actual legitimate issues at play, such as a mechanism for
informing the process to close the file (rotate usually requires
processes to respond to SIGHUP).  And of course the disk can fill up
causing a denial of service of one kind or another.  The latter is the
biggest source of problems.

Of course you could just log using the standard libc syslog facility.
Or better yet, start your process from a systemd unit file and let the
journal automatically log stderr.  In both cases that would satisfy the
technical objections of the author of that little treatise.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Jon Ribbens via Python-list]

On 2024-11-14, Michael Torrie  wrote:
> On 11/14/24 12:03 AM, Left Right wrote:
>>> On any Unix system this is untrue.  Rotating a log file is quite simple:
>> 
>> I realized I posted this without cc'ing the list:
>> http://jdebp.info/FGA/do-not-use-logrotate.html .
>> 
>> The link above gives a more detailed description of why log rotation
>> on the Unix system is not only not simple, but is, in fact,
>> unreliable.
>
> Nothing in that article contradicts what I said about renaming log
> files.  His argument is that renaming log files messes with tail -F, and
> therefore broken and unreliable.  Which a pretty strange argument.  tail
> -F might not see some data during the rotation, but the log files
> themselves don't miss anything, which was my contention.  In all my
> years of sysadmin-ing I have never once worried about problems GNU tail
> might have with a file that gets rotated out from under you.  Not sure
> why the author is so fixated on it.

I really wouldn't worry about anything Jonathan de Boyne Pollard says.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Ethan Furman via Python-list]

On 11/13/24 23:03, Left Right via Python-list wrote:

>> On any Unix system this is untrue.  Rotating a log file is quite simple:
>
> I realized I posted this without cc'ing the list:
> http://jdebp.info/FGA/do-not-use-logrotate.html .
>
> The link above gives a more detailed description of why log rotation
> on the Unix system is not only not simple, but is, in fact,
> unreliable.

Having read the linked article, I see it is not relevant to Python, as Python's 
logging tool is
the writer/rotator program, thus no window for lost entries exists.

> NB. Also, it really rubs me the wrong way when the word "standard" is
> used to mean "common" (instead of "as described in a standard
> document").

Yes, that is irritating.

>  And when it comes to popular tools, oftentimes "common"
> is wrong because commonly the tool is used by amateurs rather than
> experts.  In other words, you only reinforced what I wrote initially:
> plenty of application developers don't know how to do logging well.
> It also appears that they would lecture infra / ops people on how to
> do something that they aren't experts on, while the latter are :)

Well, since this is a Python list, perhaps you could make sure your advice is also Python appropriate.  I appreciate 
diversions into general areas and learning new things, but your general claims were untrue when it comes to Python 
specifically, and that was unclear until I read your linked post.


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Chris Angelico via Python-list]

On Thu, 14 Nov 2024 at 18:05, Left Right via Python-list
 wrote:
>
> > On any Unix system this is untrue.  Rotating a log file is quite simple:
>
> I realized I posted this without cc'ing the list:
> http://jdebp.info/FGA/do-not-use-logrotate.html .
>
> The link above gives a more detailed description of why log rotation
> on the Unix system is not only not simple, but is, in fact,
> unreliable.

You're assuming a very specific tool here. Log rotation isn't
necessarily performed by that one tool. There are many ways to do it.

Log to stderr. That puts the power in the hands of the sysadmin,
rather than forcing trickery like setting the log file name to be
/proc/self/fd/2 to get around it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Barry via Python-list]

> On 14 Nov 2024, at 14:07, Loris Bennett via Python-list 
>  wrote:
> 
> I don't quite understand what your suggestion is.  Do you mean that I
> should log to stderr and then run my program as
> 
>  my_program ... 2>&1 | logger

On almost all Linux distros you would run a long running program as a systemd 
service and let it put logs into the journal. I wonder if that was what was 
being hinted at?

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[Loris Bennett via Python-list]

Left Right  writes:

>> I am not entirely convinced by NB2.  I am, in fact, a sort of sysadmin
>> person and most of my programs write to a log file.  The programs are
>> also moderately complex, so a single program might access a database,
>> query an LDAP server, send email etc., so potentially quite a lot can go
>> wrong.  They are also not programs whose output I would pipe to another
>> command.  What would be the advantage of logging to stderr?  Quite apart
>> from that, I find having a log file a useful for debugging when I am
>> developing.
>
> First, the problem with writing to files is that there is no way to
> make these logs reliable.  This is what I mean by saying these are
> unreliable: since logs are designed to grow indefinitely, the natural
> response to this design property is log rotation.  But, it's
> impossible to reliably rotate a log file.  There's always a chance
> that during the rotation some log entries will be written to the file
> past the point of rotation, but prior to the point where the next logs
> volume starts.

> There are similar reliability problems with writing to Unix or
> Internet sockets, databases etc.  For different reasons, but at the
> end of the day, whoever wants logs, they want them to be reliable.
> Both simplicity and convention selected for stderr as the only and the
> best source of logging output.

If I understand correctly you are not saying that logrotate is less
reliable that the other methods mentioned above.  But in that case,
reliability seems no more of a reason not to log to files than it is a
reason not to write to a socket or to a database.

> Programs that write their output to log files will always irritate
> their users because users will have to do some detective work to
> figure out where those files are, and in some cases they will have to
> do administrative works to make sure that the location where the
> program wants to store the log files is accessible, has enough free
> space, is speedy enough etc.

All your points regarding the work involved are valid, but many
programs, such as MariaDB, OpenLDAP or SSSD, do write to a log file (and
it is usually under /var/log or /var/log/something.  So it seems like a
common approach.

Besides, I define the location of the logfile in the config file for the
program (the problem in my original question arose from this filename in
the config file not existing).  So finding the location is not an issue.
You have to find the config file, of course, but I think /etc or
/usr/local/etc are fairly standard and my programs generally have an
option '--config-file' anyway.

>  So, from the ops perspective, whenever I
> come across a program that tries to write logs to anything other than
> stderr, I make an earnest effort to throw that program into the gutter
> and never touch it again.  It's too much headache to babysit every
> such program, to remember the location of the log files of every such
> program, the required permissions, to provision storage.  If you are
> in that line of work, you just want all logs to go to the same place
> (journal), where you can later filter / aggregate / correlate and
> perform other BI tasks as your heart desires.

That may be true in many cases, but those I am dealing with don't
require much filtering beyond 'grep' and also don't require aggregation
or correlation.

> Of course, if you only administer your own computer, and you have low
> single digits programs to run, and their behavior doesn't change
> frequently, and you don't care to drop some records every now and
> then... it's OK to log to files directly from a program.  But then you
> aren't really in the sysadmin / infra / ops category, as you are more
> of a hobby enthusiast.

What I do is indeed a bit of a niche, but I do get paid for this, so I
would not consider myself a 'hobby enthusiast'.

> Finally, if you want your logs to go to a file, and currently, your
> only option is stderr, your shell gives you a really, really simple
> way of redirecting stderr to a file.  So, really, there aren't any
> excuses to do that.

I don't quite understand what your suggestion is.  Do you mean that I
should log to stderr and then run my program as

  my_program ... 2>&1 | logger

?

Cheers,

Loris

-- 
This signature is currently under constuction.
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: eGenix PyRun - One file Python Runtime 2.6.0

[eGenix Team via Python-list]

*ANNOUNCING*


   eGenix PyRun - One file Python Runtime

Version 2.6.0

Python runtime taking up just 4-6MB on disk

This announcement is also available on our web-site for online reading:
https://www.egenix.com/company/news/eGenix-PyRun-2.6.0-GA.html


*INTRODUCTION*

*eGenix PyRun*™  
is our open source, one file, no installation version of Python, making 
the distribution of a Python interpreter to run Python based scripts and 
applications to Unix based systems simple and efficient.


eGenix PyRun's executable only needs 4-6MB on disk, but still supports 
most Python applications and scripts.


Compared to a regular Python installation of typically 100MB on disk, 
eGenix PyRun is ideal for applications and scripts that need to be 
distributed to containers, VMs, clusters, client installations, 
customers or end-users.


It makes "installing" Python on a Unix based system as simple as copying 
a single file.


eGenix has been using eGenix PyRun as run-time for the Linux version of 
mxODBC Connect Server 
 product since 
2008 with great success and decided to make it available as a 
stand-alone open-source product.


We provide the source archive to build your own *eGenix PyRun on Github* 
, as well as a few binary 
distributions to get you started on Linux x86_64. In the future, we will 
set up automated builds for several other platforms.


Please see the product page for more details:

>>> eGenix PyRun - One file Python Runtime 





*NEWS*

This major release of eGenix PyRun 
 comes with the following 
enhancements:


 * Added support for *Python 3.12*
 * Added support for LTO release builds
 * Added dev build targets for development; these don't use PGO and
   thus build faster

For a complete list of changes, please see the *eGenix PyRun Changelog 
*.



Enjoy,

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 13 2024)

Python Projects, Coaching and Support ...https://www.egenix.com/
Python Product Development ...https://consulting.egenix.com/



::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48

D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
--
https://mail.python.org/mailman/listinfo/python-list

Re: FileNotFoundError thrown due to file name in file, rather than file itself

[D'Arcy Cain via Python-list]

On 11/13/24 02:12, Roel Schroeven via Python-list wrote:
What I most often do is use one logfile per day, with the date in the 
filename. Then simply delete all files older than 7 days, or 30 days, or 
whatever is useful for the task at hand. Not only does that sidestep any 
issues with rotating logs, but I also find it's very useful to have the 
date in the filename.


I do something similar for my system logs except that I let the system 
use the usual names and, at midnight, I rename the file appending the 
previous day's date to it and restarting services.


--
D'Arcy J.M. Cain
Vybe Networks Inc.
http://www.VybeNetworks.com/
IM:da...@vybenetworks.com VoIP: sip:da...@vybenetworks.com
--
https://mail.python.org/mailman/listinfo/python-list