RFC: Proposal: Deterministic Object Destruction
Informal Background === Python's lack of Deterministic Object Destruction semantics strikes me as very unpythonic. This state of affairs spawns lengthy diatribes on __del__(), a variety of specialised and onerous resource management solutions and malignant carbuncles such as PEP 343 (the "with" statement). None of these match the practical utility, simplicity, generality and robustness of Deterministic Object Destruction combined with the RAII idiom to manage resources. I have therefore drafted the radical but simple and 100% backward compatible proposal below to remedy this situation and pave the way for numerous tidy ups (out of scope of this proposal). I am now retiring to a safe distance and invite your comments on the PEP below:- Abstract This PEP proposes that valid python interpreters *must* synchronously destroy objects when the last reference to an object goes out of scope. This interpreter behaviour is currently permitted and exhibited by the reference implementation [CPython], but it is optional. Motivation == To make the elegance (simplicity, generality, and robustness) of the "Resource Acquisition Is Initialization" (RAII) idiom available to all python applications and libraries. To render less elegant and more onerous resource management solutions, such as the "with" Statement [PEP 343] obsolete. Specification = When the last reference to an object goes out of scope the intepreter must synchronously, in the thread that releases the last reference, invoke the object's __del__() method and then free the memory occupied by that object. Backwards Compatibility === This proposal preserves 100% backwards compatibility. The language currently does not prescribe when, or even if, __del()__ will be called and object memory freed. Existing python applications and libraries can make no assumptions about this behaviour. Reference Implementation CPython References == PEP 343 -- The "with" Statement -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Wednesday, February 28, 2018 at 11:02:17 PM UTC, Chris Angelico wrote:
> On Thu, Mar 1, 2018 at 9:51 AM, ooomzay wrote:
> > Specification
> > =
> >
> > When the last reference to an object goes out of scope the intepreter must
> > synchronously, in the thread that releases the last reference, invoke the
> > object's __del__() method and then free the memory occupied by that object.
> >
>
> If it were that simple, why do you think it isn't currently mandated?
We know the implementation is feasible because CPython, the reference
interpreter implementation, already does this. I am guessing that there is a
lobby from garbage platforms where it would be harder to implement and
integrate however the object of this proposal is the integrity of the language,
not on making it easier to implement using garbage platforms.
> Here's one example: reference cycles. When do they get detected?
Orphan cycle _detection_ is orthogonal to this proposal. As cycles are always a
symptom of a design error I personally have little interest in them other than
some facility to detect/debug and eliminate them from applications by design
(another topic).
> Taking a really simple situation:
>
> class Foo:
> def __init__(self):
> self.self = self
> print("Creating a Foo")
> def __del__(self):
> print("Disposing of a Foo")
>
> foo = Foo()
> foo = 1
>
> When do you expect __del__ to be called?
At the point of (re)assignment: "foo = 1"
> How do you implement this efficiently and reliably?
CPython already has a reliable and reasonably efficient implementation based on
reference counting.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Wednesday, February 28, 2018 at 11:45:24 PM UTC, ooo...@gmail.com wrote:
> On Wednesday, February 28, 2018 at 11:02:17 PM UTC, Chris Angelico wrote:
> > On Thu, Mar 1, 2018 at 9:51 AM, ooomzay wrote:
> > [snip]
> > Taking a really simple situation:
> >
> > class Foo:
> > def __init__(self):
> > self.self = self
> > print("Creating a Foo")
> > def __del__(self):
> > print("Disposing of a Foo")
> >
> > foo = Foo()
> > foo = 1
> >
> > When do you expect __del__ to be called?
>
> At the point of (re)assignment: "foo = 1"
Oh... I now see there is a (non-weak) self reference in there.
So in this case it would be orphaned. It is a design error and should be
recoded. I don't care how it is detected for current purposes.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Thursday, March 1, 2018 at 12:35:38 PM UTC, Richard Damon wrote: > [snip] > I disagree with the original complaint that these > are always 'errors', if you know you have garbage collection, the > allowance of cycles knowing they will still get cleaned up is a useful > simplification if you don't need the immediate clean up. I (O.P.) regret making that complaint, it was flippant and has distracted from the essence of the PEP. Whilst my informal introduction was deliberately a little provocative, my PEP is sincere and does not affect the ability to create orphan cycles. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Thursday, March 1, 2018 at 12:15:57 AM UTC, Paul Rubin wrote:
> RAII is similar to Python's "with" statement. So it sounds like OP
> wants to replace one "malignant carbuncle" with another one.
I would like to understand why you think RAII is not substantially more
pythonic than "With". Below I have sketched out a File-like resource
management scenario using both PEP343 and RAII idioms for comparison.
First lets look at the implementation of the competing resource management
classes:-
class PEP343FileAccess():
'''
A File Access-like resource with self-contained Context Manager for use
with "with".
This could be acheived with two separate classes but I don't think that
adds anything except lines of code.
'''
# Structors
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
self.handle = None
# dummy file content accessible only when handle is not None
self.lines = ['one', 'two', 'three']
# PEP343 Context Manager compliance
def __enter__(self):
self.handle = low_level_file_open(self.filename, self.mode) # fictitious
return self
def __exit__(self, extype, exvalue, extraceback):
low_level_file_close(self.handle) # fictitious function
self.handle = None
# Example methods
def __iter__(self):
assert self.handle, "File has been closed"
for line in self.lines:
yield line
def write(self, line):
assert self.handle, "File has been closed"
self.lines.append(line)
class RAIIFileAccess():
'''File Access-like Resource using RIAA idiom'''
# Structors
def __init__(self, filename, mode):
self.handle = low_level_file_open(filename, mode) # fictitious
# dummy content accessible as long as the object exists (invariant)
self.lines = ['one', 'two', 'three']
def __del__(self):
low_level_file_close(self.handle) # fictitious function
# Example methods
def __iter__(self):
for line in self.lines:
yield line
def write(self, line):
self.lines.append(line)
What I see is that PEP343 requires two new methods: __enter__ & __exit__.
RIAA requires no new methods.
RIAA resources are invariant: If you have a reference to it you can use it.
PEP343 resources can not be invariant: To be robust the enter/exit state
must be tracked and checked. (assert self.handle in the example)
Now lets look at example resource usage:-
def pep343_example():
with PEP343FileAccess("src.txt", 'r') as src,
PEP343FileAccess("dst.txt", 'w') as dst:
for line in src:
dst.write(line)
def raii_example():
src = RAIIFileAccess("src.txt", 'r')
dst = RAIIFileAccess("dst.txt", 'w')
for line in src:
dst.write(line)
PEP343 requires specialised "with" syntax, RIAA requires no new syntax.
Furthermore, although src & dst objects are still accessible outside the
PEP343 "with" block they are not in a usable state (not invariant).
In the RIAA case the resources are guaranteed to be in a usable state as long
as any reference exists (invariant). References can also be safely
passed around.The resource will be freed/closed when the last man has finished
with it, even in the face of exceptions.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited)
On Thursday, March 1, 2018 at 11:51:50 PM UTC, Lawrence D’Oliveiro wrote: > On Friday, March 2, 2018 at 12:39:01 PM UTC+13, ooo...@gmail.com wrote: > > class RAIIFileAccess(): > > '''File Access-like Resource using [RAII] idiom''' > > > > ... > > > > def __del__(self): > > low_level_file_close(self.handle) # fictitious function > > This may be OK for files opening for reading, not so good for writing. Please could you explain the issue you perceive with writing? -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Thursday, March 1, 2018 at 11:59:26 PM UTC, Chris Angelico wrote:
> On Fri, Mar 2, 2018 at 10:38 AM, ooomzay wrote:
> > def raii_example():
> >
> > src = RAIIFileAccess("src.txt", 'r')
> > dst = RAIIFileAccess("dst.txt", 'w')
> >
> > for line in src:
> > dst.write(line)
>
> What happens if we make this change?
>
> def raii_example():
> global dst
> src = RAIIFileAccess("src.txt", 'r')
> dst = RAIIFileAccess("dst.txt", 'w')
>
> for line in src:
> dst.write(line)
>
> When does the destination file get closed?
When you execute:-
del dst
or:-
dst = something_else
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 12:14:53 AM UTC, Chris Angelico wrote:
> On Fri, Mar 2, 2018 at 11:07 AM, ooomzay wrote:
> > On Thursday, March 1, 2018 at 11:59:26 PM UTC, Chris Angelico wrote:
> >> On Fri, Mar 2, 2018 at 10:38 AM, ooomzay wrote:
> >> > def raii_example():
> >> >
> >> > src = RAIIFileAccess("src.txt", 'r')
> >> > dst = RAIIFileAccess("dst.txt", 'w')
> >> >
> >> > for line in src:
> >> > dst.write(line)
> >>
> >> What happens if we make this change?
> >>
> >> def raii_example():
> >> global dst
> >> src = RAIIFileAccess("src.txt", 'r')
> >> dst = RAIIFileAccess("dst.txt", 'w')
> >>
> >> for line in src:
> >> dst.write(line)
> >>
> >> When does the destination file get closed?
> >
> > When you execute:-
> >
> >del dst
> >
> > or:-
> >
> >dst = something_else
>
> What if you don't?
Then the resource will remain open until your script exits at which point it is
probably not very well defined exactly when or even if the destructor/__del__
will be called.
I.e. Don't do this! Did you have some realistic case in mind or are you just
probing the behaviour?
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 12:22:13 AM UTC, MRAB wrote: > On 2018-03-01 23:38, ooomzay wrote: [Snip] > > PEP343 requires two new methods: __enter__ & __exit__. > > RIAA requires no new methods. > > > > RIAA resources are invariant: If you have a reference to it you can use it. > > PEP343 resources can not be invariant: To be robust the enter/exit state > > must be tracked and checked. (assert self.handle in the example) [Snip] > > PEP343 requires specialised "with" syntax, RIAA requires no new syntax. > > Furthermore, although src & dst objects are still accessible outside the > > PEP343 "with" block they are not in a usable state (not invariant). > > > > In the RIAA case the resources are guaranteed to be in a usable state as > > long > > as any reference exists (invariant). References can also be safely > > passed around.The resource will be freed/closed when the last man has > > finished > > with it, even in the face of exceptions. > > [snip] > > > What's the difference between 'RAIIFileAccess' and 'open'? I listed all the differences I know of in my preceding post. (I have snipped the other stuff so they are easy to see). -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 8:16:22 AM UTC, Paul Rubin wrote:[snip] > controlling stuff like file handles > with scopes (like with "with") is fine. How does with work for non-trivial/composite objects that represent/reference multiple resources or even a hierarchy of such objects where all the resources held must be released in a timely fashion when finished with? -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited)
On Friday, March 2, 2018 at 1:59:02 AM UTC, Lawrence D’Oliveiro wrote:
> On Friday, March 2, 2018 at 1:03:08 PM UTC+13, ooo...@gmail.com wrote:
> > On Thursday, March 1, 2018 at 11:51:50 PM UTC, Lawrence D’Oliveiro wrote:
> >> On Friday, March 2, 2018 at 12:39:01 PM UTC+13, ooo...@gmail.com wrote:
> >>>
> >>> class RAIIFileAccess():
> >>> '''File Access-like Resource using [RAII] idiom'''
> >>> ...
> >>> def __del__(self):
> >>> low_level_file_close(self.handle) # fictitious function
> >>
> >> This may be OK for files opening for reading, not so good for writing.
> >
> > Please could you explain the issue you perceive with writing?
>
> Compare the difference in behaviour between
>
> f = open("/dev/full", "w")
> f.write("junk")
> f.close()
>
> and
>
> f = open("/dev/full", "w")
> f.write("junk")
> f = None
>
> Should there be a difference in behaviour?
Could you please be very explicit about why you think the PEP would be more
suitable when reading a file than when writing a file? You have responded with
another question that I just can't see the relevance of (yet).
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 4:35:41 AM UTC, Steven D'Aprano wrote:
> On Thu, 01 Mar 2018 16:26:47 -0800, ooomzay wrote:
>
> >> >> When does the destination file get closed?
> >> >
> >> > When you execute:-
> >> >
> >> >del dst
> >> >
> >> > or:-
> >> >
> >> >dst = something_else
> >>
> >> What if you don't?
> >
> > Then the resource will remain open until your script exits at which
> > point it is probably not very well defined exactly when or even if the
> > destructor/__del__ will be called.
> >
> > I.e. Don't do this! Did you have some realistic case in mind or are you
> > just probing the behaviour?
>
>
> If you're going to *require* the programmer to explicitly del the
> reference:
>
> f = open("file")
> text = f.read()
> del f
But I am not! On the contrary RAII frees the programmer from even having to
remember to close the file. The poster asked what would happen if the resource
was deliberately kept open by storing a reference at global scope.
In practice CPython destroys it cleanly on exit - but I am not sure the
language guarantees this - in any case RAII won't make things any worse in this
respect. (Logfiles are a common example of such global resource.)
> then you might as well require them to explicitly close the file:
>
> f = open("file")
> text = f.read()
> f.close()
>
> which we know from many years experience is not satisfactory except for
> the simplest scripts that don't need to care about resource management.
> That's the fatal flaw in RAII:
We must be discussing a different RAII. That is the raison d'etre of RAII: RAII
directly addresses this problem in an exception-safe way that does not burden
the resource user at all.
> the problem is that the lifespan of the resource may
> not be the same as the lifetime of the object.
>
> Especially for files, the
> problem is that the lifespan of resource (the time you are actually using
> it) may be significantly less than the lifespan of the object holding
> onto that resource. Since there's no way for the interpreter to know
> whether or not you have finished with the resource, you have a choice:
>
> - close the resource yourself (either explicitly with file.close(),
> or implicitly with a context manager);
>
> - or keep the resource open indefinitely, until such eventual time
> that the object is garbage collected and the resource closed.
Hence my PEP! It enables RAII. The interpreter merely has to call __del__ as
soon as the object is no longer referenced (as does CPYthon).
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 2:43:09 PM UTC, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 1:18 AM, ooomzay wrote: > > On Friday, March 2, 2018 at 8:16:22 AM UTC, Paul Rubin wrote:[snip] > >> controlling stuff like file handles > >> with scopes (like with "with") is fine. > > > > How does with work for non-trivial/composite objects that > > represent/reference multiple resources or even a hierarchy of such objects > > where all the resources held must be released in a timely fashion when > > finished with? > > > > Can you give me an example that works with RAII but doesn't work in a > 'with' statement? My claim is about the relative elegance/pythonic nature of RAII w.r.t. 'with'. Probably with enough coding and no ommissions or mistakes 'with' could do achieve the same result that RAII does automatically without any burden on the user. Consider the hierarchy/tree of resource-holding objects postulated above... Presumably __enter__, __exit__ (or functional equivalent) would have to be implemented at every layer breaking the invariance. Thus complicating and making less robust every object it touches. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 10:43:57 PM UTC, Steven D'Aprano wrote:
> On Fri, 02 Mar 2018 07:09:19 -0800, ooomzay wrote:
> [...]
> >> If you're going to *require* the programmer to explicitly del the
> >> reference:
> >>
> >> f = open("file")
> >> text = f.read()
> >> del f
> >
> > But I am not! On the contrary RAII frees the programmer from even having
> > to remember to close the file. The poster asked what would happen if the
> > resource was deliberately kept open by storing a reference at global
> > scope.
>
> You say that as if it were difficult to do, requiring the programmer to
> take extraordinary steps of heroic proportion. It doesn't.
> It is unbelievably easy to store the reference at global scope, which
> means that the programmer needs to remember to close the file explicitly
> and RAII doesn't help.
Can you expand on what you mean by "unbelievably easy to store the reference
at global scope". I will assume you are claiming that it is easy to
_inadvertently_ store a global reference. Can you give an example of how
this might happen? (Apart from writing leaky exception handlers, which I
trust no one thinks is a good idea)
> [...]
> Your justification for requiring RAII is to ensure the timely closure of
> resources -- but to do that, you have to explicitly close or delete the
> resource. It simply isn't true that "RAII frees the programmer from even
> having to remember to close the file".
In the special case (keyword global) that you reference resources from global
scope - then yes, you must remember to del the global references before you
exit to avoid relying on the vagaries of exit processing.
This problem with destruction of globals on exit is not unique to python.
There are patterns to manage this for code you control, such as putting them
all in one global "bucket" object and explicitly deleting it before exit.
> >> then you might as well require them to explicitly close the file:
> >>
> >> f = open("file")
> >> text = f.read()
> >> f.close()
> >>
> >> which we know from many years experience is not satisfactory except for
> >> the simplest scripts that don't need to care about resource management.
> >> That's the fatal flaw in RAII:
> >
> > We must be discussing a different RAII. That is the raison d'etre of
> > RAII: RAII directly addresses this problem in an exception-safe way that
> > does not burden the resource user at all.
>
> But as you said yourself, if the resource is held open in a global
> reference, it will stay open indefinitely. And remember, global in this
> context doesn't just mean the main module of your application, but
> *every* module you import.
Can you give an example of such a case where an application-managed
(acquired/opened & released/closed) resource is referenced strongly by
a 3rd party module at global scope? ...in my experience this pattern
usually smells.
> I think you have just put your finger on the difference between what RAII
> *claims* to do and what it *actually* can do.
I can assure you that RAII does what it says on the tin and is relied on in
many critical systems to release resources robustly ... given the
pre-requisite deterministic destruction.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Saturday, March 3, 2018 at 4:33:59 PM UTC, Michael Torrie wrote: > On 03/03/2018 09:02 AM, ooomzay wrote: > > I can assure you that RAII does what it says on the tin and is relied on in > > many critical systems to release resources robustly ... given the > > pre-requisite deterministic destruction. > > Sure but did you read what Paul Moore wrote? Yes. > He said RAII works in C++ > because objects are allocated on the *stack* with strict lifetimes and > scopes. Well he was not telling you the whole story: RAII works just as well with heap objects using smart pointers (unique_ptr and friends) which are a closer analogy to python object references. > In C++, Heap-allocated objects must still be managed manually, without > the benefit of RAII, No one should be manually managing resources on the heap in C++. They should be using smart pointers. > for much of the same reasons as people are giving > here for why RAII is not a good fit for Python. ...for much the same reasons I am giving here for why RAII could be a very good fit for python. > There are smart pointer > objects that try to give RAII semantics to heap-allocated objects, with > varying degrees of success. In other words there are some limitations. Not sure what limitations you are evoking here but I have not had to write a delete or suffered a resource leak in C++ for many years. (we rolled our own smart pointers before they were standardised). > Python does not have stack-allocated objects, so the same issues that > prevent RAII from automatically applying in C++ to heap objects exist here. False premise, false conclusion. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 3:37:25 PM UTC, Paul Moore wrote:
> [...]
> RAII works in C++ (where it was initially invented) because it's used
> with stack-allocated variables that have clearly-defined and limited
> scope.
RAII also works with smart pointers, which are a closer analogue
to python references.
> In my experience writing C++, nobody uses RAII with
> heap-allocated variables - those require explicit allocation and
> deallocation and so are equivalent to having an explicit "close()"
> method in Python (or using __del__ in CPython as it currently exists).
In my experience no-one has been been explicitly deallocating
heap-variables for many years: they have been using smart pointers
and RAII.
> Python doesn't have stack allocation, nor does it have a deterministic
> order of deletion of objects when their last reference goes out of
> scope (which can happen simultaneously for many objects):
>
> class Tracker:
> def __init__(self, n):
> self.n = n
> def __del__(self):
> print("Deleting instance", self.n)
>
> def f():
> a = Tracker(1)
> b = Tracker(2)
>
> f()
>
> The language doesn't guarantee that a is removed before b. Are you
> proposing to make that change to the language as well?
It would improve things more, but I have not included such a proposal
in the current PEP as this is not essential for the main benefit.
> Also Python only has function scope, so variables local to a
> smaller-than-the-function block of code aren't possible. That's
> something that is used in C++ a lot to limit the lifetime of resources
> under RAII. How do you propose to address that (without needing
> explicit del statements)?
Create a sub function.
> That's why the with statement exists, to
> clearly define lifetimes smaller than "the enclosing function".
It does that in a half-hearted and smelly way compared to a function:
The object references are left dangling in a broken state (not invariant).
In fact you would do well to wrap your "with"s in a function to contain
the smell.
> Your proposal doesn't offer any equivalent (other than an extra function).
An equivalent is not needed. Using a function is the ideomatic and does not
leak - unlike the "with" hack. (IMO block-scoping would enhance the langauge
but is not essential so not in this PEP).
> Consider C++:
>
> void fn() {
> for (i = 0; i < 1; ++i) {
> char name[100];
> sprintf(name, "file%d.txt, i);
> File f(name); // I don't think std::ofstream doesn't support RAII
> f << "Some text";
> }
> }
>
> Or (real Python):
>
> def fn():
> for i in range(1):
> with open(f"file{i}.txt", "w") as f:
> f.write("Some text")
>
> How would you write this in your RAII style - without leaving 10,000
> file descriptors open until the end of the function?
def write_some_text_to_file(fname):
f = RAIIFileAccess(fname, 'w')
f.write("Some text")
def fn():
for i in range(1):
write_some_text_to_file(f"file{i}.txt")
> That's both less efficient (function calls have a cost)
Oh come now. No one is choosing to use python for its efficiency and
functions that deal with real resources, such as files, are likely
dominated by the os fopen/close etc.
> and less maintainable than the with-statement version.
I disagree. Even for such a trivial example, as soon as you
start to change the detail of what you do then the usual maintainability
benefits of small well factored functions comes into play. That's why we
have functions in the language isn't it? Or do you just write one
long script?
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, March 2, 2018 at 5:29:54 AM UTC, Rustom Mody wrote: > Please excuse if this has been addressed above and/or its too basic: > What's the difference between RAII and python's with/context-managers? They address the same problem but I am claiming that RAII achieves this in a significantly more elegant/pythonic way without involving any special keywords or methods. in summary _if_ the PEP was adopted and/or you are using CPython today then:- def riaa_file_copy(srcname, dstname): src = RAIIFile(srcname, 'r') dst = RAIIFile(dstname, 'w') for line in src: dst.write(line) becomes equivalent to: def pep343_file_copy(srcname, dstname): with open(srcname, 'r') as src, open(dstname, 'w') as dst: for line in src: dst.write(line) RAII resource management is also simpler to implement only requiring existing __init__ and __del__ methods (e.g. to open/close the underlying file) and the resource objects are invariant. Which means the objects/managers do not need to track the enter/exit state - as there is no way to access them when they are not "open" in RAII. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Saturday, 3 March 2018 17:16:14 UTC, Ned Batchelder wrote:
> On 3/2/18 10:36 AM, Paul Moore wrote:
> > Or (real Python):
> >
> > def fn():
> > for i in range(1):
> > with open(f"file{i}.txt", "w") as f:
> > f.write("Some text")
> >
> > How would you write this in your RAII style - without leaving 10,000
> > file descriptors open until the end of the function?
>
> IIUC, if the OP's proposal were accepted, the __del__ method would be
> called as soon as the *value*'s reference count went to zero. That means
> this wouldn't leave 10,000 files open, since each open() would assign a
> new file object to f, which would make the previous file object's ref
> count be zero, and it would be closed.
You have understood correctly. Here is the equivalent RAII version:-
def fn():
for i in range(1):
f = RAIIFile(f"file{i}.txt", "w")
f.write("Some text")
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Saturday, 3 March 2018 17:44:08 UTC, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon > > Yes, stack allocated object in C++ have a nice lifetime to allow RAII to > > work, but it doesn't just work with stack allocated objects. A lot of RAII > > objects are members of a class object that may well be allocated on the > > heap, and RAII makes sure that all the needed cleanup gets done when that > > object gets destroyed. > > How do you guarantee that the heap object is properly disposed of when > you're done with it? Your RAII object depends 100% on the destruction > of the heap object. Smart pointers (unique_ptr and friends) are used to manage heap object lifecycles n . These are analogous to python object references. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Saturday, 3 March 2018 23:52:34 UTC, Steven D'Aprano wrote: > I know that laziness and hubris are programmer virtues, but there is > still such a thing as *too much laziness*. RAII works in C++ where > instances are allocated in the stack, but even there, if you have an > especially long-lived function, your resources won't be closed promptly. > In Python terms: > > def function(): > x = open_resource() > process(x) > # and we're done with x now, but too lazy to explicitly close it > sleep(1) # Simulate some more work. Lots of work. > return > # and finally x is closed (2.8 hours after you finished using it) > > The answer in C++ is "well don't do that then". The answer is Python is, > "don't be so lazy, just use a with statement". The answer in C++ would be to say "don't be so lazy just put the x manipulation in a function or sub-block". The answer with Python + this PEP would be "don't be so lazy just put the x manipulation in a function or explicitly del x" ...no new syntax. > If you want deterministic closing of resources, with statements are the > way to do it. > > def function(): > with open_resource() as x: > process(x) > # and x is guaranteed to be closed What a palava! -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote:
[snip]
> def fn():
> for i in range(1):
> with open(f"file{i}.txt", "w") as f:
> f.write("Some text")
>
> How would you write this in your RAII style - without leaving 10,000
> file descriptors open until the end of the function?
def fn():
for i in range(1):
f = RAIIFile(f"file{i}.txt", "w")
f.write("Some text")
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 01:58:02 UTC, Gregory Ewing wrote: > ooomzay wrote: > > Well he was not telling you the whole story: RAII works just as well with > > heap objects using smart pointers (unique_ptr and friends) which are a > > closer > > analogy to python object references. > > By that definition, *all* resource management in Python is > based on RAII[1]. I think not. Objects may have a close() in their __del__ method as a back-up - but currently this is not guaranteed to be called in a timely fashion. Hence the justification for "with" and my proposal to obviate it by actually guaranteeing that __del__ is always called in a timely fashion. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: > On Sat, 03 Mar 2018 18:19:37 -0800, Ooomzay wrote: > > >> def function(): > >> x = open_resource() > >> process(x) > >> # and we're done with x now, but too lazy to explicitly close it > >> sleep(1) # Simulate some more work. Lots of work. > >> return > >> # and finally x is closed (2.8 hours after you finished using it) > >> > >> The answer in C++ is "well don't do that then". The answer is Python > >> is, "don't be so lazy, just use a with statement". > > > > The answer in C++ would be to say "don't be so lazy just put the x > > manipulation in a function or sub-block". > > Right -- so you still have to think about resource management. The > premise of this proposal is that RAII means you don't have to think about > resource management, it just happens, but that's not really the case. That is not my main premise: Which is that RAII is a more elegant (no specialised syntax at all) and robust way to manage resources. Please consider the case of a composite resource: You need to implement __enter__, __exit__ and track the open/closed state at every level in your component hierarchy - even if some levels hold no resources directly. This is burdensome, breaks encapsulation, breaks invariance and is error prone ...very unpythonic. > > The answer with Python + this > > PEP would be "don't be so lazy just put the x manipulation in a function > > or explicitly del x" ...no new syntax. > > Sure -- but it doesn't gain you anything we don't already have. See above. > It imposes enormous burdens on the maintainers of at least five > interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which > will need to be re-written to have RAII semantics guaranteed; Yes. This is a substantial issue that will almost certainly see it rejected by HWCNBN on political, rather than linguistic, grounds. My PEP is about improving the linguistic integrity and fitness for resource management purpose of the language. > it will > probably have significant performance costs; and at the end of the day, > the benefit over using a with statement is minor. > > (Actually, I'm not convinced that there is *any* benefit. If anything, I > think it will be a reliability regression -- see below.) > > >> If you want deterministic closing of resources, with statements are the > >> way to do it. > >> > >> def function(): > >> with open_resource() as x: > >> process(x) > >> # and x is guaranteed to be closed > > > > What a palava! > > I don't know that word, and neither do any of my dictionaries. I think > the word you might mean is "pelaver"? I don't know that word, and neither do any of my dictionaries. I think the word you might mean is "Palaver"? Anyway Palava/Pelaver/Palaver/Palavra/Palabra derives from the word for "word", but in England it is often used idiomatically to mean a surfeit of words, or even more generally, a surfeit of effort. I intend it in both senses: The unnecessary addition of the words "with", "as", "__enter__" & "__exit__" to the language and the need implement the latter two methods all over the place. > In any case, you might not like with statements, but I think they're > infinitely better than: > > def meaningless_function_that_exists_only_to_manage_resource(): > x = open_resource() > process(x) > def function(): > meaningless_function_that_exists_only_to_manage_resource() > sleep(1) # simulate a long-running function Why would you prefer a new construct? Functions _are_ pythons scoping context! Giving one a pejorative name does not change that. > In other words, your solutions are just as much manual resource > management as the with statement. The only differences are: > > - instead of explicitly using a dedicated syntax designed for > resource management, you're implicitly using scope behaviour; Excellent: With the benefit of automatic, exception safe, destruction of resources, including composite resources. > - the with block is equivalent to a try...finally, and so it is > guaranteed to close the resource even if an exception occurs; > your solution isn't. > If process(x) creates a non-local reference to x, and then raises an > exception, and that exception is caught elsewhere, x will not go out of > scope and won't be closed. > A regression in the reliability of the code. This PEP does not affec
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: > [...] > [This PEP] imposes enormous burdens on the maintainers of at least five > interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which > will need to be re-written to have RAII semantics guaranteed; Not so:- CPython, the reference interpreter, already implements the required behaviour, as mentioned in the PEP. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 03:16:31 UTC, Paul Rubin wrote: > Chris Angelico writes: > > Yep, cool. Now do that with all of your smart pointers being on the > > heap too. You are not allowed to use ANY stack objects. ANY. Got it? > > That's both overconstraining and not even that big a problem the way you > phrase it. > > 1) Python has both the "with" statement and try/finally. Both of these > run code at the exit from a syntactically defined scope. So they are > like stack allocation in C++, where a deallocator can run when the scope > exits. > > 2) Even with no scope-based de-allocation, it's common to put smart > pointers into containers like lists and vectors. So you could have a > unique_ptr to a filestream object, and stash the unique_ptr someplace as > a vector element, where the vector itself could be part of some even > more deeply nested structure. At some point, the big structure gets > deleted (maybe through a manually-executed delete statement). When that > happens, if the nested structures are all standard containers full of > unique_ptrs, the top-level finalizer will end up traversing the entire > tree and freeing up the file handles and whatever else might be in > there. > > It occurs to me, maybe #2 above is closer to what the OP is really after > in Python. Yep. C++ smart pointers are a good analogue to python references for purposes of this PEP. > I guess it's doable, but refcounts don't seem like the right > way. Well refcounts are definitely "doable": This is how the reference python implementation, CPython, currently manages to comply with this PEP and can therefore be used for RAII. This PEP is an attempt to _guarantee_ this behaviour and make the elegance of RAII available to all pythonistas that want it. Without this guarantee python is not attractive to applications that must manage non-trivial resources reliably. Aside: I once read somewhere that must have seemed authoritative at the time, that CPython _guarantees_ to continue to behave like this - but now the subject is topical again I can find no trace of this guarantee. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 03:00:13 UTC, Chris Angelico wrote: > This thread is dead. The OP wants to wave a magic wand and say > "__del__ is now guaranteed to be called immediately", No "magic" required: Just one line change in the language reference will do it. > without any explanation The PEP says it all really: To make the very pythonic RAII idiom available in python. > - and, from the look of things, without any understanding > - of what that means for the language What impact on the _language_ (c.f. interpreter) do you think I have not understood? It is is 100% backwards compatible with the language. It breaks nothing. It allows people who want to use the RAII idiom to do so. It allows people who want to use the "with" idiom to continue do so. > and the interpreters. I am well aware of what it will mean for interpreters. For some interpreters it will have zero impact (e.g. CPython) and for some others it would unlikely be economic to make them comply. The decision here is does python want to be more pythonic, and make itself attractive for resource management applications or does it want to be compromised by some implementations? > Everyone else is saying "your magic wand is broken". This is not going to go > anywhere. Well I see a lot of posts that indicate peeps here are more comfortable with the "with" idiom than the RAII idiom but I have not yet seen a single linguistic problem or breakage. As it happens I have used RAII extensively with CPython to manage a debugging environment with complex external resources that need managing very efficiently. (I would use C++ if starting from scratch because it _guarantees_ the required deterministic destruction whereas python does not) -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: > Are you including cyclic references in your assertion that CPython > behaves as you want? Yes. Because the only behaviour required for RAII is to detect and debug such cycles in order to eliminate them. It is a design error/resource leak to create an orphan cycle containing RAII objects. def main(): gc,disable -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 15:24:08 UTC, Steven D'Aprano wrote:
> On Sun, 04 Mar 2018 03:37:38 -0800, Ooomzay wrote:
>
> > Please consider the case of a composite resource: You need to implement
> > __enter__, __exit__ and track the open/closed state at every level in
> > your component hierarchy - even if some levels hold no resources
> > directly.
> > This is burdensome, breaks encapsulation, breaks invariance and is error
> > prone ...very unpythonic.
>
> Without a more concrete example, I cannot comment on these claims.
Here is an example of a composite resource using RAII:-
class RAIIFileAccess():
def __init__(self, fname):
print("%s Opened" % fname)
def __del__(self):
print("%s Closed" % fname)
class A():
def __init__(self):
self.res = RAIIFileAccess("a")
class B():
def __init__(self):
self.res = RAIIFileAccess("b")
class C():
def __init__(self):
self.a = A()
self.b = B()
def main():
c = C()
Under this PEP this is all that is needed to guarantee that the files "a"
and "b" are closed on exit from main or after any exception has been handled.
Also note that if you have a reference to these objects then they are
guaranteed to be in a valid/useable/open state (invariant) - no danger
or need to worry/check about enter/exit state.
Now repeat this example with "with".
> [...]
> > My PEP is about improving the linguistic integrity and fitness for
> > resource management purpose of the language.
>
> So you claim, but *requiring* reference counting semantics does not
> improve the integrity or fitness of the language.
We will just have to disagree on that for now.
> And the required
> changes to programming styles and practices (no cycles,
No change required. But if you _choose_ to benefit from RAII you had better
not create orphan cycles with RAII objects in them, as that
is clearly a resource leak.
> no globals,
No change required. But if you _choose_ to benefit from RAII you had better
take care to delete any RAII resources you choose to hold at global scope in
a robust way. (These are exceptional in my experience).
> put all resources inside their own scope)
No change required. But if you _choose_ to benefit from RAII you can make use
of python's existing scopes (functions) or del to restrict resource lifetimes.
> >> In any case, you might not like with statements, but I think they're
> >> infinitely better than:
> >>
> >> def meaningless_function_that_exists_only_to_manage_resource():
> >> x = open_resource()
> >> process(x)
> >
> >> def function():
> >> meaningless_function_that_exists_only_to_manage_resource()
> >> sleep(1) # simulate a long-running function
> >
> > Why would you prefer a new construct?
>
> I don't prefer a new construct. The "with" statement isn't "new". It goes
> back to Python 2.5 (`from __future__ import with_statement`) which is
> more than eleven years old now. That's about half the lifetime of the
> language!
>
> I prefer the with statement because it is *explicit*, simple to use, and
> clear to read. I can read some code and instantly see that when the with
> block ends, the resource will be closed, regardless of how many
> references to the object still exist.
>
> I don't have to try to predict (guess!) when the last reference will go
> out of scope, because that's irrelevant.
If you don't care about what the other references might be then
RAII is not for you. Fine.
> RAII conflates the lifetime of the object with the lifetime of the
> resource held by the object.
This "conflation" is called "invariance" and is usually considered a
"very good thing" as you cant have references floating around to
half-baked resources.
> They are not the same, and the object can
> outlive the resource.
Not with RAII it can't. Simple. Good.
> Your position is:
>
> "RAII makes it really elegant to close the file! All you need to do is
> make sure that when you want to close the file, you delete all the
> references to the file, so that it goes out of scope, and the file will
> be closed."
>
> My position is:
>
> "If I want to close the file, I'll just close the file. Why should I care
> that there are zero or one or a million references to it?"
Because if you have no idea what references there are you can not assume it
is OK to close the file! That would be a truly terrible program design.
> >> - the with block is equivalent to a try...finally, and so it is
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 23:57:24 UTC, Mark Lawrence wrote:
> On 04/03/18 02:28, Ooomzay wrote:
> > On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote:
> > [snip]
> >> def fn():
> >> for i in range(1):
> >> with open(f"file{i}.txt", "w") as f:
> >> f.write("Some text")
> >>
> >> How would you write this in your RAII style - without leaving 10,000
> >> file descriptors open until the end of the function?
> >
> > def fn():
> > for i in range(1):
> > f = RAIIFile(f"file{i}.txt", "w")
> > f.write("Some text")
> >
> Over my dead body.
Care to expand on that?
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 23:55:33 UTC, Ned Batchelder wrote: > On 3/4/18 5:25 PM, Ooomzay wrote: > > On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: > >> Are you including cyclic references in your assertion that CPython > >> behaves as you want? > > Yes. Because the only behaviour required for RAII is to detect and debug > > such cycles in order to eliminate them. It is a design error/resource leak > > to create an orphan cycle containing RAII objects. > > > This isn't a reasonable position. Cycles exist, and the gc exists for a > reason. Your proposal isn't going to go anywhere if you just naively > ignore cycles. I am not naively ignoring them. But _if_ you want to use RAII then do not leak them in cycles. Put anything else in there you like and gc them as before. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Saturday, 3 March 2018 17:10:53 UTC, Dietmar Schwertberger wrote: > CPython does *not* guarantee destruction when the object reference goes > out of scope, even if there are no other references. > I would very much appreciate such a deterministic behaviour, at least > with CPython. > > I recently had to debug an issue in the matplotlib wx backend (*). Under > certain conditions, the wx device context was not destroyed when the > reference went out of scope. Adding a del to the end of the method or > calling the Destroy method of the context did fix the issue. (There was > also a hidden reference, but avoiding this was not sufficient. The del > was still required.) You say the reference was out of scope but that a del was still required. What were you delling if the reference was out of scope? Could you sketch the code. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 01:11:43 UTC, Richard Damon wrote: > On 3/4/18 6:55 PM, Ned Batchelder wrote: > > On 3/4/18 5:25 PM, Ooomzay wrote: > >> On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: > >>> Are you including cyclic references in your assertion that CPython > >>> behaves as you want? > >> Yes. Because the only behaviour required for RAII is to detect and > >> debug such cycles in order to eliminate them. It is a design > >> error/resource leak to create an orphan cycle containing RAII objects. > >> > >> def main(): > >> gc,disable > >> > >> > >> > > > > This isn't a reasonable position. Cycles exist, and the gc exists for > > a reason. Your proposal isn't going to go anywhere if you just > > naively ignore cycles. > > > > --Ned. > > While Ooomzay seems to want to say that all cycles are bad, I only want to say that orphan cycles with RAII objects in them are bad. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, March 5, 2018 at 6:38:49 AM UTC, Mark Lawrence wrote:
> On 05/03/18 01:01, Ooomzay wrote:
> > On Sunday, 4 March 2018 23:57:24 UTC, Mark Lawrence wrote:
> >> On 04/03/18 02:28, Ooomzay wrote:
> >>> On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote:
> >>> [snip]
> >>>> def fn():
> >>>> for i in range(1):
> >>>> with open(f"file{i}.txt", "w") as f:
> >>>> f.write("Some text")
> >>>>
> >>>> How would you write this in your RAII style - without leaving 10,000
> >>>> file descriptors open until the end of the function?
> >>>
> >>> def fn():
> >>> for i in range(1):
> >>> f = RAIIFile(f"file{i}.txt", "w")
> >>> f.write("Some text")
> >>>
> >
> >> Over my dead body.
> >
> > Care to expand on that?
> >
>
> Sure, when you state what you intend doing about reference cycles, which
> you've been asked about countless times.
Nothing. No change whatever. As I stated in my second post ref cycles are
orthogonal to this PEP.
If you want to use RAII objects then you will make sure you avoid adding them
to orphan cycles by design. If you don't know how to do that then don't write
applications that manage critical resources.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Sunday, 4 March 2018 23:56:09 UTC, Chris Angelico wrote:
> On Sun, Mar 4, 2018 at 10:37 PM, Ooomzay wrote:
> > Please consider the case of a composite resource: You need to implement
> > __enter__, __exit__ and track the open/closed state at every level in
> > your component hierarchy - even if some levels hold no resources directly.
> >
> > This is burdensome, breaks encapsulation, breaks invariance and is error
> > prone
> > ...very unpythonic.
>
> Why do you need to? I don't understand your complaint here - can you
> give an example of a composite resource that needs this kind of
> special management?
Here is an example of a composite resource using RAII:-
class RAIIFileAccess():
def __init__(self, fname):
print("%s Opened" % fname)
def __del__(self):
print("%s Closed" % fname)
class A():
def __init__(self):
self.res = RAIIFileAccess("a")
class B():
def __init__(self):
self.res = RAIIFileAccess("b")
class C():
def __init__(self):
self.a = A()
self.b = B()
def main():
c = C()
Under this PEP this is all that is needed to guarantee that the files "a"
and "b" are closed on exit from main or after any exception has been handled.
Also note that if you have a reference to these objects then they are
guaranteed to be in a valid/useable/open state (invariant) - no danger
or need to worry/check about enter/exit state.
Now repeat this exercise with "with".
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote:
> On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote:
> > Here is an example of a composite resource using RAII:-
> >
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > def __del__(self):
> > print("%s Closed" % fname)
> >
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> >
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> >
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >
> > def main():
> > c = C()
> >
> > Under this PEP this is all that is needed to guarantee that the files "a"
> > and "b" are closed on exit from main or after any exception has been
> > handled.
>
> Okay. And all your PEP needs is for reference count semantics, right?
> Okay. I'm going to run this in CPython, with reference semantics. You
> guarantee that those files will be closed after an exception is
> handled? Right.
>
> >>> def main():
> ... c = C()
> ... c.do_stuff()
> ...
> >>> main()
> a Opened
> b Opened
> Traceback (most recent call last):
> File "", line 1, in
> File "", line 3, in main
> AttributeError: 'C' object has no attribute 'do_stuff'
> >>>
>
> Uhh I'm not seeing any messages about the files getting closed.
Then that is indeed a challenge. From CPython back in 2.6 days up to
Python36-32 what I see is:-
a Opened
b Opened
Traceback (most recent call last):
...
AttributeError: 'C' object has no attribute 'dostuff'
a Closed
b Closed
> Maybe exceptions aren't as easy to handle as you think?
Well there is a general issue with exceptions owing to the ease
with which one can create cycles that may catch out newbs. But
that is not the case here.
> Or maybe you
> just haven't tried any of this (which is obvious from the bug in your
> code
Or maybe I just made a typo when simplifying my test case and failed to retest?
Here is my fixed case, if someone else could try it in CPython and report back
that would be interesting:-
class RAIIFileAccess():
def __init__(self, fname):
print("%s Opened" % fname)
self.fname = fname
def __del__(self):
print("%s Closed" % self.fname)
class A():
def __init__(self):
self.res = RAIIFileAccess("a")
class B():
def __init__(self):
self.res = RAIIFileAccess("b")
class C():
def __init__(self):
self.a = A()
self.b = B()
def main():
c = C()
c.dostuff()
main()
> You keep insisting that this is an easy thing. > We keep pointing out
> that it isn't. Now you're proving that you haven't even attempted any
> of this.
Nonsense. But you have got a result I have never seen in many years
and I would like to get to the bottom of it.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 13:59:35 UTC, Ooomzay wrote:
> On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote:
> > On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote:
> > > Here is an example of a composite resource using RAII:-
> > >
> > > class RAIIFileAccess():
> > > def __init__(self, fname):
> > > print("%s Opened" % fname)
> > > def __del__(self):
> > > print("%s Closed" % fname)
> > >
> > > class A():
> > > def __init__(self):
> > > self.res = RAIIFileAccess("a")
> > >
> > > class B():
> > > def __init__(self):
> > > self.res = RAIIFileAccess("b")
> > >
> > > class C():
> > > def __init__(self):
> > > self.a = A()
> > > self.b = B()
> > >
> > > def main():
> > > c = C()
> > >
> > > Under this PEP this is all that is needed to guarantee that the files "a"
> > > and "b" are closed on exit from main or after any exception has been
> > > handled.
> >
> > Okay. And all your PEP needs is for reference count semantics, right?
> > Okay. I'm going to run this in CPython, with reference semantics. You
> > guarantee that those files will be closed after an exception is
> > handled? Right.
> >
> > >>> def main():
> > ... c = C()
> > ... c.do_stuff()
> > ...
> > >>> main()
> > a Opened
> > b Opened
> > Traceback (most recent call last):
> > File "", line 1, in
> > File "", line 3, in main
> > AttributeError: 'C' object has no attribute 'do_stuff'
> > >>>
> >
> > Uhh I'm not seeing any messages about the files getting closed.
>
> Then that is indeed a challenge. From CPython back in 2.6 days up to
> Python36-32 what I see is:-
>
> a Opened
> b Opened
> Traceback (most recent call last):
> ...
> AttributeError: 'C' object has no attribute 'dostuff'
> a Closed
> b Closed
>
> > Maybe exceptions aren't as easy to handle as you think?
>
> Well there is a general issue with exceptions owing to the ease
> with which one can create cycles that may catch out newbs. But
> that is not the case here.
>
> > Or maybe you
> > just haven't tried any of this (which is obvious from the bug in your
> > code
>
> Or maybe I just made a typo when simplifying my test case and failed to
> retest?
>
> Here is my fixed case, if someone else could try it in CPython and report
> back that would be interesting:-
>
> class RAIIFileAccess():
> def __init__(self, fname):
> print("%s Opened" % fname)
> self.fname = fname
>
> def __del__(self):
> print("%s Closed" % self.fname)
>
> class A():
> def __init__(self):
> self.res = RAIIFileAccess("a")
>
> class B():
> def __init__(self):
> self.res = RAIIFileAccess("b")
>
> class C():
> def __init__(self):
> self.a = A()
> self.b = B()
>
> def main():
> c = C()
> c.dostuff()
>
> main()
>
> > You keep insisting that this is an easy thing. > We keep pointing out
> > that it isn't. Now you're proving that you haven't even attempted any
> > of this.
>
> Nonsense. But you have got a result I have never seen in many years
> and I would like to get to the bottom of it.
Ahah... I see now you are running it from a shell so the exception is staying
in scope. We just need to include normal exception handling in the example to
fix this:-
class RAIIFileAccess():
def __init__(self, fname):
print("%s Opened" % fname)
self.fname = fname
def __del__(self):
print("%s Closed" % self.fname)
class A():
def __init__(self):
self.res = RAIIFileAccess("A")
class B():
def __init__(self):
self.res = RAIIFileAccess("B")
class C():
def __init__(self):
self.a = A()
self.b = B()
def main():
try:
c = C()
c.dostuff()
except:
print("Boom!")
main()
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 14:36:30 UTC, Chris Angelico wrote:
> On Tue, Mar 6, 2018 at 1:25 AM, Ooomzay wrote:
> > Ahah... I see now you are running it from a shell so the exception is
> > staying in scope. We just need to include normal exception handling in the
> > example to fix this:-
> >
> > def main():
> > try:
> > c = C()
> > c.dostuff()
> > except:
> > print("Boom!")
> >
> > main()
> >
>
> So RAII depends on absorbing every exception close to where the
> resource is being managed? You can't permit that exception to bubble
> up lest the resource get leaked??
The exception can bubble up as many layers as you like but in python the
exception leaks out of the handling context and needs its scope limiting. I
have previously pointed out that such scoping is still recommended and that a
function is pythons scoping construct - so use them for the job.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 15:17:13 UTC, bartc wrote:
> On 05/03/2018 13:58, Ooomzay wrote:
> > On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote:
> >> On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote:
> >>> Here is an example of a composite resource using RAII:-
> >>>
> >>> class RAIIFileAccess():
> >>> def __init__(self, fname):
> >>> print("%s Opened" % fname)
> >>> def __del__(self):
> >>> print("%s Closed" % fname)
> >>>
> >>> class A():
> >>> def __init__(self):
> >>> self.res = RAIIFileAccess("a")
> >>>
> >>> class B():
> >>> def __init__(self):
> >>> self.res = RAIIFileAccess("b")
> >>>
> >>> class C():
> >>> def __init__(self):
> >>> self.a = A()
> >>> self.b = B()
> >>>
> >>> def main():
> >>> c = C()
> >>>
> >>> Under this PEP this is all that is needed to guarantee that the files "a"
> >>> and "b" are closed on exit from main or after any exception has been
> >>> handled.
> >>
> >> Okay. And all your PEP needs is for reference count semantics, right?
> >> Okay. I'm going to run this in CPython, with reference semantics. You
> >> guarantee that those files will be closed after an exception is
> >> handled? Right.
> >>
> >>>>> def main():
> >> ... c = C()
> >> ... c.do_stuff()
> >> ...
> >>>>> main()
> >> a Opened
> >> b Opened
> >> Traceback (most recent call last):
> >>File "", line 1, in
> >>File "", line 3, in main
> >> AttributeError: 'C' object has no attribute 'do_stuff'
> >>>>>
> >>
> >> Uhh I'm not seeing any messages about the files getting closed.
> >
> > Then that is indeed a challenge. From CPython back in 2.6 days up to
> > Python36-32 what I see is:-
> >
> > a Opened
> > b Opened
> > Traceback (most recent call last):
> > ...
> > AttributeError: 'C' object has no attribute 'dostuff'
> > a Closed
> > b Closed
> >
> >> Maybe exceptions aren't as easy to handle as you think?
> >
> > Well there is a general issue with exceptions owing to the ease
> > with which one can create cycles that may catch out newbs. But
> > that is not the case here.
> >
> >> Or maybe you
> >> just haven't tried any of this (which is obvious from the bug in your
> >> code
> >
> > Or maybe I just made a typo when simplifying my test case and failed to
> > retest?
> >
> > Here is my fixed case, if someone else could try it in CPython and report
> > back that would be interesting:-
> >
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > self.fname = fname
> >
> > def __del__(self):
> > print("%s Closed" % self.fname)
> >
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> >
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> >
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >
> > def main():
> > c = C()
> > c.dostuff()
> >
> > main()
>
> I get A and B closed messages when running on CPython 2.7 and 3.6, with
> the code run from a .py file. (I never use interactive mode.)
>
> But not when running on a PyPy version of 2.7 (however that is not CPython).
Thanks bartc, I have made the example more complete by adding an exception
scope - this means it works as designed - in any context. See my reply to Chris.
We do not expect this to work in PyPy.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 16:47:02 UTC, Steven D'Aprano wrote:
> On Sun, 04 Mar 2018 16:58:38 -0800, Ooomzay wrote:
>
> > Here is an example of a composite resource using RAII:-
> >
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > def __del__(self):
> > print("%s Closed" % fname)
> >
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> >
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> >
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >
> > def main():
> > c = C()
>
> Looking at that code, my major thought is that there is far too much OO
> design, not enough simplicity.
If this is far too much OO for you then RAII will be of no
interest to you.
This example was specifically in response to a request to
illustrate the relative simplicity of RAII in the case of a
composite (OO) resource.
> Perhaps I'm missing something, but I have no idea what benefit there is
> in that style of code over:
>
> with open('a') as a:
> with open('b') as b:
> process(a, b)
Encapsulation. Your application code is now managing details
that should be hidden in the object. This PEP and RAII are
unashamedly targeted at OO designs.
If you would like to have a shot at coding this without RAII,
but preserving the OO design, you will find that it is
considerably _simpler_ than the with/context manager approach.
> So long as you only process a or b inside the nested block, you are
> guaranteed that they will be open.
>
> And unlike your RAII example, they will be closed when you exit,
> regardless of how many references to them you have, regardless of whether
> an exception occurs or not, regardless of whether there are cycles or
> whether they are globals or whether the interpreter is shutting down.
If you choose RAII you will not be cavalier with your references.
> I think that at this point, you have convinced me that you want to impose
> enormous costs on all Python interpreters *and* Python developers, in
> order to allow you to write C++ code in Python rather than learn Pythonic
> idioms like the with statement.
On interpreters yes. On developers no. You can carry on exactly as you are.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico wrote:
> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote:
> > Here is my fixed example, if someone else could try it in CPython and
> > report back that would be interesting:-
> >
> > class RAIIFileAccess():
> > def __init__(self, fname):
> > print("%s Opened" % fname)
> > self.fname = fname
> >
> > def __del__(self):
> > print("%s Closed" % self.fname)
> >
> > class A():
> > def __init__(self):
> > self.res = RAIIFileAccess("a")
> >
> > class B():
> > def __init__(self):
> > self.res = RAIIFileAccess("b")
> >
> > class C():
> > def __init__(self):
> > self.a = A()
> > self.b = B()
> >
> > def main():
> > c = C()
> > c.dostuff()
> >
> > main()
>
> Here's how I'd do it with context managers.
>
> from contextlib import contextmanager
>
> @contextmanager
> def file_access(fname):
> try:
> print("%s Opened" % fname)
> yield
> finally:
> print("%s Closed" % fname)
>
> @contextmanager
> def c():
> try:
> print("Starting c")
> with file_access("a") as a, file_access("b") as b:
> yield
> finally:
> print("Cleaning up c")
>
> def main():
> with c():
> dostuff() # NameError
Thank you for having a go...
However you have broken the encapsulation of class A and B. These
are trivial for the sake of example. I should have used
_underscores (i.e. self._res) to make the intent of
this example more obvious.
Please try again but preserving the integrity/encapsulation
of class A & B & C, just as the RAII example does.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 17:58:40 UTC, Chris Angelico wrote:
> On Tue, Mar 6, 2018 at 4:53 AM, Ooomzay wrote:
> > On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico wrote:
> >> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote:
> >> > Here is my fixed example, if someone else could try it in CPython and
> >> > report back that would be interesting:-
> >> >
> >> > class RAIIFileAccess():
> >> > def __init__(self, fname):
> >> > print("%s Opened" % fname)
> >> > self.fname = fname
> >> >
> >> > def __del__(self):
> >> > print("%s Closed" % self.fname)
> >> >
> >> > class A():
> >> > def __init__(self):
> >> > self.res = RAIIFileAccess("a")
> >> >
> >> > class B():
> >> > def __init__(self):
> >> > self.res = RAIIFileAccess("b")
> >> >
> >> > class C():
> >> > def __init__(self):
> >> > self.a = A()
> >> > self.b = B()
> >> >
> >> > def main():
> >> > c = C()
> >> > c.dostuff()
> >> >
> >> > main()
> >>
> >> Here's how I'd do it with context managers.
> >>
> >> from contextlib import contextmanager
> >>
> >> @contextmanager
> >> def file_access(fname):
> >> try:
> >> print("%s Opened" % fname)
> >> yield
> >> finally:
> >> print("%s Closed" % fname)
> >>
> >> @contextmanager
> >> def c():
> >> try:
> >> print("Starting c")
> >> with file_access("a") as a, file_access("b") as b:
> >> yield
> >> finally:
> >> print("Cleaning up c")
> >>
> >> def main():
> >> with c():
> >> dostuff() # NameError
> >
> >
> > Thank you for having a go...
> >
> > However you have broken the encapsulation of class A and B. These
> > are trivial for the sake of example. I should have used
> > _underscores (i.e. self._res) to make the intent of
> > this example more obvious.
> >
> > Please try again but preserving the integrity/encapsulation
> > of class A & B & C, just as the RAII example does.
>
> What is B? Is it something that's notionally a resource to be managed?
Yes. For example a supply of electrical power controlled via a serial protocol
to a programmable power supply - the file is a private detail used for the
communications. And lets imagine that this powersupply object needs to keep
track of some state such as the voltage - and it has a long lifetime - not just
created then destroyed in scope of one function i.e. it is a substantial object.
> If so, you can trivially add another level to the nesting.
Please illustrate. I really do want to be able to compare like for like.
--
https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 19:14:05 UTC, Paul Rubin wrote: > Ooomzay writes: > > If you want to use RAII objects then you will make sure you avoid > > adding them to orphan cycles by design. If you don't know how to do > > that then don't write applications that manage critical resources. > > My claim is managing critical resources with refcounts is bug-prone in > the case where the refcounts can become arbitrarily large at runtime. What case is that? Don't go there! You may be jaded because python forgives bad design practices and peeps are inclined to leak resources all over the place for no good reason whatever and then complain when they have trouble cleaning up or exiting! > You say you wrote a program that worked that way, but it sounds > horrendous and I'd like to know how you tested it, maintained it, kept > it maintainable by other programmers, etc. > It's painful to even think about. I too used to suffer pain with python - until I saw that light and worked out how to use it for RAII. So here's the keys for much less pain:- * Use CPython (or C++ ;) * Use RAII for every resource-holding class: Implement __del__ to release any resources acquired in __init__. (This is significantly less effort, and more reliable than adding __enter__ & __exit__ to every class). * Wrap your try-except blocks in functions to prevent exceptions persisting outside the handler because in python they leak. This is typically a very natural factorization. * Take care not to create Cyclic Exceptions in your Exception handling logic. This was the one that had me scratching my head for a couple of hours the first time I inadvertantly created a cycle. * If you have no application-level requirement for persistent orphan cycles, and few, if any, applications do, then disable gc on entry and raise an exception on exit, or any other convenient moment if there is any garbage to be collected. * Immediately plug/bug any leaks that you discover. Do not let them build up or you will drown and loose faith that there can be a better way. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction
On Monday, 5 March 2018 23:06:53 UTC, Steven D'Aprano wrote: > On Mon, 05 Mar 2018 09:22:33 -0800, Ooomzay wrote: > [...] > > If you would like to have a shot at coding this without RAII, but > > preserving the OO design, you will find that it is considerably > > simpler than the with/context manager approach. > > Preserving the OO design, you say? Okay, since my application apparently > isn't allowed to know that it is processing two files, I'll simply > delegate that to the object: > > class C(A, B): > def run(self): > with open(self.a) as a: > with open(self.b) as b: > process(a, b) > > # Later. > if __name__ = '__main__': > # Enter the Kingdom of Nouns. > c = C() > c.run() > > There you go. Encapsulation and OO design. 1. This does not execute. It is only when you actually flesh out these deliberately minimal classes A, B & C with PEP232 paraphernalia that we all be able to see clearly how much cleaner or messier things are with PEP232. 2. Your Class C breaks A & B's encapsulation by inheriting rather than composing them. Apart from increasing coupling and preventing substitution, C contained A & B in this example to illustrate that there is no RAII related burden whatever_on this intermediate class as it manages no external resources directly. It does not even need to implement __del__. 3. You have assumed a single threaded application. Please imagine that A and B are classes managing some remote valves and maintain their own threads as well as a serial port for comms. And C is there to coordinate them towards some greater purpose. If you would like to try and add PEP232 support to classes A,B & C to the point that you can create and destroy c = C() in an exception-safe way we may all learn something. > I'm not trying to dissuade you from using RAII in your own applications, > if it works for you, great. Unfortunately, despite having conquered it, without a _guarantee_ of this behaviour from the language, or at least one mainstream implementation, I will not invest in python again. Nor recommend any one else with a serious real world resource management application to do so. This was the original motive for my PEP. -- https://mail.python.org/mailman/listinfo/python-list
Re: Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction
On Tuesday, 6 March 2018 14:12:38 UTC, Peter Otten wrote: > Chris Angelico wrote: > > > On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano > > wrote: > >> # Later. > >> if __name__ = '__main__': > >> # Enter the Kingdom of Nouns. > > > > Don't you need a NounKingdomEnterer to do that for you? > > No, for some extra flexibility there should be a NounKingdomEntererFactory For example open(). > -- which of course has to implement the AbstractNounKingdomEntererFactory > interface. For example a PEP 343 "Context Manager". In the swamplands of the pythonistas the one-eyed man is BDFL! Amen. -- https://mail.python.org/mailman/listinfo/python-list
Re: Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction
On Wednesday, 7 March 2018 06:43:10 UTC, Ooomzay wrote: > On Tuesday, 6 March 2018 14:12:38 UTC, Peter Otten wrote: > > Chris Angelico wrote: > > > > > On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano > > > wrote: > > >> # Later. > > >> if __name__ = '__main__': > > >> # Enter the Kingdom of Nouns. > > > > > > Don't you need a NounKingdomEnterer to do that for you? > > > > No, for some extra flexibility there should be a NounKingdomEntererFactory > > For example open(). > > > -- which of course has to implement the AbstractNounKingdomEntererFactory > > interface. > > For example a PEP 343 "Context Manager". > > In the swamplands of the pythonistas the one-eyed man is BDFL! > > Amen. Damn. That should have read:- In the swamplands of the pythonistas the man with a badly leaking boat is BDFL. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited)
On Thursday, 1 March 2018 22:44:59 UTC, Rob Gaddi wrote: > On 03/01/2018 02:24 PM, Lawrence D’Oliveiro wrote: > > On Thursday, March 1, 2018 at 6:44:39 PM UTC+13, Paul Rubin wrote: > >> DOM trees are a classic example (see the various DOM modules in the > >> Python stdlib). Non-leaf nodes have a list of child nodes, child nodes > >> have pointers back upwards to their parent, and each child node has > >> pointers to its left and right siblings if they exist. It's all very > >> circular. > > > > This is why you have weak refs. > > > But making sure that the failure to use weakrefs doesn't memory leak the > program into the pavement is why you have background garbage collection > with intelligent cyclical reference breaking. That is the worst justification for gc I have seen yet, and also the most truthful. gc.set_debug(gc.DEBUG_LEAK) is your friend my friends. -- https://mail.python.org/mailman/listinfo/python-list
