On Wed, Mar 21, 2018 at 3:22 AM, Tom Evans via Python-list <python-list@python.org> wrote: > Hi all > > I'm writing my first C extension for Python here, and all is going > well. However, I was reading [1], and the author there is advocating > Py_INCREF 'ing *every* borrowed reference. > > Now, I get that if I do something to mutate and perhaps invalidate the > PyObject that was borrowed I can get unpredictable results - eg, by > getting a borrowed reference to a value from a dictionary, clearing > the dictionary and then expecting to use the borrowed reference. > > However, if my code does not do things like that, is there a chance of > a borrowed reference being invalidated that is not related to my use > of the thing I borrowed it from? Is this cargo cult advice (sometimes > the gods need this structure, and so to please the gods it must be > everywhere), sensible belt and braces or needless overkill?
Yep, that's cargo cult advice. If it were good advice, the Python API would have been built to never return borrowed references, so you always have to decref everything. You should incref those sorts of borrowed references if you're going to do something that could call arbitrary Python code and then continue using the reference; but most of the time, that borrowed ref is going to be used immediately and then finished with, and in those very common situations, it's pointless to incref and decref. It's also perfectly safe to use borrowed refs with immutable objects that you have refs to; for instance, you can inspect the elements of a tuple that way. The tuple MUST be holding references to all its members, and the tuple itself can't be disposed of while you have a reference to it, so you have a guarantee that those borrowed refs are valid. """The analogy in C is having two pointers to the same memory location: so who is responsible for freeing the memory and what happens if the other pointer tries to access that free’d memory?""" That ignores the entire point of reference counting. By using a borrowed reference, you're clearly declaring that the OTHER guy is responsible for freeing it. From footnote 3: """Of course we never remove items in a list we merely decrement their reference count (and if that hits zero then they are deleted).""" This is conflating two separate concepts. You most assuredly CAN remove items from a list, and the code in the footnote does exactly that. What it won't necessarily do is destroy those objects. """An important takeaway here is that incrementing and decrementing reference counts is a cheap operation but the consequences of getting it wrong can be expensive. A precautionary approach in your code might be to always increment borrowed references when they are instantiated and then always decrement them before they go out of scope. That way you incur two cheap operations but eliminate a vastly more expensive one.""" Actually, managing ref counts isn't always cheap. So instead of just always incrementing and always decrementing, how about this: If you are going to call arbitrary code that could do anything at all, guard around THAT CODE and nothing else, by protecting any borrowed references. People talk a lot about calling functions that could do literally anything, but honestly, how often does that even happen? Not often enough to justify cargo cult advice. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
