Re: Typing on child class' methods of a Generic base class
On 2022-03-10 12:31, Dieter Maurer wrote:
Nicolas Haller wrote at 2022-3-9 10:53 -0500:
...
The documentation about "user-defined generic types"[1] says that I can
fix some types on a child class (class MyDict(Mapping[str, T]):) but
doesn't say much about the signature of the methods I need to
implement/override on that child class.
I have the fealing that this is a case of (generic type) "specialization".
In this setup, `Mapping` would be a generic type with two type
variables `K` (the key type) and `V` (the value type).
The signatures of its methods would use those type variables.
In your example, you specialize the key type to `str` (and
leave the value type generic). The signatures of the methods
would automatically follow this specialization -- without the need
to do anything.
If I understand correctly, you're saying I should use the first
alternative by keeping the signature as it is in the base class:
---
T = TypeVar("T")
class Base(Generic[T], metaclass=ABCMeta):
"""A base class."""
@abstractmethod
def _private_method(self, an_arg: T) -> T:
...
def public_method(self, an_arg: T) -> T:
from_private_method = self._private_method(an_arg)
return from_private_method
class Alternative1(Base[int]):
def _private_method(self, an_arg: T) -> T: # I keep T
return 42
---
The problem with it is that mypy doesn´t seem quite happy with it:
./scratch.py:22: error: Incompatible return value type (got "int",
expected "T")
Do you think this is error is incorrect? If so, I can open an issue with
mypy.
Thanks,
--
Nicolas
--
https://mail.python.org/mailman/listinfo/python-list
Re: Typing on child class' methods of a Generic base class
Nicolas Haller wrote at 2022-3-12 12:05 -0500:
>On 2022-03-10 12:31, Dieter Maurer wrote:
>> Nicolas Haller wrote at 2022-3-9 10:53 -0500:
>>> ...
>>> The documentation about "user-defined generic types"[1] says that I can
>>> fix some types on a child class (class MyDict(Mapping[str, T]):) but
>>> doesn't say much about the signature of the methods I need to
>>> implement/override on that child class.
>>
>> I have the fealing that this is a case of (generic type) "specialization".
>> In this setup, `Mapping` would be a generic type with two type
>> variables `K` (the key type) and `V` (the value type).
>> The signatures of its methods would use those type variables.
>> In your example, you specialize the key type to `str` (and
>> leave the value type generic). The signatures of the methods
>> would automatically follow this specialization -- without the need
>> to do anything.
>
>If I understand correctly, you're saying I should use the first
>alternative by keeping the signature as it is in the base class:
>---
>T = TypeVar("T")
>
>
>class Base(Generic[T], metaclass=ABCMeta):
> """A base class."""
>
> @abstractmethod
> def _private_method(self, an_arg: T) -> T:
> ...
>
> def public_method(self, an_arg: T) -> T:
> from_private_method = self._private_method(an_arg)
> return from_private_method
>
>class Alternative1(Base[int]):
> def _private_method(self, an_arg: T) -> T: # I keep T
> return 42
>---
>
>The problem with it is that mypy doesn´t seem quite happy with it:
>./scratch.py:22: error: Incompatible return value type (got "int",
>expected "T")
>
>Do you think this is error is incorrect?
No, I do not think so.
The `Base[int]` means that the generic type variable `T` was
specialized to `int`. This means that in this context
`_private_method` returns `int` (not a generic `T`).
You either do not type annotate the overriding methods
or you use the specialization (if any).
--
https://mail.python.org/mailman/listinfo/python-list
Best practice for caching hash
I have a custom immutable object, and I added a cache for its hash value. The problem is the object can be composed of mutable or immutable objects, so the hash can raise TypeError. In this case I currently cache the value -1. The subsequent calls to __hash__() will check if the value is -1. If so, a TypeError is immediately raised. The problem is the first time I get an error with details, for example: TypeError: unhashable type: 'list' The subsequent times I simply raise a generic error: TypeError Ok, I can improve it by raising, for example, TypeError: not all values are hashable. But do you think this is acceptable? Now I'm thinking about it and it seems a little hacky to me. Furthermore, in the C extension I have to define another property in the struct, ma_hash_calculated, to track if the hash value is cached or not, since there's no bogus value I can use in cache property, ma_hash, to signal this. If I don't cache unhashable values, -1 can be used to signal that ma_hash contains no cached value. So if I do not cache if the object is unhashable, I save a little memory per object (1 int) and I get a better error message every time. On the other hand, if I leave the things as they are, testing the unhashability of the object multiple times is faster. The code: try: hash(o) except TypeError: pass execute in nanoseconds, if called more than 1 time, even if o is not hashable. Not sure if this is a big advantage. What do you think about? Here is the python code: https://github.com/Marco-Sulla/python-frozendict/blob/35611f4cd869383678104dc94f82aa636c20eb24/frozendict/src/3_10/frozendictobject.c#L652-L697 -- https://mail.python.org/mailman/listinfo/python-list
Problem slicing a list with the C API
I have a C API project where I have to slice a list into two parts.
Unfortunately the documentation on the slice objects is not clear enough for me
to understand how to do this, and I haven’t found enough useful info through
research. The list contains tuple records where each tuple consists of a
dictionary object and a string.
The relevant part of the Python code is:
half_slice = int(len(dictdata) * 0.5)
subdata_a = dictdata[half_slice:]
subdata_b = dictdata[:half_slice]
This is what I’ve done so far with the C API:
int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count)
{
long round_half = records_count * 0.5;
PyObject* half_slice = PyLong_FromLong(round_half);
PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0);
PyObject* subdata_a = PyObject_GetItem(pDictddata, slice);
return 0;
}
On the final line (subdata_a) I get a segfault. I know that the second
parameter of PyObject_GetItem is a “key” and I suspect that’s where the
problem comes from, but I don’t understand what a key is in this context.
The code shown above omits error handling but none of the objects leading up to
the final line is null, they all succeed.
Thanks for any ideas.
Jen
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best practice for caching hash
On 2022-03-12 at 21:45:56 +0100, Marco Sulla wrote: [ ... ] > So if I do not cache if the object is unhashable, I save a little > memory per object (1 int) and I get a better error message every time. > On the other hand, if I leave the things as they are, testing the > unhashability of the object multiple times is faster. The code: > > try: > hash(o) > except TypeError: > pass > > execute in nanoseconds, if called more than 1 time, even if o is not > hashable. Not sure if this is a big advantage. Once hashing an object fails, why would an application try again? I can see an application using a hashable value in a hashable situation again and again and again (i.e., taking advantage of the cache), but what's the use case for *repeatedly* trying to use an unhashable value again and again and again (i.e., taking advantage of a cached failure)? So I think that caching the failure is a lot of extra work for no benefit. -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
On 2022-03-12 21:24, Jen Kris via Python-list wrote:
I have a C API project where I have to slice a list into two parts.
Unfortunately the documentation on the slice objects is not clear enough for me
to understand how to do this, and I haven’t found enough useful info through
research. The list contains tuple records where each tuple consists of a
dictionary object and a string.
The relevant part of the Python code is:
half_slice = int(len(dictdata) * 0.5)
subdata_a = dictdata[half_slice:]
subdata_b = dictdata[:half_slice]
This is what I’ve done so far with the C API:
int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count)
{
long round_half = records_count * 0.5;
PyObject* half_slice = PyLong_FromLong(round_half);
PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0);
PyObject* subdata_a = PyObject_GetItem(pDictddata, slice);
return 0;
}
On the final line (subdata_a) I get a segfault. I know that the second
parameter of PyObject_GetItem is a “key” and I suspect that’s where the
problem comes from, but I don’t understand what a key is in this context.
The code shown above omits error handling but none of the objects leading up to
the final line is null, they all succeed.
Thanks for any ideas.
Use PySequence_GetSlice to slice the list.
Also, why use floats when you can use integers?
long round_half = records_count / 2;
(In Python that would be half_slice = len(dictdata) // 2.)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
On Sun, 13 Mar 2022 at 08:25, Jen Kris via Python-list wrote: > PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); > PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); > > On the final line (subdata_a) I get a segfault. I know that the second > parameter of PyObject_GetItem is a “key” and I suspect that’s where the > problem comes from, but I don’t understand what a key is in this context. > The key is simply whatever would be in the square brackets in Python code, so that part looks fine. But dictionaries aren't usually subscripted with slices, so I'm a bit confused as to what's going on here. What exactly is dictdata/pDictdata? Have you confirmed that pDictdata (a) isn't NULL, (b) is the object you intend it to be, and (c) contains the objects you expect it to? The segfault might not be from the slice object itself, it might be from actually iterating over the thing being sliced and touching all its elements. For instance, if dictdata is actually a list, that call will be constructing a new list with references to the same elements, so if one of them is broken (maybe NULL), it'll break badly. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
Thanks to you both. I am going to implement PySequence_Get_Slice now. If I
have trouble then, per comments from Chris Angelico, I will iterate through
pDictData to verify it because I haven't done that. It is not null, however.
Jen
Mar 12, 2022, 13:40 by pyt...@mrabarnett.plus.com:
> On 2022-03-12 21:24, Jen Kris via Python-list wrote:
>
>> I have a C API project where I have to slice a list into two parts.
>> Unfortunately the documentation on the slice objects is not clear enough for
>> me to understand how to do this, and I haven’t found enough useful info
>> through research. The list contains tuple records where each tuple consists
>> of a dictionary object and a string.
>>
>> The relevant part of the Python code is:
>>
>> half_slice = int(len(dictdata) * 0.5)
>> subdata_a = dictdata[half_slice:]
>> subdata_b = dictdata[:half_slice]
>>
>> This is what I’ve done so far with the C API:
>>
>> int64_t Calc_Slices(PyObject* pDictdata, int64_t records_count)
>> {
>> long round_half = records_count * 0.5;
>> PyObject* half_slice = PyLong_FromLong(round_half);
>>
>> PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0);
>> PyObject* subdata_a = PyObject_GetItem(pDictddata, slice);
>>
>> return 0;
>> }
>>
>> On the final line (subdata_a) I get a segfault. I know that the second
>> parameter of PyObject_GetItem is a “key” and I suspect that’s where the
>> problem comes from, but I don’t understand what a key is in this context.
>>
>> The code shown above omits error handling but none of the objects leading up
>> to the final line is null, they all succeed.
>>
>> Thanks for any ideas.
>>
> Use PySequence_GetSlice to slice the list.
>
> Also, why use floats when you can use integers?
>
> long round_half = records_count / 2;
>
> (In Python that would be half_slice = len(dictdata) // 2.)
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a dictionary object and a string. Mar 12, 2022, 13:41 by ros...@gmail.com: > On Sun, 13 Mar 2022 at 08:25, Jen Kris via Python-list > wrote: > >> PyObject* slice = PySlice_New(PyLong_FromLong(0), half_slice, 0); >> PyObject* subdata_a = PyObject_GetItem(pDictddata, slice); >> >> On the final line (subdata_a) I get a segfault. I know that the second >> parameter of PyObject_GetItem is a “key” and I suspect that’s where the >> problem comes from, but I don’t understand what a key is in this context. >> > > The key is simply whatever would be in the square brackets in Python > code, so that part looks fine. > > But dictionaries aren't usually subscripted with slices, so I'm a bit > confused as to what's going on here. What exactly is > dictdata/pDictdata? > > Have you confirmed that pDictdata (a) isn't NULL, (b) is the object > you intend it to be, and (c) contains the objects you expect it to? > The segfault might not be from the slice object itself, it might be > from actually iterating over the thing being sliced and touching all > its elements. For instance, if dictdata is actually a list, that call > will be constructing a new list with references to the same elements, > so if one of them is broken (maybe NULL), it'll break badly. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
On Sun, 13 Mar 2022 at 08:54, Jen Kris wrote: > > > pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a > dictionary object and a string. > Ah, gotcha. In that case, yeah, slicing it will involve referencing the tuples all the way down the line (adding to their refcounts, so if there's a borked one, kaboom). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
Chris, you were right to focus on the list pDictData itself. As I said, that is a list of 2-tuples, but I added each of the 2-tuples with PyList_Append, but you can only append a tuple to a list with the extend method. However, there is no append method in the C API as far as I can tell -- hence pDictData is empty. I tried with PyList_SetItem but that doesn't work. Do you know of way to "extend" a list in the C API. Thanks very much. Jen Mar 12, 2022, 13:57 by ros...@gmail.com: > On Sun, 13 Mar 2022 at 08:54, Jen Kris wrote: > >> >> >> pDictData, despite the name, is a list of 2-tuples where each 2-tuple is a >> dictionary object and a string. >> > > Ah, gotcha. In that case, yeah, slicing it will involve referencing > the tuples all the way down the line (adding to their refcounts, so if > there's a borked one, kaboom). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
On Sun, 13 Mar 2022 at 10:30, Jen Kris wrote: > > > Chris, you were right to focus on the list pDictData itself. As I said, > that is a list of 2-tuples, but I added each of the 2-tuples with > PyList_Append, but you can only append a tuple to a list with the extend > method. However, there is no append method in the C API as far as I can tell > -- hence pDictData is empty. I tried with PyList_SetItem but that doesn't > work. Do you know of way to "extend" a list in the C API. > Hmm. Not entirely sure I understand the question. In Python, a list has an append method, which takes any object (which may be a tuple) and adds that object to the end of the list: >>> x = ["spam", "ham"] >>> x.append((1,2)) >>> x ['spam', 'ham', (1, 2)] A list also has an extend method, which takes any sequence (that also includes tuples), and adds *the elements from it* to the end of the list: >>> x = ["spam", "ham"] >>> x.extend((1,2)) >>> x ['spam', 'ham', 1, 2] The append method corresponds to PyList_Append, as you mentioned. It should be quite happy to append a tuple, and will add the tuple itself, not the contents of it. So when you iterate over the list, you'll get tuples. Extending a list can be done with the sequence API. In Python, you can write extend() as +=, indicating that you're adding something onto the end: >>> x = ["spam", "ham"] >>> x += (1, 2) >>> x ['spam', 'ham', 1, 2] This corresponds to PySequence_InPlaceConcat, so if that's the behaviour you want, that would be the easiest way to do it. Based on your other comments, I would suspect that appending the tuples is probably what you want here? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
Thanks for PySequence_InPlaceConcat, so when I need to extend I'll know what to use. But my previous email was based on incorrect information from several SO posts that claimed only the extend method will work to add tuples to a list. I found that's wrong -- even my own Python code uses the append method. But my PyList_Append is not doing the job so that's where I'm looking now. Thanks very much for your reply. Mar 12, 2022, 15:36 by ros...@gmail.com: > On Sun, 13 Mar 2022 at 10:30, Jen Kris wrote: > >> >> >> Chris, you were right to focus on the list pDictData itself. As I said, >> that is a list of 2-tuples, but I added each of the 2-tuples with >> PyList_Append, but you can only append a tuple to a list with the extend >> method. However, there is no append method in the C API as far as I can >> tell -- hence pDictData is empty. I tried with PyList_SetItem but that >> doesn't work. Do you know of way to "extend" a list in the C API. >> > > Hmm. Not entirely sure I understand the question. > > In Python, a list has an append method, which takes any object (which > may be a tuple) and adds that object to the end of the list: > x = ["spam", "ham"] x.append((1,2)) x > ['spam', 'ham', (1, 2)] > > A list also has an extend method, which takes any sequence (that also > includes tuples), and adds *the elements from it* to the end of the > list: > x = ["spam", "ham"] x.extend((1,2)) x > ['spam', 'ham', 1, 2] > > The append method corresponds to PyList_Append, as you mentioned. It > should be quite happy to append a tuple, and will add the tuple > itself, not the contents of it. So when you iterate over the list, > you'll get tuples. > > Extending a list can be done with the sequence API. In Python, you can > write extend() as +=, indicating that you're adding something onto the > end: > x = ["spam", "ham"] x += (1, 2) x > ['spam', 'ham', 1, 2] > > This corresponds to PySequence_InPlaceConcat, so if that's the > behaviour you want, that would be the easiest way to do it. > > Based on your other comments, I would suspect that appending the > tuples is probably what you want here? > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Problem slicing a list with the C API
On Sun, 13 Mar 2022 at 10:41, Jen Kris wrote: > > > Thanks for PySequence_InPlaceConcat, so when I need to extend I'll know what > to use. But my previous email was based on incorrect information from > several SO posts that claimed only the extend method will work to add tuples > to a list. I found that's wrong -- even my own Python code uses the append > method. But my PyList_Append is not doing the job so that's where I'm > looking now. > Ah. I guess that's one of the fundamental vulnerabilities of Stack Overflow: people answer the question in front of them, which isn't necessarily the question that YOU are trying to answer. Sometimes, the solutions can be misleading, because your goal is actually different. Fortunately, tuples in Python are objects, like every other value. It's extremely convenient - to word that another way, it's extremely INconvenient to work with a language where that isn't the case, and you need to work differently depending on whether you're using integers or strings. This is how you work with a trie (actually an artifact name now, as it's implemented with a hashtable) in SourceMod - this is the equivalent of dictionary subscript assignment: https://sm.alliedmods.net/new-api/adt_trie/SetTrieValue - works for integers, floats, and handles (which are integers) https://sm.alliedmods.net/new-api/adt_trie/SetTrieString - works for strings Yup, it's annoying :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
