Validators and django.contrib.postgres.fields JSONField

[Ryan Causey]

Hello,

I've stumbled upon an issue when using JSONField from 
django.contrib.postgres.fields where I can't get it to run my additional 
validators when calling full_clean(). To show a simplified example:

validators.py:

@deconstructible
class TestValidator(object):
def __call__(self, foo):
raise ValidationError(message = _("Expected Error"))


models.py:

class DummyModel(models.Model):
dummyJson = JSONField(validators = [TestValidator])


shell output:

>>> from Dummy.models import DummyModel
>>> import json
>>> dummyJsonInput = json.loads("""{"foo" : "bar"}""")
>>> dummyModelInstance = DummyModel(dummyJson = dummyJsonInput)
>>> dummyModelInstance.full_clean()
>>>


Am I doing something wrong? I've read some documentation about needing to 
define __eq__ for migration purposes, but I tried defining it and it didn't 
seem to make a difference.

Any help is appreciated.

Thanks,
-Ryan Causey

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5f87cf69-ea62-44f2-a79c-0b6915d184bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Validators and django.contrib.postgres.fields JSONField

[Ryan Causey]

So I've dug into this a little more, and I've come up with the debugger 
trace below.

-> dummyModelInstance.full_clean()
  c:\program files\python35\lib\site-packages\django\db\models\base.py(1210)
full_clean()
-> self.clean_fields(exclude=exclude)
  c:\program files\python35\lib\site-packages\django\db\models\base.py(1252)
clean_fields()
-> setattr(self, f.attname, f.clean(raw_value, self))
  c:\program files\python35\lib\site-packages\django\db\models\fields\
__init__.py(592)clean()
-> self.run_validators(value)
  c:\program files\python35\lib\site-packages\django\db\models\fields\
__init__.py(544)run_validators()
-> v(value)
> c:\program files\python35\lib\site-packages\django\utils\deconstruct.py(16
)__new__()
-> def __new__(cls, *args, **kwargs):
(Pdb) ll
 16  -> def __new__(cls, *args, **kwargs):
 17 # We capture the arguments to make returning them 
trivial
 18 obj = super(klass, cls).__new__(cls)
 19 obj._constructor_args = (args, kwargs)
 20 return obj

For some reason, when run_validators() calls the custom TestValidator, it 
seems that a TestValidator object is constructed rather than entering the 
__call__() method of the class. I can't see a reason for this to happen, so 
what am I doing wrong here?

Thanks,
-Ryan Causey

On Sunday, August 21, 2016 at 9:32:50 PM UTC-7, Ryan Causey wrote:
>
> Hello,
>
> I've stumbled upon an issue when using JSONField from 
> django.contrib.postgres.fields where I can't get it to run my additional 
> validators when calling full_clean(). To show a simplified example:
>
> validators.py:
>
> @deconstructible
> class TestValidator(object):
> def __call__(self, foo):
> raise ValidationError(message = _("Expected Error"))
>
>
> models.py:
>
> class DummyModel(models.Model):
> dummyJson = JSONField(validators = [TestValidator])
>
>
> shell output:
>
> >>> from Dummy.models import DummyModel
> >>> import json
> >>> dummyJsonInput = json.loads("""{"foo" : "bar"}""")
> >>> dummyModelInstance = DummyModel(dummyJson = dummyJsonInput)
> >>> dummyModelInstance.full_clean()
> >>>
>
>
> Am I doing something wrong? I've read some documentation about needing to 
> define __eq__ for migration purposes, but I tried defining it and it didn't 
> seem to make a difference.
>
> Any help is appreciated.
>
> Thanks,
> -Ryan Causey
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/e0460414-4ec6-4711-ad24-508c5740e140%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Validators and django.contrib.postgres.fields JSONField

[Ryan Causey]

Ah! Fantastic! I love it when I make a dumb mistake because it's an easy 
solution!

Instead of:
class DummyModel(models.Model):
dummyJson = JSONField(validators = [TestValidator])#Using class, wrong!

I should be doing:

class DummyModel(models.Model):
dummyJson = JSONField(validators = [TestValidator()])#Using instance of 
class, right!

As for using a class as a validator, I am performing some complex 
validation on the data inside the JSON to ensure it adheres to a specific 
format above and beyond the normal JSON validation, so a class based 
validator seemed like the right way to go. I based this off things like the 
Regex validator in django.core.validators found here: 
https://docs.djangoproject.com/en/1.10/_modules/django/core/validators/#RegexValidator.
 
Perhaps this wasn't the right approach?

Thanks for your help!
-Ryan Causey

On Sunday, August 28, 2016 at 3:46:31 AM UTC-7, Daniel Roseman wrote:
>
> On Sunday, 28 August 2016 07:54:47 UTC+1, Ryan Causey wrote:
>>
>> So I've dug into this a little more, and I've come up with the debugger 
>> trace below.
>>
>> -> dummyModelInstance.full_clean()
>>   c:\program files\python35\lib\site-packages\django\db\models\base.py(
>> 1210)full_clean()
>> -> self.clean_fields(exclude=exclude)
>>   c:\program files\python35\lib\site-packages\django\db\models\base.py(
>> 1252)clean_fields()
>> -> setattr(self, f.attname, f.clean(raw_value, self))
>>   c:\program files\python35\lib\site-packages\django\db\models\fields\
>> __init__.py(592)clean()
>> -> self.run_validators(value)
>>   c:\program files\python35\lib\site-packages\django\db\models\fields\
>> __init__.py(544)run_validators()
>> -> v(value)
>> > c:\program files\python35\lib\site-packages\django\utils\deconstruct.py
>> (16)__new__()
>> -> def __new__(cls, *args, **kwargs):
>> (Pdb) ll
>>  16  -> def __new__(cls, *args, **kwargs):
>>  17 # We capture the arguments to make returning them 
>> trivial
>>  18 obj = super(klass, cls).__new__(cls)
>>  19 obj._constructor_args = (args, kwargs)
>>  20 return obj
>>
>> For some reason, when run_validators() calls the custom TestValidator, it 
>> seems that a TestValidator object is constructed rather than entering the 
>> __call__() method of the class. I can't see a reason for this to happen, so 
>> what am I doing wrong here?
>>
>> Thanks,
>> -Ryan Causey
>>
>>>
>>>
> This isn't anything to do with JSONFields, but your misunderstanding about 
> what callables are and what `__call__` does. That is not a class method, 
> but an instance method; calling a class merely instantiates it (as you 
> should expect). Whatever is passed as a validator is called; if it's a 
> function, it will be executed, if it's an instance, its `__call__` method 
> will be called, but if it's a class it will be instantiated.
>
> I'm not sure why you are passing a class at all here. Normally a validator 
> is a method; if you need a class for some reason then you should pass an 
> *instance* of that class, not the class itself.
> -- 
> DR.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/64097c88-4239-4631-96be-168260fd53b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Atomic IntegerField increment based on another model instance

[Ryan Causey]

Assume the following models for this question:

class Jurisdiction(models.Model):
name = models.CharField(max_length = 100)

class Account(models.Model):
account_number = models.PositiveIntegerField(blank = True)
jurisdiction = models.ForeignKey("Jurisdiction", on_delete = models.
CASCADE)

class Meta:
"""Meta model to define unique_together constraints."""
#make the account_number unique for a jurisdiction
unique_together = ("account_number", "jurisdiction")

def create_account_number(self):
"""Create the account number for this account."""
# If the account number is already set, do nothing.
if self.account_number is not None:
return

# Get the last account created for the jurisdiction, based on 
having the largest account number.
last_account_for_jurisdiction = Account.objects.filter(jurisdiction 
= self.jurisdiction).order_by("-account_number").first()
# If the account exists set the account number.
if last_account_for_jurisdiction is not None:
self.account_number = last_account_for_jurisdiction.account_number 
+ 1
# Else there are no other accounts for this jurisdiction, set the 
first account number.
else:
self.account_number = 1

def save(self, *args, **kwargs):
# create the account number
self.create_account_number()
# call the superclass save to write the account to the database.
super().save(*args, **kwargs)

Is there a way in Django to be able to atomically set account_number on 
save based on the largest account number for a given jurisdiction? I was 
hoping some combination of F expressions 
, 
conditional expressions 
, 
and subquery expressions 
 would allow 
me to do this, but they don't appear to allow referencing a different 
model's fields.

The unique_together will properly prevent duplicate Jurisdiction + 
account_number pairs from being entered into the DB, but I was hoping there 
was a way to perform the setting of account_number in an atomic way so I 
wouldn't have to handle an IntegrityError and try again. I get the feeling 
what I'm asking for isn't possible, or there's a completely different way I 
should be going about this. I haven't used AutoField as I need a sequental 
series of numbers within each Jurisdiction.

Thanks in advance for the help.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fc7a0001-a576-46e5-b0a1-aeec500ad77c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Atomic IntegerField increment based on another model instance

[Ryan Causey]

The solution I ended up going with was using a PostgreSQL table lock:
@contextmanager
@transaction.atomic
def lock_table(model):
"""Performs a table lock on the provided model.

Note that this locks the entire table for the given model
and prevents any other operations from occurring on that
model. This should be used sparingly and not be used on
long running processes if possible.

This only works on PostgreSQL databases.
"""
with connection.cursor() as cursor:
# We use the most restrictive lock mode, ACCESS EXCLUSIVE, to prevent any
# other operations from being performed on the table while we hold this 
lock.
# See: https://www.postgresql.org/docs/current/explicit-locking.html
cursor.execute(f"LOCK TABLE {model._meta.db_table} IN ACCESS EXCLUSIVE MODE"
)
yield

I found, perhaps accidentally a way of doing it purely in the ORM but it 
seemed gross, was slower, and probably was a happy accident rather than a 
real solution. For the morbidly curious, it took making account_number a 
nullable field and updating the account number after save like so:

# My best guess is this locks the entire table? Without this it doesn't 
work.
list(Account.objects.select_for_update().all())
# Update ourself
Account.objects.filter(
pk = self.pk,
).annotate(
other_accounts_exist = models.Exists(
Account.objects.filter(
jurisdiction = models.OuterRef("jurisdiction"),
account_number__isnull = False,
).exclude(
pk = models.OuterRef("pk"),
),
),
).update(
account_number = models.Case(
models.When(
other_accounts_exist = True,
then = models.Subquery(
Account.objects.filter(
jurisdiction = models.OuterRef("jurisdiction"),
account_number__isnull = False,
).exclude(
pk = models.OuterRef("pk"),
).order_by(
"-account_number",
).values(
"account_number",
)[:1]
) + models.Value(1),
),
default = 1,
)
)
# Since we did an update we need to refresh ourselves from the db.
self.refresh_from_db()


On Saturday, June 1, 2019 at 8:44:29 PM UTC-7, Ryan Causey wrote:
>
> Assume the following models for this question:
>
> class Jurisdiction(models.Model):
> name = models.CharField(max_length = 100)
>
> class Account(models.Model):
> account_number = models.PositiveIntegerField(blank = True)
> jurisdiction = models.ForeignKey("Jurisdiction", on_delete = models.
> CASCADE)
>
> class Meta:
> """Meta model to define unique_together constraints."""
> #make the account_number unique for a jurisdiction
> unique_together = ("account_number", "jurisdiction")
>
> def create_account_number(self):
> """Create the account number for this account."""
> # If the account number is already set, do nothing.
> if self.account_number is not None:
> return
>
> # Get the last account created for the jurisdiction, based on 
> having the largest account number.
> last_account_for_jurisdiction = Account.objects.filter(jurisdiction 
> = self.jurisdiction).order_by("-account_number").first()
> # If the account exists set the account number.
> if last_account_for_jurisdiction is not None:
> self.account_number = 
> last_account_for_jurisdiction.account_number 
> + 1
> # Else there are no other accounts for this jurisdiction, set the 
> first account number.
> else:
> self.account_number = 1
>
> def save(self, *args, **kwargs):
> # create the account number
> self.create_account_number()
> # call the superclass save to write the account to the database.
> super().save(*args, **kwargs)
>
> Is there a way in Django to be able to atomically set account_number on 
> save based on the largest account number for a given jurisdiction? I was 
> hoping some combination of F expressions 
> <https://docs.djangoproject.com/en/2.2/ref/models/expressions/#f-expressions>,
>  
> conditional expressions 
> <https://docs.djangoproject.com/en/2.2/ref/models/conditional-expressions/>, 
> and subquery expressions 
> <https://docs.djangoproject.com/en/2.2/ref/models/expressions/> would 
> allow me to do this, but they don't appear to allow referencing a different 
> model's fields.
>
> The unique_together will properly prevent duplicate Jurisdiction + 
> account_number pairs from being entered into t

Understanding timezone with queryset day lookups

[Ryan Causey]

I ran into what I consider somewhat surprising behavior the other day when 
using the `__date` lookup to compare a datetime to a date in a queryset.

It appears according to these docs that the Queryset will convert the 
datetime value in the DateTimeField to the current timezone before 
comparing.
https://docs.djangoproject.com/en/dev/ref/models/querysets/#date

The reason I found this surprising is that `timezone.now()` always returns 
the time in UTC and in this contrived instance I was wanting to compare 
against UTC "today". I know this contrived example doesn't make sense, but 
the longer answer is I had other models' fields being set to UTC "today" 
using `timezone.now().date()`, and was using those fields' values to 
compare against a DateTimeField. The queryset behavior (I believe) doesn't 
work the way I expected in this context. Maybe this is pointing to UTC 
"today" is the "wrong" way to do it?

Is this queryset behavior due to the way timezone aware inputs in forms 
works? I.E. Naive datetime in form -> made aware in currently active 
timezone -> converted to UTC and stored in database if I understand 
correctly?

Thanks,
-Ryan Causey


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/46be92f6-aca9-4b32-a0ce-799467f73e30o%40googlegroups.com.

Having trouble understanding Timezone

[Ryan Causey]

Hello,

I'm new to Python and Django so please bear with me on this one.

I've been reading the Django docs, googling, and searching this mailing 
list on the proper configuration and usage of Django when USE_TZ = True. 
However, I'm still a bit fuzzy on things and would like clarification on 
the following:


   1. The documentation says that when USE_TZ = True that "Django stores 
   datetime information in UTC in the database, uses time-zone-aware datetime 
   objects internally, and translates them to the end user’s time zone in 
   templates and forms." Does this mean that the TIME_ZONE setting in 
   settings.py should be UTC? Or should it be the timezone in which the 
   database server is located?
   2. I am using a postgresql database, and the installation defaulted the 
   database's timezone to my local one. In support of Django storing all 
   datetime information in UTC in the database, does the database's timezone 
   needs to be set to UTC? Or is it correct for the database's timezone to be 
   the local one?
   3. The documentation also says that one should use the UTC timezone in 
   the code, and only convert to local time when dealing with end users. I 
   take this to mean that in any code I write I should set tzinfo = UTC for 
   all datetime objects I create programmatically and let Django translate 
   them in forms and views to the end user's timezone automatically. Is this 
   correct?

Thank you in advance for the help,

-Ryan Causey

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b37291e8-3d99-49a4-8e9a-13a6639ef29b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Having trouble understanding Timezone

[Ryan Causey]

Thank you everyone for your responses. I believe I have a clear 
understanding of how it all interacts and what I need to do on my end.

-Ryan Causey

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/76f144fc-9346-4921-8a51-8ffec41965b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reusable apps that support being nested wihin other apps

[Ryan Causey]

Is it possible in Django to reverse URLs within a reusable app such that 
the reversing will still work if the app is nested with another? It's 
essentially the same question as this stack overflow question 

 
but the answer there of moving the URLs into the root urlconf seems wrong 
to me. For example:

# app2 template
{% url 'app2:test' %}


# app2 urlconf
app_name = "app2"
urlpatterns = [
path("test", views.test_view.as_view(), name="test"),
]


# app1 urlconf
app_name = "app1"
urlpatterns = [
path("", include("app2.urls")),
]


# root urlconf
urlpatterns = [
path("", include("app1.urls")),
]

Attempting to render the app2 template currently results in a 
NoReverseMatch due to "app2" not being a registered namespace. If the app2 
template is overridden in app1 to instead use: 
{% url 'app1:app2:test' %}
the reverse issue goes away. It seems like there should be a way to 
mitigate this issue without overriding the app2 template in app1.

Is it not a valid use case for reusable apps to nest the url patterns of 
other reusable apps within themselves without expecting to have to override 
all places where a URL reverse is done in the nested reusable app?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/fa8d7f73-bcbd-43a7-bfe8-a0c0c6bd1fcb%40googlegroups.com.