Reduce number of calls to database
Hi,
I am new to Django and I am performing some iteration on a queryset. When I
use django_debug_toolbar to look at the SQL usage, I realised that Django
is actually calling the database once in each iteration. Is there a way to
make it call the database only once and somehow store it locally so that I
can iterate on it?
Code:
for level_id in levels_id:
levels.append(get_object_or_404(Level, id=level_id))
I am trying this modified code, but it seems that it's still calling the
same number of times.
levels_dict = repr(Level.objects.values('id', 'name'))
for level_id in levels_id:
levels.append(levels_dict.get(id=level_id).get('name'))
I am trying to use this pattern in quite a few places, where I want to get
some data from a list of objects retrieved from the database and iterate
over them.
I would be happy to provide any missing information.
Could anyone kindly help me out? Cheers!
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/631e4503-c3ff-432a-9f6b-b0f6b6105b26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reduce number of sql calls to database
Hi,
I am new to Django and I am trying to reduce the number of calls to
database as it's slowing down the app. I am performing iteration over the
queryset and I used django_debug_toolbar to check the number of queries
made, and the number is huge. It looks like django is making a query call
to the db in each iteration. Is there a way to make it only send one query
and then store the result locally and iterate over it?
Code:
for level_id in levels_id:
levels.append(get_object_or_404(Level, id=level_id))
I have been trying out this code, but it seems that the number of sql
queries is still the same.
levels_dict = Level.objects.values('id', 'name')
for level_id in levels_id:
levels.append(levels_dict.get(id=level_id).get('name'))
I am using the same pattern in a few places to get data from each of the
object in the queryset, is there a way to improve the performance? I am
happy to provide any missing information. Cheers!
P.S. I tried to post just now but the post seems to be missing so I am
trying again, hopefully I didn't post twice!
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/3745df48-5278-4b59-ad1e-f03ba855da15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Reduce number of calls to database
Thanks! Would this still give 404 errors when a certain object does not
exist?
On Tuesday, 2 June 2015 17:07:49 UTC+1, Simon Charette wrote:
>
> Hi Cherie,
>
> A `id__in` queryset lookup should issue a single query.
>
> levels = Level.objects.filter(id__in=level_ids)
>
> Cheers,
> Simon
>
> Le mardi 2 juin 2015 11:42:19 UTC-4, Cherie Pun a écrit :
>>
>> Hi,
>>
>> I am new to Django and I am performing some iteration on a queryset. When
>> I use django_debug_toolbar to look at the SQL usage, I realised that Django
>> is actually calling the database once in each iteration. Is there a way to
>> make it call the database only once and somehow store it locally so that I
>> can iterate on it?
>>
>> Code:
>> for level_id in levels_id:
>>levels.append(get_object_or_404(Level, id=level_id))
>>
>>
>> I am trying this modified code, but it seems that it's still calling the
>> same number of times.
>>
>> levels_dict = repr(Level.objects.values('id', 'name'))
>> for level_id in levels_id:
>> levels.append(levels_dict.get(id=level_id).get('name'))
>>
>> I am trying to use this pattern in quite a few places, where I want to
>> get some data from a list of objects retrieved from the database and
>> iterate over them.
>> I would be happy to provide any missing information.
>>
>> Could anyone kindly help me out? Cheers!
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/844b1d2b-8665-4176-bee5-06297ebe7247%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Reduce number of calls to database
Thanks! On Friday, 5 June 2015 08:53:48 UTC+1, kel...@ist-total.org wrote: > > On Fri Jun 5 08:32:52 2015 GMT+0100, Cherie Pun wrote: > > > > Thanks! Would this still give 404 errors when a certain object does not > > exist? > > Nope, but if you really need to ensure all ids exist in the database, > compare the length of your level_ids with levels.count() > Make sure there are no duplicates in the id list (e.g. use > set(level_ids)). > > If they differ raise Http404 manually. > > -- > Florian > > > On Tuesday, 2 June 2015 17:07:49 UTC+1, Simon Charette wrote: > > > > > > Hi Cherie, > > > > > > A `id__in` queryset lookup should issue a single query. > > > > > > levels = Level.objects.filter(id__in=level_ids) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1e5b6e7f-72bc-45f4-aac5-a3976945f990%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Squashed migration
Hi, I have trying to experiment with squashmigration to see if it will make it faster to build the database when running tests. So I have squashed the migrations following the instructions on the Django website. However when I run the tests, it still uses the original migrations. I thought Django automatically uses the squashed one over the separated ones. Is there some settings that I have to configure? Also, it says that no optimisation was available even though there are a few AddField which should in theory be combined into AddModel. Any help or suggestions will be much appreciated, thanks! Cherie -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9288958d-0214-4cd2-9b51-dd173a8e83ae%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Squashed migration
Hi, Thanks for your replies! I am still on Django 1.7 but we are hoping to upgrade to 1.8 soon. I manage to use the squashed migrations when creating the test database now, but there's syntax error in the automatically generated squash migration. The error occurs on the line on which it reference the user-defined functions in other migrations. It seems fine when it ran the first methods though. Cheers, Cherie On Friday, June 12, 2015 at 12:17:38 PM UTC+1, Filipe Ximenes wrote: > > Hi Cherie, maybe you can run your tests without the migrations. This > django app may solve your problem: > https://github.com/henriquebastos/django-test-without-migrations/ > > On Fri, Jun 12, 2015 at 6:38 AM, Cherie Pun > wrote: > >> Hi, >> >> I have trying to experiment with squashmigration to see if it will make >> it faster to build the database when running tests. So I have squashed the >> migrations following the instructions on the Django website. However when I >> run the tests, it still uses the original migrations. I thought Django >> automatically uses the squashed one over the separated ones. Is there some >> settings that I have to configure? Also, it says that no optimisation was >> available even though there are a few AddField which should in theory be >> combined into AddModel. >> >> Any help or suggestions will be much appreciated, thanks! >> >> Cherie >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-users...@googlegroups.com . >> To post to this group, send email to django...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/9288958d-0214-4cd2-9b51-dd173a8e83ae%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/9288958d-0214-4cd2-9b51-dd173a8e83ae%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > > *Filipe Ximenes*+55 (81) 8245-9204 > > *Vinta Software Studio*http://www.vinta.com.br > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d997ffc2-b35e-4587-a278-cb2176f2cff5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Squashed migration
Hi, So the original problem was that I was running in the repo which didn't have the squashed migration. Django does know when to switch to the squashed migrations when you have both squashed and unsquashed migration files coexist in the folder. As for the syntax error it was because python cannot import from modules that starts with numbers. Also, the idea was that we shouldn't use the migration files as modules to import code from, that's why the file name stayed as it was. Users should manually move their RunPython methods manually and resolve those invalid references to migration files. Hope this helps and sorry for confusing people that squash migration doesn't work! I am still new to Python and Django and I really appreciate everyone's replies. Cheers, Cherie On Monday, June 15, 2015 at 9:54:09 AM UTC+1, aRkadeFR wrote: > > Thanks for the good explanation as always :) > > From the documentation plus the explanation of the problem, > I wanted to know if by deleting the old migration it would still > run the old migrations. > > Do we know the end of the story? Where it comes from? > > Thanks > > On 06/12/2015 05:42 PM, Carl Meyer wrote: > > On 06/12/2015 06:32 AM, aRkadeFR wrote: > >> You need to delete your old migrations so it uses only the squashed > >> one after. > > No, the squashed migration should be used in place of the old ones for > > any new database, even if the old ones are still present. This is the > > point of the squashmigrations feature; that you can keep the old ones > > around (which is necessary for any deployments that may not yet have > > applied all of them) while still gaining the benefit of the new squashed > > migration for new deployments (and tests). > > > > I know this works, because I just did it recently myself. It sounds like > > Cherie was able to get it working too, though we didn't get any > > clarification on why it didn't seem to work originally. > > > >> In the documentation: > >> "This enables you to squash and not mess up systems currently in > >> production that aren’t fully up-to-date yet. The recommended process is > >> to squash, keeping the old files, commit and release, wait until all > >> systems are upgraded with the new release (or if you’re a third-party > >> project, just ensure your users upgrade releases in order without > >> skipping any), and then remove the old files, commit and do a second > >> release." > > That's right. The length of time you need to wait before removing can > > vary widely. For a third-party app, it may be a full release cycle or > > two (as long as you can tell your users to upgrade version-by-version). > > For a project with only a few deployments, all under your control, it > > may be the same day. But regardless, the squashed migration will still > > be used in tests immediately, before you remove the old migrations. > > > > Carl > > > > -- > aRkadeFR > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a46519d9-3819-40d5-8a18-236a4bec846e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Diable basic auth on some pages
Hi, In the dev version of the website, we are using basic auth to limit access. However, we are currently developing an api for the website and I would like to use the api without going through the basic auth. Is it possible to disable basic auth for all the api urls? Cheers, Cherie -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/0d9a681e-74c4-4b74-9648-e0e6c74beeb6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Diable basic auth on some pages
Hi, We are using 'deploy.middleware.basicauth.BasicAuthMiddleware' right now. If I remove that line, I will be able to access the api page. However if it's included, when I access the api page, even after I have typed in the username and password, the api page will show that I am not authenticated to view the page. I am using django_rest_framework by the way, maybe it's some sort of restriction on their side? Cheers, Cherie On Tuesday, 30 June 2015 18:12:30 UTC+1, Alex Heyden wrote: > > Cherie, > > By default, there are no authentication controls in Django. Barring some > custom middleware, authentication checks are performed within the views > themselves or through something like the @login_required or > @user_passes_test decorators. Are you certain that authentication is what's > blocking your requests? > > On Tue, Jun 30, 2015 at 9:43 AM, Cherie Pun > wrote: > >> Hi, >> >> In the dev version of the website, we are using basic auth to limit >> access. However, we are currently developing an api for the website and I >> would like to use the api without going through the basic auth. Is it >> possible to disable basic auth for all the api urls? >> >> Cheers, >> Cherie >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-users...@googlegroups.com . >> To post to this group, send email to django...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/0d9a681e-74c4-4b74-9648-e0e6c74beeb6%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/0d9a681e-74c4-4b74-9648-e0e6c74beeb6%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4381311c-34c9-42ca-9069-30853f5a1fde%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Diable basic auth on some pages
Yes it is, would it be possible to disable it for a particular page? Cheers, Cherie On Wednesday, July 1, 2015 at 10:15:57 AM UTC+1, monoBOT monoBOT wrote: > > > 2015-07-01 8:51 GMT+01:00 Cherie Pun >: > >> >> We are using 'deploy.middleware.basicauth.BasicAuthMiddleware' right now. >> If I remove that line, I will be able to access the api page. However if >> it's included, when I access the api page, even after I have typed in the >> username and password, the api page will show that I am not authenticated >> to view the page. I am using django_rest_framework by the way, maybe it's >> some sort of restriction on their side? >> > > Go check the settings.py to see if you have a permission setting for the > whole application. > > > > -- > *monoBOT* > Visite mi sitio(Visit my site): monobotsoft.es/blog/ > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/cef52288-e081-4243-b9ec-a385b9363ec8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Diable basic auth on some pages
I see. Thanks for the replies, really appreciate your help :) Cheers, Cherie On Wednesday, July 1, 2015 at 1:47:26 PM UTC+1, monoBOT monoBOT wrote: > > No, if that permission is applied to the whole application. > > But you can aply the permission per view ... just delete that > configuration in the settings file and manually apply on the views you want > to. > > 2015-07-01 11:03 GMT+01:00 Cherie Pun >: > >> Yes it is, would it be possible to disable it for a particular page? >> >> Cheers, >> Cherie >> >> On Wednesday, July 1, 2015 at 10:15:57 AM UTC+1, monoBOT monoBOT wrote: >> >>> >>> 2015-07-01 8:51 GMT+01:00 Cherie Pun : >>> >>>> >>>> We are using 'deploy.middleware.basicauth.BasicAuthMiddleware' right >>>> now. If I remove that line, I will be able to access the api page. However >>>> if it's included, when I access the api page, even after I have typed in >>>> the username and password, the api page will show that I am not >>>> authenticated to view the page. I am using django_rest_framework by the >>>> way, maybe it's some sort of restriction on their side? >>>> >>> >>> Go check the settings.py to see if you have a permission setting for >>> the whole application. >>> >>> >>> >>> -- >>> *monoBOT* >>> Visite mi sitio(Visit my site): monobotsoft.es/blog/ >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-users...@googlegroups.com . >> To post to this group, send email to django...@googlegroups.com >> . >> Visit this group at http://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/cef52288-e081-4243-b9ec-a385b9363ec8%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/cef52288-e081-4243-b9ec-a385b9363ec8%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > *monoBOT* > Visite mi sitio(Visit my site): monobotsoft.es/blog/ > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/42fd5c8a-62e7-4f5e-b704-e9cafaeb2424%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Checking user permission denied in Django Selenium tests
On the local server that I start up, if the user does not have the required permission for a particular view, they will be redirected to the 403 page (I am using the permission_required decorator) However, in the selenium test, the PermissionDenied exception is thrown and the user is redirected to the internal server error page (500) instead. Does the server not work the same way when in selenium testing environment? Should I catch that exception in order to check permission is denied instead of checking for the 403 page? Cheers! -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/dccd8e33-8f30-4aef-a030-644112efaf13%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
