Website slowed down drasticaly
Hello , I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly since Feb '19 there has been a drastic decrease in my website's response rate. For sure , there has been some minor increase in the no of hits, but the performance is too bad. Initially , the thought came that the hardware of the server was old , so a new server with high configuration was bought. We have done the new indexing also on it. Still the sought for a higher performance is awaited. Is it that the Django support for 1.8.1 or Python support for 2.7.12 has reduced and that is casing the website to slow down or I am missing out on something ? Kindly help. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Website slowed down drasticaly
Hello Uri, The project I handle is quite big , so before taking it up as a major task, I wish to get it from the experts that is the django version the issue or python version or am I missing out on something major. The issue started to escalate from Feb '19 , after which even a new powerful server could'nt give us good performance. *Now after adding extra indexes , at least the website is loading.* We are still not sure if we can handle around 5K - 10K concurrent request with a 128GB SSD server. On Thursday, 2 May 2019 18:11:05 UTC+5:30, Uri Even-Chen wrote: > > Why don't you upgrade Django at least to 1.11? Django 1.8 extended support > ended last year. > > https://www.djangoproject.com/download/ > אורי > u...@speedy.net > > > On Thu, May 2, 2019 at 2:39 PM Saurabh Adhikary > wrote: > >> Hello , >> >> I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly >> since Feb '19 there has been a drastic decrease in my website's response >> rate. >> For sure , there has been some minor increase in the no of hits, but the >> performance is too bad. >> >> Initially , the thought came that the hardware of the server was old , so >> a new server with high configuration was bought. >> We have done the new indexing also on it. >> Still the sought for a higher performance is awaited. >> >> >> Is it that the Django support for 1.8.1 or Python support for 2.7.12 has >> reduced and that is casing the website to slow down or I am missing out on >> something ? >> Kindly help. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django...@googlegroups.com . >> To post to this group, send email to django...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/08bc1409-322e-41ce-a94b-5dd9db888e86%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Website slowed down drasticaly
Hi Chetan, 1) Yes . We are infact trying this. Thank You. 2) *Taken* 3) We are on MySql , migration will cost a lot of man hours, which can be invested only if the result is concrete. 4) Up-gradation, that's our last resort. 5) Tried almost all. All in vain. Thanks for your inputs. :) On Thursday, 2 May 2019 17:52:11 UTC+5:30, Chetan Ganji wrote: > > Not sure about whats the issue. You could do couple of things to > understand whats the root cause of the problem. > I know they are generic guidelines. Anyone couldnt be more specific than > this. > > >1. Benchmark the time required to process each request. You could >write a middleware to track this time. >Attach starttime to each request object and read that at the time of >returning the response. > >2. Try using the django debug toolbar to see how much time it is >taking to execute the sql queries. >There might be some room for improvement as most developers dont >practice sql regularly. >https://django-debug-toolbar.readthedocs.io/en/latest/ > >3. Maybe try using a different database - PostgreSQL, MySQL, etc. > >4. But if you have time for R&D, you could try using the different >python implementation, it is said to be faster in many cases than cpython. >It's not 100% compliant i.e. some packages might not work with pypy. >So please do your research before walking down this road. >https://pypy.org/ > >5. Try using a different web server. If you are using Apache, try >using nginx. Also use a different wsgi server. If you are using gunicorn, >try with waitress or others. > > > > Regards, > Chetan Ganji > +91-900-483-4183 > ganji...@gmail.com > http://ryucoder.in > > > On Thu, May 2, 2019 at 5:09 PM Saurabh Adhikary > wrote: > >> Hello , >> >> I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly >> since Feb '19 there has been a drastic decrease in my website's response >> rate. >> For sure , there has been some minor increase in the no of hits, but the >> performance is too bad. >> >> Initially , the thought came that the hardware of the server was old , so >> a new server with high configuration was bought. >> We have done the new indexing also on it. >> Still the sought for a higher performance is awaited. >> >> >> Is it that the Django support for 1.8.1 or Python support for 2.7.12 has >> reduced and that is casing the website to slow down or I am missing out on >> something ? >> Kindly help. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django...@googlegroups.com . >> To post to this group, send email to django...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/django-users. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-users/238a6da2-8f34-4b8b-939c-e20d4306545b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/64e51c12-1ff3-4e9f-a45b-e0d32a6551b3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Website slowed down drasticaly
Hi Nick, We are using uwsgi . Uwsgi is installed via pip. Bad setup : is my area of interest as in some senior professionals can highlight the probable area of mistakes while shifting website from one server to another (literal hardware shift). Although we have some good people around, but there can a slip here and there. Inefficient coding: Not my area of concern as of now, it was working fair in our old server, so should work fine here too. Another area: Django & Python support end-life issue ? I'm yet to here anything concrete on this. On Friday, 3 May 2019 15:31:50 UTC+5:30, Nick Sarbicki wrote: > > Saurabh what do you use for the WSGI HTTP Server (gunicorn etc.)? Worth > checking that as well to make sure it has enough workers. > > In general performance problems are rarely version related. Usually it's > either a bad setup or inefficient code. > > Django toolbar and most of the other suggestions will help you profile the > code but worth seeing how it is served as well. > > - Nick > > > On Fri, May 3, 2019 at 10:21 AM Chetan Ganji > wrote: > >> One more thing - >> >> All the third party resources i.e. js and css files e.g. bootstrap, >> jquery, etc; don't fetch them from your server. >> Use CDN for those. This will help the load times a lot. >> >> >> Regards, >> Chetan Ganji >> +91-900-483-4183 >> ganji...@gmail.com >> http://ryucoder.in >> >> >> On Fri, May 3, 2019 at 2:47 PM Chetan Ganji > > wrote: >> >>> Hi Saurabh, >>> >>> You are welcome :) >>> >>> No 1 will give you insights about which endpoints are taking the most >>> time to load. Then, you can drill down the specific endpoints for >>> bottlenecks. >>> >>> Two more things you could try. >>> >>> 1. Remove unused/unnecessary middlewares from the middlewares list in >>> the settings.py of the project. >>> As they are executed before and after every single request, they could >>> add hugh unnecessary overhead easily. >>> >>> 2. Normally requests are processed in synchronous manner. Gunicorn has >>> aysnc worker types. >>> I did not have a need to look into it yet, but they could help speed up >>> things. >>> I would not suggest it for payment gateway endpoints. If your project is >>> a ERP/CRM type project, it could help you a little to look into it. >>> https://www.spirulasystems.com/blog/2015/01/20/gunicorn-worker-types/ >>> >>> Nginx already handles requests async manner. Try combining Nginx + Async >>> worker in gunicorn. >>> You could get some boost. >>> >>> >>> Regards, >>> Chetan Ganji >>> +91-900-483-4183 >>> ganji...@gmail.com >>> http://ryucoder.in >>> >>> >>> On Fri, May 3, 2019 at 11:00 AM Saurabh Adhikary >> > wrote: >>> >>>> Hi Chetan, >>>> >>>> 1) Yes . We are infact trying this. Thank You. >>>> 2) *Taken* >>>> 3) We are on MySql , migration will cost a lot of man hours, which can >>>> be invested only if the result is concrete. >>>> 4) Up-gradation, that's our last resort. >>>> 5) Tried almost all. All in vain. >>>> >>>> Thanks for your inputs. :) >>>> >>>> On Thursday, 2 May 2019 17:52:11 UTC+5:30, Chetan Ganji wrote: >>>>> >>>>> Not sure about whats the issue. You could do couple of things to >>>>> understand whats the root cause of the problem. >>>>> I know they are generic guidelines. Anyone couldnt be more specific >>>>> than this. >>>>> >>>>> >>>>>1. Benchmark the time required to process each request. You could >>>>>write a middleware to track this time. >>>>>Attach starttime to each request object and read that at the time >>>>>of returning the response. >>>>> >>>>>2. Try using the django debug toolbar to see how much time it is >>>>>taking to execute the sql queries. >>>>>There might be some room for improvement as most developers dont >>>>>practice sql regularly. >>>>>https://django-debug-toolbar.readthedocs.io/en/latest/ >>>>> >>>>>3. Maybe try using a different database - PostgreSQL, MySQL, etc. >>>>> >>>>>4. But if you have time for R&D, you could
Re: Website slowed down drasticaly
Hey Sally,Alex,Scott, I'm real glad to see that , all of you are exactly pointing out the area of my concern. Thank you so much. The point is , there are no major change that was deployed on to the code since Dec'18. As I did mention that there has been some minor load increase of our users in the new year since Jan, but the change is not significant. (Originally we had around 2million unique people , ~5million approx approx users in total, and around 50K new users during this time) We are on MySQL 14.14 Distrib 5.1.73. My worry is if *Django support(for 1.8.1)* or *Python support (for 2.7.12*) diminishing is the *main reason for slowing down.* Rest, the debugging has been done, wherin we could'nt find a fault as of yet. My project is really big and I hence wish to seek some experience or concrete knowledge before investing time for upgrading Django & Python. Regards, Saurabh On Sunday, 5 May 2019 19:36:25 UTC+5:30, Sally Middleton wrote: > > Hi > > From a generic debugging point of view, you need to find out the root > cause of the performance decline. Ideally you need to isolate and test each > part of the system in turn and look for the one which is causing the > problem. > > 1. If it can be reproduced on a prod-like environment then that's really > great as you can make changes easily to test. Always keep a note of EXACTLY > what you've changed and put it back the way it was previously before your > next test. If you cannot reproduce that also tells you something, for > example it's due to data quantities or prod-hardware. > > 2. Put tracing/logging on your prod environment database and check for SQL > for degradation (maybe compared to your development env). Maybe all is OK > or one query is slow or everything is slow. Any performance issues will > need tuning, but it sounds like you are looking for something quite > fundamental. > > 3. What happened in February? Did you perform a release to Production? > What was in it? If you roll-back (in test environment) does it fix the > problem? Maybe it is a component that has been updated. Any configuration > changes to your web server? > > 4. If it isn't the database then it's something else. You need to isolate > each part of the system in turn and put on logging then check the results. > > Best regards > Sally > > > On Thursday, 2 May 2019 12:40:20 UTC+1, Saurabh Adhikary wrote: >> >> Hello , >> >> I'm using Django version 1.8.1 and Python version 2.7.12 . Suddenly >> since Feb '19 there has been a drastic decrease in my website's response >> rate. >> For sure , there has been some minor increase in the no of hits, but the >> performance is too bad. >> >> Initially , the thought came that the hardware of the server was old , so >> a new server with high configuration was bought. >> We have done the new indexing also on it. >> Still the sought for a higher performance is awaited. >> >> >> Is it that the Django support for 1.8.1 or Python support for 2.7.12 has >> reduced and that is casing the website to slow down or I am missing out on >> something ? >> Kindly help. >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/85afc1b7-f8bb-468d-8759-c41bf3645dce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Additional column in Queryset with condition
Hello ,
I need some advice / tips.
I am in a fix.
I have 2 databases & I want to join 2 tables of that. *As we know the
foreign key concept with 2 separate databases is not possible. *
Basically what I want is to add a custom column to a queryset and pass the
*current row.name
<http://row.name/> *value to a function and that function will return
*location*. The return value should be saved for each row individually.
Each row will have different values. Currently, by annotate the function is
calling but :
1) * name* values are not getting passed
2) anything returned once is saved for the entire queryset.
*Constraint - change in models is difficult for my prod environment.*
database1.table1
database2.table1
*name* *rating*
*outlet* *location*
pizza hut 5
pizza-hut Mumbai
mc donalds 4
mc-donalds Washington
Now, I want to get *location *from database2.table1 and sort it with this
field .
What I have tried :
1 - queryset.annotate(loc=Value('abc', output_field=CharField())) and later
updating the queryset. But the main queryset doesn't gets updated by each
row of the custom column. All the values gets updated by the same value.
2 - queryset.extra(select={'loc': myfunc('*name*')}) but here the column
values are not getting passed.
3 - queryset.extra(select={'loc': "select '*location*' from *database2*.table1
where outlet == %s"},select_params=('*name*',)) but here the column values
are not getting passed.
Kindly , if you can help find a solution by :
- handling the queryset in a better way
- writing a query like Count as that is working fine for a similar column
(although I know this works on Foreign key, but still if the logic is
helped how to write Count type of function I can write my own snippet)
Regards,
Saurabh Adhikary
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/c2447190-9a56-4abb-b554-41dab35ec4c9%40googlegroups.com.
Re: JSON data
data = [{"exped": "MA", "destin": "AL", "count": 2}, {"exped": "MA",
"destin": "BS", "count": 1}]
mylist = []
for a in data :
mylist.append([a['exped'],a['destin'],a['count']])
On Sunday, 24 May 2020 17:16:46 UTC+5:30, HJ wrote:
>
> hello guys hope you are doing well
>
> I am trying to make a highchart but I found a problem in the view function
> so my chart doesn't show becuase of that
>
> view function
>
> def jsonDepend(request):
> dataset = mail_item_countries_depend.objects.all().values('exped',
> 'destin','count')
> data = list(dataset)
>
> return JsonResponse(data, safe=False)
>
>
> my views function show something like this : [{"exped": "MA", "destin":
> "AL", "count": 2}, {"exped": "MA", "destin": "BS", "count": 1}]
>
> how can I store it like that : [["MA", "AL", 2], ["MA", "BS", 1]]
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/0a644423-e96c-46d5-9f3e-c9dd3195f189%40googlegroups.com.
How to create a function like Count
I want to create a custom function which will get modified into a sql querry and will work like Count ...i.e. for every row of the queryset -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/7a301d3e-d690-4b49-b6da-02d00c97758e%40googlegroups.com.
