Hi Melvyn, Thanks for your response.
We are using the Celery for all tasks which could be done offline or separately. Report generation is one of it. We trigger the task from Browser but the report creation is run via Celery which when completed sends the csv filepath to the browser to be downloaded. But the CPU usage and time taken are high. Though right now we are using a single multicore HW. Since I am not quite adept in Django, hence, I raised the query if there is a way to optimise. I have used defer, only etc to lighten the query but still its taking time and CPU. But seems like ORM is a bottle-neck. I agree it's prudent to have the reporting on a separate HW. Would consider it. Thanks. On Friday, March 10, 2017 at 5:25:39 PM UTC+5:30, Melvyn Sopacua wrote: > > On Friday 10 March 2017 03:06:12 Web Architect wrote: > > > Hi James, > > > > > > Thanks for your response. Melvyn also posed a similar point of not > > > loading the whole records. > > > > > > But all the records are needed for reporting purposes - where the data > > > is read from the DB and a csv report is created. I am not quite an > > > expert on Django but I am not sure if there is a better way to do it. > > > > > > The scenario is as follows to make it clearer: > > > > > > Ours is an ecommerce site built on Django. Our admin/accounting team > > > needs to download reports now and then. We have a Django model for > > > the line items purchased. Now there could be 10k line items sold and > > > each line items are associated with other models like payments, > > > shipments etc which is a complex set of relations. > > > > The most scalable solution is to not send the CSV to the browser and not > do it at the webserver. > > Use some tasking system like Celery to generate the report at a different > server. Use a management command to do it. Then mail the report or make it > available as static file via rsync/ssh/whathavyou. > > > > You get bonus points for setting up the report generating server with a > read-only slave of the database. > > > > This scales much better and doesn't tie up webserver resources. > > -- > > Melvyn Sopacua > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a9efcf50-7638-47fd-a4a8-cd04192c6af3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
