If you do want to go down the streaming path despite the caching, static and other advice, I can see a couple of options: 1) Write your own sitemap view that doesn't use the sitemap.xml template and instead streams back results using a StreamingHttpResponse and generator based get_url methods in sitemap classes. If that works out well, it might be an appropriate dropin replacement for the current view, although that would remove the availability of a customisable template. 2) Ticket 13910 has a pull request to enable streaming templates, which would be a stepping stone to allowing refactoring of the existing sitemap view to stream a response using the template. The same caveat applies about whether this would improve memory usage or response-time performance if the sitemap urls are pre-calculated.
- Roger [1] https://code.djangoproject.com/ticket/13910 On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote: > > We ended up splitting out sitemaps up... and caching them heavily. > Pre-generating periodically would be equally effective. > > I can, however, see the merit in a generator style SiteMap engine with a > StreamingHttpResponse. It would allow the client to consume the data as > it's being consumed, instead of potentially timing out waiting for the > whole document to be generated. > > However, this may hide the growing problem for people whose sitemaps > gradually become large enough to warrant caching, but won't notice because > nothing is timing out, per se. > > Then again, micro-caching at the nginx/apache layer would also mitigate > the problem. > > -- > Curtis Maloney > > > > On 25 July 2013 23:51, Benjamin Wohlwend <[email protected] > <javascript:>>wrote: > >> Hi Julian, >> >> after having a site almost taken down by an over-eager spider downloading >> the sitemaps dozens of times in parallel, I learned the lesson that large >> sitemaps shouldn't be rendered on demand. Since that day a few years ago, I >> always generate the sitemap periodically and let apache/nginx serve it. >> Recently, I even found a package that does all the heavy lifting for you: >> https://github.com/xaralis/django-static-sitemaps >> >> Kind regards, >> Benjamin >> >> >> On Thu, Jul 25, 2013 at 10:06 AM, julianb <[email protected]<javascript:> >> > wrote: >> >>> Since 1.5 we have streaming responses. What is the state of >>> contrib.sitemaps in this regard? I have some very large sitemaps and >>> experimented with making them faster a few years ago. >>> >>> If the do not yet stream, I think this would be a good idea to get >>> memory usage down. Is there anything to keep an eye on? Would it be >>> valuable to Django if this is looked into? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to >>> [email protected]<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/django-developers. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to >> [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/django-developers. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
