Do you expect your background threads to be equivalent to or greater than the number of requests you're normally servicing? Usually background tasks are much less frequent than the web requests, so a little overhead w/r/t database connections isn't even going to be noticed.
Looking at what Django does, at the start and end of each request it calls connection.close_if_unusable_or_obsolete(). That function does careful checks to see if the connection is even worth using. Unless you do something similar in your thread_start (adding more complication than I've suggested), having a TLS connection will cause more problems than it saves you. To make this work in general you'd at least need a hook at the point the thread is removed from and added back to the pool, not when the thread exits. Also, the connection won't be opened unless you actually do something that needs it. Personally, I think this sounds like something you're trying to optimize before you've profiled that the benefit it worth it. On Thu, Jun 2, 2016 at 6:33 PM, Cristiano Coelho <[email protected]> wrote: > I'm not starting threads by hand but rather a pool which handles any > threads for me, I basically just send a function to the pool and leave it > run. > You are right, I could wrap every function sent to the pool with the code > you proposed, but I also don't want to open and close a connection on every > function call, but instead only when the thread from the pool is no longer > required and disposed, pretty much on application exist, although the pool > handler can do what ever it wants with the threads. > > El jueves, 2 de junio de 2016, 20:22:21 (UTC-3), Stephen Butler escribió: >> >> I'm still a bit confused. What is the advantage of having connections >> closed automatically when the thread exits? It seems to me that you can >> quickly solve your problem by modifying your thread start routines: >> >> from django.db import connection >> from contextlib import closing >> >> def my_thread_start(): >> with closing(connection): >> # do normal work >> >> You can even create a quick decorator if that's too much modification. >> >> On Thu, Jun 2, 2016 at 5:48 PM, Cristiano Coelho <[email protected]> >> wrote: >> >>> Florian, >>> >>> Sorry about the SIGTERM and SIGKILL confusion, I think I read somewhere >>> some time ago that SIGTERM would instantly finish any pending request, so I >>> assumed it would also kill any thread in not a really nice way. However now >>> that you mention it, there's one SIGKILL from the apache logs (compared to >>> the thousands of sigterm due to restarts). However the connections that >>> were somehow stuck and never close dated from about 2 weeks ago, yes, there >>> were connections that were opened 2 weeks ago and never closed, even if >>> apache was restarded many times every day! >>> >>> About thread pool, I'm talking about python's thread pool I'm using to >>> offload work, not any django's pool, and these pools are the ones I have no >>> control over its threads as they are completely managed by the thread pool >>> library. >>> >>> 3 days has passed since I noticed those hanged out connections and the >>> issue didn't repeat again yet, maybe it was some really odd condition that >>> caused them, but the thread pool's threads connections are indeed being >>> correctly closed on servers restart, so a very odd case created those >>> hanged connections. >>> >>> So just to be sure, is SIGTERM actually propagated to python code so it >>> can gracefully kill all threads, garbage collect and close connections? >>> Would a SIGKILL actually prevent any kind of cleanup leaving a chance for >>> python/django leave some connections opened? >>> >>> Maybe this is a postgres issue instead that happened for some very odd >>> reason. >>> >>> >>> Finally, would it be possible through any kind of callbacks of the >>> thread local object to fire a connection close before a thread dies? This >>> would certainly help rather than waiting for the connection to get garbage >>> collected. You mentioned that connections could end up being shared by >>> threads, but I don't see that being something being done in django at all. >>> >>> >>> El jueves, 2 de junio de 2016, 19:32:15 (UTC-3), Florian Apolloner >>> escribió: >>>> >>>> On Thursday, June 2, 2016 at 11:55:41 PM UTC+2, Cristiano Coelho wrote: >>>>> >>>>> Not always, for example, on amazon elastic beasntalk when you either >>>>> restart the app server or upload a new version, it basically kills apache >>>>> and all WSGI processes through a sigterm >>>>> >>>> >>>> A SIGTERM is a normal signal an should cause a proper shutdown. >>>> >>>> >>>>> so those thread pools are probably killed in a bad way and you don't >>>>> really have control over that. >>>>> >>>> >>>> Absolutely not, you are mixing up SIGTERM and SIGKILL. >>>> >>>> >>>>> Also you don't really have control on the life of a thread pool >>>>> thread, so a given thread could be gracefully stopped by the pool >>>>> implementation >>>>> >>>> >>>> Once again: there is no pool. >>>> >>>> As ayneric pointed out, it seems like those connections are correctly >>>>> closed most of the time when a thread dies, but for some reason, postgres >>>>> would keep some connections opened. >>>>> >>>> >>>> If a connection is closed properly, postgres will close it accordingly. >>>> The only way possible for a connection to stay open while the app is gone >>>> is that you are running into tcp timeouts while getting killed with SIGTERM >>>> (dunno if the postgres protocol has keep alive support on the protocol >>>> level, most likely not). As long as you are not sending a SIGTERM, python >>>> should clean up properly which should call garbage collection, which then >>>> again should delete all connections and therefore close the connection. Any >>>> other behavior seems like a bug in Python (or maybe Django, but as long as >>>> Python shuts down properly I think we are fine). >>>> >>>> Are there any rare cases where even if the thread is stopped the >>>>> connection won't be closed? The only thing I can think of are that those >>>>> threads are never garbage collected or something. >>>>> >>>> >>>> Depends on the python version you are using, especially thread local >>>> behavior changed a lot… >>>> >>>> Cheers, >>>> Florian >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/562bf2f9-8a44-495c-bacd-9a42012aa011%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-developers/562bf2f9-8a44-495c-bacd-9a42012aa011%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/b54b3699-d755-4d19-ac9f-ee0c0fb69d13%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/b54b3699-d755-4d19-ac9f-ee0c0fb69d13%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAD4ANxWGJdx75Kgh2m7vmCQUidgRwH%2BHbY%2B-aGNT2xUG_n-AyA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
