I've come up with a patch to the amqp 1.0 driver that resets the connection state if the pid of the current process is different from the process that has created the connection:
https://review.openstack.org/#/c/134684/ I've managed to get this to work using rpc_workers=4 in neutron.conf, which failed consistently pre-patch. On Tue, Nov 25, 2014 at 11:16 AM, Mehdi Abaakouk <sil...@sileht.net> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > > >> Mmmm... I don't think it's that clear (re: an application issue). I >> mean, yes - the application is doing the os.fork() at the 'wrong' >> time, but where is this made clear in the oslo.messaging API >> documentation? >> I think this is the real issue here: what is the "official" guidance >> for using os.fork() and its interaction with oslo libraries? >> >> In the case of oslo.messaging, I can't find any mention of os.fork() >> in the API docs (I may have missed it - please correct me if so). >> That would imply - at least to me - that there is _no_ restrictions of >> using os.fork() together with oslo.messaging. > > > Yes, I agree we should add a note on that in oslo.messaging (and perhaps in > oslo.db too). > > And also the os.fork() is done by the service.ProcessLauncher of > oslo-incubator, > and it's not (yet) documented. But once oslo.service library will be > released, it will. > >> But in the case of qpid, that is definitely _not_ the case. >> >> The legacy qpid driver - impl_qpid - imports a 3rd party library, the >> qpid.messaging API. This library uses threading.Thread internally, >> we (consumers of this library) have no control over how that thread is >> managed. So for impl_qpid, os.fork()'ing after the driver is loaded >> can't be guaranteed to work. In fact, I'd say os.fork() and >> impl_qpid will not work - full stop. > > > Yes, I have tried it, and I have catch what happen and I can confirm that > too now, unfortunately :( And this can occurs with any driver if the 3rd > party library > doesn't work when we use os.fork() > >>> For the amqp1 driver case, I think this is the same things, it seems to >>> do lazy creation of the connection too. >>> >> >> We have more flexibility here, since the driver directly controls when >> the thread is spawned. But the very fact that the thread is used >> places a restriction on how oslo.messaging and os.fork() can be used >> together, which isn't made clear in the documentation for the library. >> >> I'm not familiar with the rabbit driver - I've seen some patch for >> heatbeating in rabbit introduce threading, so there may also be an >> implication there as well. > > > Yes, we need to check that. > >>> Personally, I don't like this API, because the behavior difference >>> between >>> '__init__' and 'start' is too implicit. >>> >> >> That's true, but I'd say that the problem of implicitness re: >> os.fork() needs to be clarified at the library level as well. >> > > I agree. > > I will write the documentation patch for oslo.messaging. > > Cheers, > > - --- > Mehdi Abaakouk > mail: sil...@sileht.net > irc: sileht > -----BEGIN PGP SIGNATURE----- > Version: OpenPGP.js v.1.20131017 > Comment: http://openpgpjs.org > > wsFcBAEBCAAQBQJUdKtPCRAYkrQvzqrryAAAqCIP/RvONngQ1MOKXEcktcku > Ok1Lr9O12O3j/zESkXaKInJbqak0EjM0FXnoXeah4qPbSoJYZIfztmCOtX4u > CSdhkAWhFqXdpHUtngXImHeB3P/eRH0Vo7R3PAmUbv5VWkn+lwcO+Ym1g79Z > vCJbcwVpldGiTDRJRDAPPb14UakJbZJGnkRDgYscNlBG+alzLw0MsaqnJ7LS > 8Yj4YkXSgthpHLF2N8Yem9r7Lh7CbYLKzlhOylgAJTd8gpGGtncwWMwYJvKc > lsMJNY34PMiNkPk1a+VSlrWcPJpafBl3aOBbrIpmMSpMe9pXC/yHW2nrtGez > VXxliFpqQ7kA5827AuhPAM8EzeMUDetLhZvLxzqY7f/SlaoQ/s/9VhfemmHv > d4wT8uiayrWSMdXVUJZcMUdM2XlJDdObokMI0ZQKQYX8OhKQL8LdaHR2xr6B > OjS4Mp4+/W4Y9wMUFqlRyGnW1LLwCFYWHpyKlhXKmYSSdKTn5L7Pcvmmfw8d > JzDcMxfKCBnM4mNRzlBqYV4/ysb6FNMUZwu+D1YxCVHmAH2O1/oODujNJFkZ > gSWAmh9kYawJKbbS0Lh7nkOJs1iOnYxG0IQmz61sffg8T2FrpbH4FNWh1/+a > mQhmYWH2L5noJIwncVQSloSRuoSWLj9rfeiTIHjq2ZnTUD5tbXK6S5dTvv4m > 4bij > =G9oX > -----END PGP SIGNATURE----- > -- Ken Giusti (kgiu...@gmail.com) _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
