[Yahoo-eng-team] [Bug 1257323] [NEW] Keystone token_flush execution may fail because of full transaction log in DB2
Public bug reported:
If there is a high number of expired tokens, a user may get the
following error message:
$ keystone-manage token_flush
2013-11-18 02:11:09.491 3806 CRITICAL keystone [-] (InternalError)
ibm_db_dbi::InternalError: Statement Execute Failed: [IBM][CLI
Driver][DB2/LINUXX8664] SQL0964C The transaction log for the database is full.
SQLSTATE=57011 SQLCODE=-964 'DELETE FROM token WHERE token.expires < ?'
(datetime.datetime(2013, 11, 18, 8, 10, 37, 519596),)
** Affects: keystone
Importance: Undecided
Assignee: John Warren (jswarren)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1257323
Title:
Keystone token_flush execution may fail because of full transaction
log in DB2
Status in OpenStack Identity (Keystone):
In Progress
Bug description:
If there is a high number of expired tokens, a user may get the
following error message:
$ keystone-manage token_flush
2013-11-18 02:11:09.491 3806 CRITICAL keystone [-] (InternalError)
ibm_db_dbi::InternalError: Statement Execute Failed: [IBM][CLI
Driver][DB2/LINUXX8664] SQL0964C The transaction log for the database is full.
SQLSTATE=57011 SQLCODE=-964 'DELETE FROM token WHERE token.expires < ?'
(datetime.datetime(2013, 11, 18, 8, 10, 37, 519596),)
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1257323/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1278939] [NEW] Insufficient upper bounds checking performed on flavor-create API
Public bug reported: The fix for bug #1243195 added upper-bound checking for flavor fields based on sys.maxint. The problem is that the value of maxint varies by platform (either 32- or 64-bit signed integers), while the INTEGER type in databases is limited to 32-bit signed integers. This means that the upper-bound checking does not work on platforms where sys.maxint returns the maximum value of a 64-bit signed integer. The original reported problems (as shown in http://paste.openstack.org/show/48988/) persist on 64-bit platforms. ** Affects: nova Importance: Undecided Status: New ** Tags: api db -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1278939 Title: Insufficient upper bounds checking performed on flavor-create API Status in OpenStack Compute (Nova): New Bug description: The fix for bug #1243195 added upper-bound checking for flavor fields based on sys.maxint. The problem is that the value of maxint varies by platform (either 32- or 64-bit signed integers), while the INTEGER type in databases is limited to 32-bit signed integers. This means that the upper-bound checking does not work on platforms where sys.maxint returns the maximum value of a 64-bit signed integer. The original reported problems (as shown in http://paste.openstack.org/show/48988/) persist on 64-bit platforms. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1278939/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1290487] [NEW] Libvirt native thread used for "forbidden" purposes
Public bug reported: In the nova.virt.libvirt.driver.LibvirtDriver. _get_new_connection method two different libvirt event handlers are registered, one for lifecycle events (_event_lifecycle_callback) and one for connection events (_close_callback). These callbacks are called by a native thread that is continually calling libvirt.virEventRunDefaultImpl() in the _native_thread method; the latter method's Docstring contains the following note: This is a native thread which runs the default libvirt event loop implementation. This processes any incoming async events from libvirtd and queues them for later dispatch. This thread is only permitted to use libvirt python APIs, and the driver.queue_event method. In particular any use of logging is forbidden, since it will confuse eventlet's greenthread integration while this rule is adhered to by the _event_lifecycle_callback method, it is violated by _close_callback (the other callback) because it calls the _set_host_enabled method which, among other things, writes to the log. The _close_callback method needs to be altered so that it does not execute any logic that may interfere with eventlet's greenthread integration. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1290487 Title: Libvirt native thread used for "forbidden" purposes Status in OpenStack Compute (Nova): New Bug description: In the nova.virt.libvirt.driver.LibvirtDriver. _get_new_connection method two different libvirt event handlers are registered, one for lifecycle events (_event_lifecycle_callback) and one for connection events (_close_callback). These callbacks are called by a native thread that is continually calling libvirt.virEventRunDefaultImpl() in the _native_thread method; the latter method's Docstring contains the following note: This is a native thread which runs the default libvirt event loop implementation. This processes any incoming async events from libvirtd and queues them for later dispatch. This thread is only permitted to use libvirt python APIs, and the driver.queue_event method. In particular any use of logging is forbidden, since it will confuse eventlet's greenthread integration while this rule is adhered to by the _event_lifecycle_callback method, it is violated by _close_callback (the other callback) because it calls the _set_host_enabled method which, among other things, writes to the log. The _close_callback method needs to be altered so that it does not execute any logic that may interfere with eventlet's greenthread integration. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1290487/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1293641] [NEW] Libvirt's close callback causes deadlocks
Public bug reported: Libvirt's close callback causes deadlocks Unlike libvirt's lifecycle event callback which is triggered every time an event occurs, the close callback is only triggered when an attempt is made to use a connection that has been closed. In that case, the sequence of events is usually as follows: LibvirtDriver._get_connection acquires _wrapped_conn_lock LibvirtDriver._get_connection calls _test_connection LibvirtDriver._test_connection calls libvirt.getLibVersion libvirt.getLibVersion triggers LibvirtDriver._close_callback (because the connection is closed and this method is the registered handler) LibvirtDriver._close_callback attempts to acquire _wrapped_conn_lock _get_connection cannot release the lock because it is waiting for _close_callback to return and the latter cannot complete until it has acquired the lock. Making the handling of the close callback asynchronous (like the lifecycle event handling) won't work, because by the time the lock is released, the connection object that was passed into the callback will no longer be equal to LibvirtDriver._wrapped_conn. Even if the connection object is ignored, the instance will have already been disabled via the _get_new_connection method's existing error-handling logic. The best solution would appear to be to simply not register a close callback. The only case where it might provide some benefit is when a connection is closed after _get_connection has returned a reference to it. The benefit of disabling the instance a little earlier in such marginal cases is arguably outweighed by the complexity of a thread-safe implementation, especially when the difficulty of testing such an implementation (to ensure it is indeed thread safe) is taken into consideration. Note that having _close_callback use _wrapped_conn_lock.acquire(False) instead of "with _wrapped_conn_lock" by itself is not a viable solution, because due to connections being opened via tpool.proxy_call, the close callback is called by a native thread, which means it should not be used to perform the various operations (including logging) involved in disabling the instance. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1293641 Title: Libvirt's close callback causes deadlocks Status in OpenStack Compute (Nova): New Bug description: Libvirt's close callback causes deadlocks Unlike libvirt's lifecycle event callback which is triggered every time an event occurs, the close callback is only triggered when an attempt is made to use a connection that has been closed. In that case, the sequence of events is usually as follows: LibvirtDriver._get_connection acquires _wrapped_conn_lock LibvirtDriver._get_connection calls _test_connection LibvirtDriver._test_connection calls libvirt.getLibVersion libvirt.getLibVersion triggers LibvirtDriver._close_callback (because the connection is closed and this method is the registered handler) LibvirtDriver._close_callback attempts to acquire _wrapped_conn_lock _get_connection cannot release the lock because it is waiting for _close_callback to return and the latter cannot complete until it has acquired the lock. Making the handling of the close callback asynchronous (like the lifecycle event handling) won't work, because by the time the lock is released, the connection object that was passed into the callback will no longer be equal to LibvirtDriver._wrapped_conn. Even if the connection object is ignored, the instance will have already been disabled via the _get_new_connection method's existing error-handling logic. The best solution would appear to be to simply not register a close callback. The only case where it might provide some benefit is when a connection is closed after _get_connection has returned a reference to it. The benefit of disabling the instance a little earlier in such marginal cases is arguably outweighed by the complexity of a thread- safe implementation, especially when the difficulty of testing such an implementation (to ensure it is indeed thread safe) is taken into consideration. Note that having _close_callback use _wrapped_conn_lock.acquire(False) instead of "with _wrapped_conn_lock" by itself is not a viable solution, because due to connections being opened via tpool.proxy_call, the close callback is called by a native thread, which means it should not be used to perform the various operations (including logging) involved in disabling the instance. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1293641/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchp
[Yahoo-eng-team] [Bug 1298420] [NEW] Libvirt's image caching fetches images multiple times
Public bug reported: When launching several VMs in rapid succession, it is possible that libvirt's image caching will fetch the same image serveral times. This can occur when all of the VMs in question are using the same base image and this base image has not been previously fetched. The inline fetch_func_sync method prevents multiple threads from fetching the same image at the same time, but it does not prevent a thread that is waiting to acquire the lock from fetching the image that was being fetched while the lock was still in use. This is because the presence of the image is checked only before the lock has been acquired, not after. ** Affects: nova Importance: Undecided Status: New ** Tags: libvirt -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1298420 Title: Libvirt's image caching fetches images multiple times Status in OpenStack Compute (Nova): New Bug description: When launching several VMs in rapid succession, it is possible that libvirt's image caching will fetch the same image serveral times. This can occur when all of the VMs in question are using the same base image and this base image has not been previously fetched. The inline fetch_func_sync method prevents multiple threads from fetching the same image at the same time, but it does not prevent a thread that is waiting to acquire the lock from fetching the image that was being fetched while the lock was still in use. This is because the presence of the image is checked only before the lock has been acquired, not after. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1298420/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1300380] [NEW] errors_out_migration decorator does not work with RPC calls
Public bug reported: The errors_out_migration decorator in nova/compute/manager.py is attempting to use positional arguments to get the migration parameter. However, at run time, the decorated methods are called via RPC calls which specify all of their arguments as keyword arguments. The decorator needs to be fixed so that it works with RPC calls. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1300380 Title: errors_out_migration decorator does not work with RPC calls Status in OpenStack Compute (Nova): New Bug description: The errors_out_migration decorator in nova/compute/manager.py is attempting to use positional arguments to get the migration parameter. However, at run time, the decorated methods are called via RPC calls which specify all of their arguments as keyword arguments. The decorator needs to be fixed so that it works with RPC calls. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1300380/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1308565] [NEW] Delete does not clean up pending resize files
Public bug reported: If an instance is deleted while a resize operation is in progress (i.e. before the VM state is RESIZED), the temporary files created during the resize operation (e.g. _resize with libvirt) are not cleaned up. This would seem to be related to bug 1285000, except in that case the temporary files are on a different node. Ideally a fix would address both. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1308565 Title: Delete does not clean up pending resize files Status in OpenStack Compute (Nova): New Bug description: If an instance is deleted while a resize operation is in progress (i.e. before the VM state is RESIZED), the temporary files created during the resize operation (e.g. _resize with libvirt) are not cleaned up. This would seem to be related to bug 1285000, except in that case the temporary files are on a different node. Ideally a fix would address both. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1308565/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1339823] [NEW] Nova compute process crash doesn't reset task state 'migrating'
Public bug reported: When a live migration (task state = migrating) task is running, and the compute service goes down, the instance remains in migrating state when the compute service restarts. The compute/manager.py _init_instance method seems to handle all states except migrating. It handles inactive migration (resize_migrating), but not live migration. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1339823 Title: Nova compute process crash doesn't reset task state 'migrating' Status in OpenStack Compute (Nova): New Bug description: When a live migration (task state = migrating) task is running, and the compute service goes down, the instance remains in migrating state when the compute service restarts. The compute/manager.py _init_instance method seems to handle all states except migrating. It handles inactive migration (resize_migrating), but not live migration. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1339823/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
