Public bug reported:
Failed to create image due to exception NotAuthenticated.
If we take a long time to upload a large image to store, it may trigger
an exception "NotAuthenticated: Authentication required" and the image
has been stuck in SAVING state, this due to the token expires after a
period of time(default is expiration=3600s -- 1 hour, this can be
modified in the keystone conf file).
PS:
Image Size: 80G
Take Time: 2 hours
Icehouse
Ceph backend
The error log below:
2015-12-12 18:45:00.416 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.417 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Failed to upload image
9366c0a1-40a9-4f4a-a821-2acac905ecd5
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 152, in
upload_data_to_store
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state='saving')
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 527, in
_do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils raise
exception.NotAuthenticated(res.read())
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils NotAuthenticated:
Authentication required
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
2015-12-12 18:45:00.447 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.448 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Unable to kill image
9366c0a1-40a9-4f4a-a821-2acac905ecd5:
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 76, in
safe_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils _kill(req,
image_id, from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 61, in
_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
** Affects: glance
Importance: Undecided
Status: New
** Description changed:
Failed to create image due to exception NotAuthenticated.
If we take a long time to upload a large image to store, it may trigger
an exception "NotAuthenticated: Authentication required" and the image
has been stuck in SAVING state, this due to the token expires after a
period of time(default is expiration=3600s -- 1 hour, this can be
modified in the keystone conf file).
PS:
Image Size: 80G
- TakeTime: 2 hours
+ Take Time: 2 hours
- Icehouse
+ Icehouse
Ceph backend
The error log below:
2015-12-12 18:45:00.416 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.417 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Failed to upload image
9366c0a1-40a9-4f4a-a821-2acac905ecd5
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 152, in
upload_data_to_store
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state='saving')
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 527, in
_do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils raise
exception.NotAuthenticated(res.read())
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
NotAuthenticated: Authentication required
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
2015-12-12 18:45:00.447 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.448 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Unable to kill image
9366c0a1-40a9-4f4a-a821-2acac905ecd5:
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 76, in
safe_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils _kill(req,
image_id, from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 61, in
_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1530328
Title:
Failed to create image due to exception NotAuthenticated
Status in Glance:
New
Bug description:
Failed to create image due to exception NotAuthenticated.
If we take a long time to upload a large image to store, it may
trigger an exception "NotAuthenticated: Authentication required" and
the image has been stuck in SAVING state, this due to the token
expires after a period of time(default is expiration=3600s -- 1 hour,
this can be modified in the keystone conf file).
PS:
Image Size: 80G
Take Time: 2 hours
Icehouse
Ceph backend
The error log below:
2015-12-12 18:45:00.416 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.417 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Failed to upload image
9366c0a1-40a9-4f4a-a821-2acac905ecd5
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 152, in
upload_data_to_store
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state='saving')
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 527, in
_do_request
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils raise
exception.NotAuthenticated(res.read())
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
NotAuthenticated: Authentication required
2015-12-12 18:45:00.417 6370 TRACE glance.api.v1.upload_utils
2015-12-12 18:45:00.447 6370 INFO glance.registry.client.v1.client
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Registry client request PUT
/images/9366c0a1-40a9-4f4a-a821-2acac905ecd5 raised NotAuthenticated
2015-12-12 18:45:00.448 6370 ERROR glance.api.v1.upload_utils
[b7ee5de9-addf-457c-88a8-552476044f0d 44edf16aa49f4469a76c90c984fe7ce7
b10adc7c9afd4506b26d42779c77b2f4 - - -] Unable to kill image
9366c0a1-40a9-4f4a-a821-2acac905ecd5:
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils Traceback (most
recent call last):
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 76, in
safe_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils _kill(req,
image_id, from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 61, in
_kill
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/api.py", line 169,
in update_image_metadata
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
from_state=from_state)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
190, in update_image
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=headers)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/registry/client/v1/client.py", line
107, in do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 69, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, *args, **kwargs)
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 386, in
do_request
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils
headers=copy.deepcopy(headers))
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.6/site-packages/glance/common/client.py", line 83, in wrapped
2015-12-12 18:45:00.448 6370 TRACE glance.api.v1.upload_utils return
func(self, method, url, body, headers)
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1530328/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
