So I finally got Ceph and cinder bootable volume to work. For those interested on what I had to do. I had to reinstall everything and before I installed the nova components I installed the latest ceph client emperor. Now in the nova install there are some ceph components that it will install. If you now install nova it will error out about some broken dependency and ceph-libs. Do not install ceph-libs, all you need to install is qemu-common. If you do a yum install it will try to install ceph-libs so do not install via yum. Download the rpm and install the rpm of qemu-common with the --nodeps option. Then run everything else. Do everything in the ceph documentation and make sure the client.volumes or whatever cinder user name you call it has access to both the glance and cinder pools you make. Done!
Edward Lezondra IT IMC Financial Markets | 233 South Wacker Drive # 4300 | Chicago, IL 60606 | http://www.imc-chicago.com Phone: +13122755425 | E-Mail: edward.lezon...@imc-chicago.com -----Original Message----- From: Alvin Starr [mailto:al...@netvel.net] Sent: Wednesday, December 11, 2013 12:30 PM To: Edward Lezondra; Haomai Wang Cc: Openstack@lists.openstack.org Subject: Re: [Openstack] Ceph bootable cinder volume not working in Havana I have noticed a similar problem. I have 2 different pools: images and volumes and both use the same ceph.user keyring. I can create a volume from an image but if I try to cause the image to copy as part of an instance start up things. fail. I wonder if the problem is using 2 different pools? On 12/11/2013 09:58 AM, Edward Lezondra wrote: > This is my cinder.conf > > [DEFAULT] > > > api_paste_config=/etc/cinder/api-paste.ini > state_path = /var/lib/cinder > glance_host=localhost > glance_api_version=2 > > rootwrap_config=/etc/cinder/rootwrap.conf > auth_strategy=keystone > > lock_path=/var/lock/cinder > > qpid_hostname=localhost > qpid_port=5672 > qpid_heartbeat=60 > qpid_tcp_nodelay=true > > iscsi_ip_address=10.198.0.191 > iscsi_helper=tgtadm > > volumes_dir = /var/lib/cinder/volumes > volume_group=cinder-volumes > > rbd_pool=nova > rbd_user=nova > rbd_ceph_conf=/etc/ceph/ceph.conf > rbd_secret_uuid=ae396956-cf03-483f-b356-2f447dc7a637 > volume_driver=cinder.volume.drivers.rbd.RBDDriver > > sql_connection = mysql://cinder:notcinder@173.194.106.60/cinder > my_ip = 10.198.0.191 > rpc_backend = cinder.openstack.common.rpc.impl_qpid > qpid_hostname = 10.198.0.191 > notification_driver=cinder.openstack.common.notifier.rpc_notifier > > Edward Lezondra > IT > > IMC Financial Markets | 233 South Wacker Drive # 4300 | Chicago, IL 60606 | > http://www.imc-chicago.com > Phone: +13122755425 | E-Mail: edward.lezon...@imc-chicago.com > > -----Original Message----- > From: Haomai Wang [mailto:haomaiw...@gmail.com] > Sent: Monday, December 09, 2013 10:03 PM > To: Edward Lezondra > Cc: Openstack@lists.openstack.org > Subject: Re: [Openstack] Ceph bootable cinder volume not working in Havana > > Could you give out your cinder.conf? > > On Tue, Dec 10, 2013 at 4:10 AM, Edward Lezondra > <edward.lezon...@imc-chicago.com> wrote: >> Hello, >> >> >> >> I’m having trouble getting a bootable cinder volume to work in Havana >> using Ceph. I managed to get glance working correctly. I got cinder >> volume to work, but once I try to create bootable cinder volume to an >> image it errors out. >> >> My cinder.conf reflect exactly what the ceph docs tell one to do to >> get it working. I was able to get this to work in grizzly using the >> CEPH_ARGS="--id volumes" statement on the volume service, but the docs >> and as a thread in the mailing list who got his to work didn’t use the >> CEPH_ARGS="--id volumes" anymore for Havana. Judging from the logs I have >> permission issue on the rbd image. I made sure my keyrings have the >> proper ownership and gave full permissions to everyone to them. >> >> My logs on cinder/volume shows the following: >> >> >> >> 2013-12-09 12:29:12.377 8468 ERROR cinder.volume.drivers.rbd >> [req-75308325-0b69-41f9-8099-1c9d64c025b6 >> ce3265c65e304ad3b018a9fba063df88 0c3cbff39d97448d8cff1816793aea94] >> error opening rbd image >> 9e967bd7-ed6c-41bc-932d-1157bd72b778 >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd Traceback >> (most recent call last): >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 209, in __init__ >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd >> read_only=read_only) >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd File >> "/usr/lib/python2.7/site-packages/rbd.py", line 351, in __init__ >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd raise >> make_ex(ret, 'error opening image %s at snapshot %s' % (name, >> snapshot)) >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd >> PermissionError: error opening image >> 9e967bd7-ed6c-41bc-932d-1157bd72b778 at snapshot snap >> >> 2013-12-09 12:29:12.377 8468 TRACE cinder.volume.drivers.rbd >> >> 2013-12-09 12:29:12.380 8468 WARNING cinder.taskflow.utils [-] >> Activating 5 rollbacks due to <cinder.taskflow.utils.FlowFailure object at >> 0x328c690>. >> >> 2013-12-09 12:29:12.485 8468 ERROR cinder.volume.flows.create_volume >> [req-75308325-0b69-41f9-8099-1c9d64c025b6 >> ce3265c65e304ad3b018a9fba063df88 0c3cbff39d97448d8cff1816793aea94] >> Unexpected build error: >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> Traceback (most recent call last): >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/patterns/linear_flow >> .py", >> line 172, in run_it >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> result = runner(context, *args, **kwargs) >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/utils.py", line 260, >> in __call__ >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> self.result = self.task(*args, **kwargs) >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__ >> init__.py", >> line 1499, in __call__ >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> **volume_spec) >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__ >> init__.py", >> line 1441, in _create_from_image >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> volume_ref, image_location, image_id) >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 729, in clone_image >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume if >> image_location is None or not self._is_cloneable(image_location): >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 720, in _is_cloneable >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> read_only=True): >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 213, in __init__ >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> raise >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> TypeError: exceptions must be old-style classes or derived from >> BaseException, not NoneType >> >> 2013-12-09 12:29:12.485 8468 TRACE cinder.volume.flows.create_volume >> >> 2013-12-09 12:29:12.486 8468 ERROR cinder.openstack.common.rpc.amqp >> [req-75308325-0b69-41f9-8099-1c9d64c025b6 >> ce3265c65e304ad3b018a9fba063df88 0c3cbff39d97448d8cff1816793aea94] >> Exception during message handling >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> Traceback (most recent call last): >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py" >> , line 441, in _process_data >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> **args) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatch >> er.py", >> line 148, in dispatch >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> return getattr(proxyobj, method)(ctxt, **kwargs) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/utils.py", line 808, in >> wrapper >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> return func(self, *args, **kwargs) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 257, >> in create_volume >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> flow.run(context.elevated()) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line >> 105, in wrapper >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> return f(self, *args, **kwargs) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/patterns/linear_flow >> .py", >> line 232, in run >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> run_it(r) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/patterns/linear_flow >> .py", >> line 212, in run_it >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> self.rollback(context, cause) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> self.gen.next() >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/patterns/linear_flow >> .py", >> line 172, in run_it >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> result = runner(context, *args, **kwargs) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/taskflow/utils.py", line 260, >> in __call__ >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> self.result = self.task(*args, **kwargs) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__ >> init__.py", >> line 1499, in __call__ >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> **volume_spec) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__ >> init__.py", >> line 1441, in _create_from_image >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> volume_ref, image_location, image_id) >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 729, in clone_image >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp if >> image_location is None or not self._is_cloneable(image_location): >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 720, in _is_cloneable >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> read_only=True): >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp File >> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line >> 213, in __init__ >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> raise >> >> 2013-12-09 12:29:12.486 8468 TRACE cinder.openstack.common.rpc.amqp >> TypeError: exceptions must be old-style classes or derived from >> BaseException, not NoneType >> >> >> >> >> ________________________________ >> >> The information in this e-mail is intended only for the person or >> entity to which it is addressed. >> >> It may contain confidential and /or privileged material. If someone >> other than the intended recipient should receive this e-mail, he / she >> shall not be entitled to read, disseminate, disclose or duplicate it. >> >> If you receive this e-mail unintentionally, please inform us >> immediately by "reply" and then delete it from your system. Although >> this information has been compiled with great care, neither IMC >> Financial Markets & Asset Management nor any of its related entities >> shall accept any responsibility for any errors, omissions or other >> inaccuracies in this information or for the consequences thereof, nor >> shall it be bound in any way by the contents of this e-mail or its >> attachments. In the event of incomplete or incorrect transmission, >> please return the e-mail to the sender and permanently delete this message >> and any attachments. >> >> Messages and attachments are scanned for all known viruses. Always >> scan attachments before opening them. >> >> _______________________________________________ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > > > -- > Best Regards, > > Wheat > > ________________________________ > > The information in this e-mail is intended only for the person or entity to > which it is addressed. > > It may contain confidential and /or privileged material. If someone other > than the intended recipient should receive this e-mail, he / she shall not be > entitled to read, disseminate, disclose or duplicate it. > > If you receive this e-mail unintentionally, please inform us immediately by > "reply" and then delete it from your system. Although this information has > been compiled with great care, neither IMC Financial Markets & Asset > Management nor any of its related entities shall accept any responsibility > for any errors, omissions or other inaccuracies in this information or for > the consequences thereof, nor shall it be bound in any way by the contents of > this e-mail or its attachments. In the event of incomplete or incorrect > transmission, please return the e-mail to the sender and permanently delete > this message and any attachments. > > Messages and attachments are scanned for all known viruses. Always scan > attachments before opening them. > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net || ________________________________ The information in this e-mail is intended only for the person or entity to which it is addressed. It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it. If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments. Messages and attachments are scanned for all known viruses. Always scan attachments before opening them. _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
