same here, did you manage to fix it ?
On Mon, Oct 28, 2013 at 3:13 PM, Kevin Weiler <kevin.wei...@imc-chicago.com> wrote: > Hi Josh, > > We did map it directly to the host, and it seems to work just fine. I > think this is a problem with how the container is accessing the rbd module. > > -- > > Kevin Weiler > > IT > > > IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL > 60606 | http://imc-chicago.com/ > > Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: > kevin.wei...@imc-chicago.com > > > > > > > > On 10/18/13 7:50 PM, "Josh Durgin" <josh.dur...@inktank.com> wrote: > >>On 10/18/2013 10:04 AM, Kevin Weiler wrote: >>> The kernel is 3.11.4-201.fc19.x86_64, and the image format is 1. I did, >>> however, try a map with an RBD that was format 2. I got the same error. >> >>To rule out any capability drops as the culprit, can you map an rbd >>image on the same host outside of a container? >> >>Josh >> >>> -- >>> >>> *Kevin Weiler* >>> >>> IT >>> >>> IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | Chicago, IL >>> 60606 | http://imc-chicago.com/ >>> >>> Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: >>> _kevin.wei...@imc-chicago.com <mailto:kevin.wei...@imc-chicago.com>_ >>> >>> >>> From: Gregory Farnum <g...@inktank.com <mailto:g...@inktank.com>> >>> Date: Friday, October 18, 2013 10:26 AM >>> To: Omar Marquez <omar.marq...@imc-chicago.com >>> <mailto:omar.marq...@imc-chicago.com>> >>> Cc: Kyle Bader <kyle.ba...@gmail.com <mailto:kyle.ba...@gmail.com>>, >>> Kevin Weiler <kevin.wei...@imc-chicago.com >>> <mailto:kevin.wei...@imc-chicago.com>>, "ceph-users@lists.ceph.com >>> <mailto:ceph-users@lists.ceph.com>" <ceph-users@lists.ceph.com >>> <mailto:ceph-users@lists.ceph.com>>, Khalid Goudeaux >>> <khalid.goude...@imc-chicago.com >>><mailto:khalid.goude...@imc-chicago.com>> >>> Subject: Re: [ceph-users] mounting RBD in linux containers >>> >>> What kernel are you running, and which format is the RBD image? I >>> thought we had a special return code for when the kernel doesn't support >>> the features used by that image, but that could be the problem. >>> -Greg >>> >>> On Thursday, October 17, 2013, Omar Marquez wrote: >>> >>> >>> Strace produces below: >>> >>> Š >>> >>> futex(0xb5637c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb56378, >>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1 >>> futex(0xb562f8, FUTEX_WAKE_PRIVATE, 1) = 1 >>> add_key(0x424408, 0x7fff82c4e210, 0x7fff82c4e140, 0x22, >>> 0xfffffffe) = 607085216 >>> stat("/sys/bus/rbd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 >>> *open("/sys/bus/rbd/add", O_WRONLY) = 3* >>> *write(3, "10.198.41.6:6789 >>> <http://10.198.41.6:6789>,10.198.41.8:678 >>> <http://10.198.41.8:678>"..., 96) = -1 EINVAL (Invalid >>>argument)* >>> close(3) = 0 >>> rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, >>> {SIG_DFL, [], 0}, 8) = 0 >>> rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, >>> 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0 >>> rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0 >>> clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, >>> parent_tidptr=0x7fff82c4e040) = 22 >>> wait4(22, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22 >>> rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, >>> NULL, 8) = 0 >>> rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, >>> 0x7fbf8a7efa90}, NULL, 8) = 0 >>> rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0 >>> write(2, "rbd: add failed: ", 17rbd: add failed: ) = 17 >>> write(2, "(22) Invalid argument", 21(22) Invalid argument) = >>>21 >>> write(2, "\n", 1 >>> ) = 1 >>> exit_group(1) = ? >>> +++ exited with 1 +++ >>> >>> >>> The app is run inside the container with setuid = 0 and the >>> container is able to mount all required filesystems Š could this >>> still be a capability problem ? Also I do not see any call to >>> capset() in the strafe log Š >>> >>> -- >>> Om >>> >>> >>> From: Kyle Bader <kyle.ba...@gmail.com> >>> Date: Thursday, October 17, 2013 5:08 PM >>> To: Kevin Weiler <kevin.wei...@imc-chicago.com> >>> Cc: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com>, Omar >>> Marquez <omar.marq...@imc-chicago.com>, Khalid Goudeaux >>> <khalid.goude...@imc-chicago.com> >>> Subject: Re: [ceph-users] mounting RBD in linux containers >>> >>> My first guess would be that it's due to LXC dropping capabilities, >>> I'd investigate whether CAP_SYS_ADMIN is being dropped. You need >>> CAP_SYS_ADMIN for mount and block ioctls, if the container doesn't >>> have those privs a map will likely fail. Maybe try tracing the >>> command with strace? >>> >>> On Thu, Oct 17, 2013 at 2:45 PM, Kevin Weiler >>> <kevin.wei...@imc-chicago.com> wrote: >>> >>> Hi all, >>> >>> We're trying to mount an rbd image inside of a linux container >>> that has been created with docker (https://www.docker.io/). We >>> seem to have access to the rbd kernel module from inside the >>> container: >>> >>> # lsmod | grep ceph >>> libceph 218854 1 rbd >>> libcrc32c 12603 3 xfs,libceph,dm_persistent_data >>> >>> And we can query the pool for available rbds and create rbds >>> from inside the container: >>> >>> # rbd -p dockers --id dockers --keyring >>> /etc/ceph/ceph.client.dockers.keyring create lxctest --size >>>51200 >>> # rbd -p dockers --id dockers --keyring >>> /etc/ceph/ceph.client.dockers.keyring ls >>> lxctest >>> >>> But for some reason, we can't seem to map the device to the >>> container: >>> >>> # rbd -p dockers --id dockers --keyring >>> /etc/ceph/ceph.client.dockers.keyring map lxctest >>> rbd: add failed: (22) Invalid argument >>> >>> I don't see anything particularly interesting in dmesg or >>> messages on either the container or the host box. Any ideas on >>> how to troubleshoot this? >>> >>> Thanks! >>> >>> >>> -- >>> >>> *Kevin Weiler* >>> >>> IT >>> >>> IMC Financial Markets | 233 S. Wacker Drive, Suite 4300 | >>> Chicago, IL 60606 | http://imc-chicago.com/ >>> >>> Phone: +1 312-204-7439 | Fax: +1 312-244-3301 | E-Mail: >>> _kevin.weiler@imc-chicago.com_ >>> >>> >>> >>>------------------------------------------------------------------------ >>> >>> The information in this e-mail is intended only for the person >>> or entity to which it is addressed. >>> >>> It may contain confidential and /or privileged material. If >>> someone other than the intended recipient should receive this >>> e-mail, he / she shall not be entitled to read, disseminate, >>> disclose or duplicate it. >>> >>> If you receive this e-mail unintentionally, please inform us >>> immediately by "re >>> >>> >>> >>>------------------------------------------------------------------------ >>> >>> The information in this e-mail is intended only for the person or >>> entity to which it is addressed. >>> >>> It may contain confidential and /or privileged material. If someone >>> other than the intended recipient should receive this e-mail, he / >>> she shall not be entitled to read, disseminate, disclose or >>> duplicate it. >>> >>> If you receive this e-mail unintentionally, please inform us >>> immediately by "reply" and then delete it from your system. Although >>> this information has been compiled with great care, neither IMC >>> Financial Markets & Asset Management nor any of its related entities >>> shall accept any responsibility for any errors, omissions or other >>> inaccuracies in this information or for the consequences thereof, >>> nor shall it be bound in any way by the contents of this e-mail or >>> its attachments. In the event of incomplete or incorrect >>> transmission, please return the e-mail to the sender and permanently >>> delete this message and any attachments. >>> >>> Messages and attachments are scanned for all known viruses. Always >>> scan attachments before opening them. >>> >>> >>> >>> -- >>> Software Engineer #42 @ http://inktank.com | http://ceph.com >>> >>> ------------------------------------------------------------------------ >>> >>> The information in this e-mail is intended only for the person or entity >>> to which it is addressed. >>> >>> It may contain confidential and /or privileged material. If someone >>> other than the intended recipient should receive this e-mail, he / she >>> shall not be entitled to read, disseminate, disclose or duplicate it. >>> >>> If you receive this e-mail unintentionally, please inform us immediately >>> by "reply" and then delete it from your system. Although this >>> information has been compiled with great care, neither IMC Financial >>> Markets & Asset Management nor any of its related entities shall accept >>> any responsibility for any errors, omissions or other inaccuracies in >>> this information or for the consequences thereof, nor shall it be bound >>> in any way by the contents of this e-mail or its attachments. In the >>> event of incomplete or incorrect transmission, please return the e-mail >>> to the sender and permanently delete this message and any attachments. >>> >>> Messages and attachments are scanned for all known viruses. Always scan >>> attachments before opening them. >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > > > ________________________________ > > The information in this e-mail is intended only for the person or entity to > which it is addressed. > > It may contain confidential and /or privileged material. If someone other > than the intended recipient should receive this e-mail, he / she shall not be > entitled to read, disseminate, disclose or duplicate it. > > If you receive this e-mail unintentionally, please inform us immediately by > "reply" and then delete it from your system. Although this information has > been compiled with great care, neither IMC Financial Markets & Asset > Management nor any of its related entities shall accept any responsibility > for any errors, omissions or other inaccuracies in this information or for > the consequences thereof, nor shall it be bound in any way by the contents of > this e-mail or its attachments. In the event of incomplete or incorrect > transmission, please return the e-mail to the sender and permanently delete > this message and any attachments. > > Messages and attachments are scanned for all known viruses. Always scan > attachments before opening them. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
