[ceph-users] 回复: scrub error with keyvalue backend

[廖建锋]

is there anybody can help ?


发件人： ceph-users
发送时间： 2014-10-10 13:34
收件人： ceph-users
主题： [ceph-users] scrub error with keyvalue backend
Dear ceph,

 # ceph -s
cluster e1f18421-5d20-4c3e-83be-a74b77468d61
health HEALTH_ERR 4 pgs inconsistent; 4 scrub errors
monmap e2: 3 mons at 
{storage-1-213=10.1.0.213:6789/0,storage-1-214=10.1.0.214:6789/0,storage-1-215=10.1.0.215:6789/0},
 election epoch 16, quorum 0,1,2 storage-1-213,storage-1-214,storage-1-215
mdsmap e7: 1/1/1 up {0=storage-1-213=up:active}, 2 up:standby
osdmap e135: 18 osds: 18 up, 18 in
pgmap v84135: 1164 pgs, 3 pools, 801 GB data, 15264 kobjects
1853 GB used, 34919 GB / 36772 GB avail
1159 active+clean
4 active+clean+inconsistent
1 active+clean+scrubbing
client io 17400 kB/s wr, 611 op/s

[root@storage-1-213:~] [Fri Oct 10 - 13:30:19]
999 => # ceph -v
ceph version 0.80.6 (f93610a4421cb670b08e974c6550ee715ac528ae)

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Ilya Dryomov]

On Fri, Oct 10, 2014 at 12:48 AM, Aquino, Ben O  wrote:
> Hello Ceph Users:
>
>
>
> Ceph baremetal client attempting to map device volume via kernel RBD Driver,
> resulting in unable to map device volume and outputs I/O error.
>
> This is Ceph client only, no MDS,OSD or MON running…see I/O error output
> below.
>
>
>
>
>
> Client Host Linux Kernel Version :
>
> [root@root ceph]# uname -a
>
> Linux root 3.10.25-11.el6.centos.alt.x86_64 #1 SMP Fri Dec 27 21:44:15 UTC
> 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> Ceph Version:
>
> [root@root ceph]# ceph -v
>
> ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
>
>
>
> Check Kernel RBD driver:
>
> [root@root ceph]# locate rbd.ko
>
> /lib/modules/3.10.25-11.el6.centos.alt.x86_64/kernel/drivers/block/rbd.ko
>
> /lib/modules/3.10.25-11.el6.centos.alt.x86_64/kernel/drivers/block/drbd/drbd.ko
>
>
>
> Check Client to Ceph-Server Connections:
>
> [root@root ceph]# ceph osd lspools
>
> 0 data,1 metadata,2 rbd,3 vsmpool_hp1,4 vsmpool_perf1,5 vsmpool_vperf1,6
> openstack_hp1,7 openstack_perf1,8 openstack_vperf1,9 vsmpool_perf2,10
> vsmpool_hp2,11 vsmpool_vperf2,12 testopnstack,13 ec_perf_pool,14
> ec_perf_pool1,15 ec_perf_pool2,16 ec_hiperf_pool1,17 ec_valperf_pool1,
>
>
>
> Created RBD:
>
> [root@root ceph]# rbd create rbd9  --size 104800 --pool vsmpool_hp1 --id
> admin
>
>
>
> Check RBD:
>
> [root@root ceph]# rbd ls vsmpool_hp1
>
> rbd1
>
> rbd2
>
> rbd3
>
> rbd4
>
> rbd5
>
> rbd6
>
> rbd7
>
> rbd8
>
> rbd9
>
>
>
> Display RBD INFO:
>
> [root@root ceph]# rbd info vsmpool_hp1/rbd9
>
> rbd image 'rbd9':
>
> size 102 GB in 26200 objects
>
> order 22 (4096 kB objects)
>
> block_name_prefix: rb.0.227915.238e1f29
>
> format: 1
>
>
>
> Map RBD:
>
> [root@root ceph]# rbd map vsmpool_hp1/rbd9 --id admin
>
> rbd: add failed: (5) Input/output error

Is there anything in dmesg?  I'm pretty sure you'll see something like
"feature set mismatch" in there if you look.  Please paste that.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 回复: scrub error with keyvalue backend

[Haomai Wang]

Hi,

keyvaluestore is a experiment backend and isn't suitable for
non-developer to use. If you want to experience keyvaluestore, you
need to compile the newest codes or get the newest release packages.

On Fri, Oct 10, 2014 at 4:09 PM, 廖建锋  wrote:
> is there anybody can help ?
>
>
> 发件人： ceph-users
> 发送时间： 2014-10-10 13:34
> 收件人： ceph-users
> 主题： [ceph-users] scrub error with keyvalue backend
> Dear ceph,
>
>  # ceph -s
> cluster e1f18421-5d20-4c3e-83be-a74b77468d61
> health HEALTH_ERR 4 pgs inconsistent; 4 scrub errors
> monmap e2: 3 mons at
> {storage-1-213=10.1.0.213:6789/0,storage-1-214=10.1.0.214:6789/0,storage-1-215=10.1.0.215:6789/0},
> election epoch 16, quorum 0,1,2 storage-1-213,storage-1-214,storage-1-215
> mdsmap e7: 1/1/1 up {0=storage-1-213=up:active}, 2 up:standby
> osdmap e135: 18 osds: 18 up, 18 in
> pgmap v84135: 1164 pgs, 3 pools, 801 GB data, 15264 kobjects
> 1853 GB used, 34919 GB / 36772 GB avail
> 1159 active+clean
> 4 active+clean+inconsistent
> 1 active+clean+scrubbing
> client io 17400 kB/s wr, 611 op/s
>
> [root@storage-1-213:~] [Fri Oct 10 - 13:30:19]
> 999 => # ceph -v
> ceph version 0.80.6 (f93610a4421cb670b08e974c6550ee715ac528ae)
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Best Regards,

Wheat
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000

[SCHAER Frederic]

-Message d'origine-
De : Loic Dachary [mailto:l...@dachary.org] 

The failure 

journal check: ondisk fsid ---- doesn't match 
expected 244973de-7472-421c-bb25-4b09d3f8d441

and the udev logs

DEBUG:ceph-disk:Journal /dev/sdc2 has OSD UUID 
----

means /usr/bin/ceph-osd -i 0 --get-journal-uuid --osd-journal /dev/sdc2

fails to read the OSD UUID from /dev/sdc2 which means something went wrong when 
preparing the journal.

It would be great if you could send the command you used to prepare the disk 
and the output (verbose if possible). I think you can reproduce the problem by 
zapping the disk with ceph-disk zap /dev/sdc and running partx -u if the 
corresponding entries in /dev/disk/by-partuuid have not been removed. That 
would also help me fix zap in the context of 
https://github.com/ceph/ceph/pull/2648 ... or have confirmation that it does 
not need fixing because it updates correctly on RHEL ;-)

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

[>- FS : -<] 
Hi Loic,

At first, some notes :
-  I noticed I have to wait at least 1sec before I run partx -u on a prepared 
disk, otherwise even with that disks won't get properly handled by udev.
Maybe some caching somewhere ?
- the '-u' option does not seem to exist under RHEL6... so maybe the RHEL6 
behaviour was just to include the kernel updating in the -a option, and not 
anymore ?
- this is CentOS 7, i.e RHEL like, but not "pure" RHEL7 even if very close.


I Zapped the disk (it seems to work as expected) : 
[root@ceph1 ~]# ll /dev/disk/by-partuuid/
total 0
lrwxrwxrwx 1 root root 10 Oct  9 15:57 668f92f5-df46-4052-92ba-e8b8f7efd2d9 -> 
../../sdb1
lrwxrwxrwx 1 root root 10 Oct  9 15:57 feb09ba1-30a2-44a8-a338-fef39ae6626a -> 
../../sdb2
[root@ceph1 ~]

partx -u :

[root@ceph1 ~]# partx -u /dev/sdc
partx: specified range <1:0> does not make sense


Right after that, I re-prepared the disk :

[root@ceph1 ~]# parted -s /dev/sdc mklabel gpt
[root@ceph1 ~]# ceph-disk -v  prepare /dev/sdc
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph 
--show-config-value=fsid
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_mkfs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_fs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_fs_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. 
--lookup osd_fs_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph 
--show-config-value=osd_journal_size
INFO:ceph-disk:Will colocate journal with data on /dev/sdc
DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/sdc
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:5120M 
--change-name=2:ceph journal 
--partition-guid=2:46bf261f-7ec3-485e-98c9-3c185de5efb8 
--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Information: Moved requested sector from 34 to 2048 in
order to align on 2048-sector boundaries.
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/sdc
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /usr/sbin/partx -a /dev/sdc
partx: /dev/sdc: error adding partition 2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Journal is GPT partition 
/dev/disk/by-partuuid/46bf261f-7ec3-485e-98c9-3c185de5efb8
DEBUG:ceph-disk:Creating osd partition on /dev/sdc
INFO:ceph-disk:Running command: /usr/sbin/sgdisk --largest-new=1 
--change-name=1:ceph data 
--partition-guid=1:436ac41b-8800-466e-98f5-098aa2c64ba9 
--typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/sdc
Information: Moved requested sector from 10485761 to 10487808 in
order to align on 2048-sector boundaries.
The operation has completed successfully.
INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/sdc
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Creating xfs fs on /dev/sdc1
INFO:ceph-disk:Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- 
/dev/sdc1
meta-data=/dev/sdc1  isize=2048   agcount=4, agsize=60686271 blks
 =   sectsz=512   attr=2, projid32bit=1
 =   crc=0
data =   bsize=4096   blocks=242745083, imaxpct=25
 =   sunit=0  swidth=0 blks
naming   =version 2  bsize=4096   ascii-ci=0 ftype=0
log  =internal log   bsize=4096   blocks=118527, version=2
 =   sectsz=512   sunit=0 blks, lazy-count=1
realtime =none   extsz=4096   blocks=0,

Re: [ceph-users] v0.86 released (Giant release candidate)

[Florian Haas]

Hi Sage,

On Tue, Oct 7, 2014 at 9:20 PM, Sage Weil  wrote:
> This is a release candidate for Giant, which will hopefully be out in
> another week or two (s v0.86).  We did a feature freeze about a month ago
> and since then have been doing only stabilization and bug fixing (and a
> handful on low-risk enhancements).  A fair bit of new functionality went
> into the final sprint, but it's baked for quite a while now and we're
> feeling pretty good about it.
>
> Major items include:
>
> * librados locking refactor to improve scaling and client performance
> * local recovery code (LRC) erasure code plugin to trade some
>   additional storage overhead for improved recovery performance
> * LTTNG tracing framework, with initial tracepoints in librados,
>   librbd, and the OSD FileStore backend
> * separate monitor audit log for all administrative commands
> * asynchronos monitor transaction commits to reduce the impact on
>   monitor read requests while processing updates
> * low-level tool for working with individual OSD data stores for
>   debugging, recovery, and testing
> * many MDS improvements (bug fixes, health reporting)
>
> There are still a handful of known bugs in this release, but nothing
> severe enough to prevent a release.  By and large we are pretty
> pleased with the stability and expect the final Giant release to be
> quite reliable.
>
> Please try this out on your non-production clusters for a preview.

Thanks for the summary! Since you mentioned MDS improvements, and just
so it doesn't get lost: as you hinted at in off-list email, please do
provide a write-up of CephFS features expected to work in Giant at the
time of the release (broken down by kernel client vs. Ceph-FUSE, if
necessary). Not in the sense that anyone is offering commercial
support, but in the sense of "if you use this limited feature set, we
are confident that it at least won't eat your data." I think that
would be beneficial to a large portion of the user base, and clear up
a lot of the present confusion about the maturity and stability of the
filesystem.

Cheers,
Florian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.86 released (Giant release candidate)

[Wido den Hollander]

On 10/10/2014 11:26 AM, Florian Haas wrote:
> Hi Sage,
> 
> On Tue, Oct 7, 2014 at 9:20 PM, Sage Weil  wrote:
>> This is a release candidate for Giant, which will hopefully be out in
>> another week or two (s v0.86).  We did a feature freeze about a month ago
>> and since then have been doing only stabilization and bug fixing (and a
>> handful on low-risk enhancements).  A fair bit of new functionality went
>> into the final sprint, but it's baked for quite a while now and we're
>> feeling pretty good about it.
>>
>> Major items include:
>>
>> * librados locking refactor to improve scaling and client performance
>> * local recovery code (LRC) erasure code plugin to trade some
>>   additional storage overhead for improved recovery performance
>> * LTTNG tracing framework, with initial tracepoints in librados,
>>   librbd, and the OSD FileStore backend
>> * separate monitor audit log for all administrative commands
>> * asynchronos monitor transaction commits to reduce the impact on
>>   monitor read requests while processing updates
>> * low-level tool for working with individual OSD data stores for
>>   debugging, recovery, and testing
>> * many MDS improvements (bug fixes, health reporting)
>>
>> There are still a handful of known bugs in this release, but nothing
>> severe enough to prevent a release.  By and large we are pretty
>> pleased with the stability and expect the final Giant release to be
>> quite reliable.
>>
>> Please try this out on your non-production clusters for a preview.
> 
> Thanks for the summary! Since you mentioned MDS improvements, and just
> so it doesn't get lost: as you hinted at in off-list email, please do
> provide a write-up of CephFS features expected to work in Giant at the
> time of the release (broken down by kernel client vs. Ceph-FUSE, if
> necessary). Not in the sense that anyone is offering commercial
> support, but in the sense of "if you use this limited feature set, we
> are confident that it at least won't eat your data." I think that
> would be beneficial to a large portion of the user base, and clear up
> a lot of the present confusion about the maturity and stability of the
> filesystem.
> 

Agreed, that would be very useful. In some deployment people want to use
CephFS (with all the risks that come with it), but the limitations
should be known(ish).

It's probably that you should stay away from snapshots and multi-MDS,
but there is probably more!

> Cheers,
> Florian
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rados Gateway and Swift create containers/buckets that cannot be opened

[M Ranga Swami Reddy]

Yehuda -  With this fix work and removed the "WSGIChunkedRequest On "
from the configuration.  Its work  fine for me without error.

Thanks
Swami

On Thu, Oct 9, 2014 at 11:49 PM, Yehuda Sadeh  wrote:
> Here's the fix, let me know if you need any help with that.
>
> Thanks,
> Yehuda
>
> diff --git a/src/rgw/rgw_swift.cc b/src/rgw/rgw_swift.cc
> index d9654a7..2445e17 100644
> --- a/src/rgw/rgw_swift.cc
> +++ b/src/rgw/rgw_swift.cc
> @@ -505,6 +505,8 @@ int RGWSwift::validate_keystone_token(RGWRados
> *store, const string& token, stru
>
>  validate.append_header("X-Auth-Token", admin_token);
>
> +validate.set_send_length(0);
> +
>  int ret = validate.process(url.c_str());
>  if (ret < 0)
>return ret;
>
>
>
> On Thu, Oct 9, 2014 at 10:30 AM, M Ranga Swami Reddy
>  wrote:
>> Hi Yehuda,
>> Please share the fix/patch, we could test and confirm the fix status.
>>
>> Thanks
>> Swami
>>
>> On Thu, Oct 9, 2014 at 10:42 PM, Yehuda Sadeh  wrote:
>>> I have a trivial fix for the issue that I'd like to check and get this
>>> one cleared, but never got to it due to some difficulties with a
>>> proper keystone setup in my environment. If you can and would like to
>>> test it so that we could get it merged it would be great.
>>>
>>> Thanks,
>>> Yehuda
>>>
>>> On Wed, Oct 8, 2014 at 6:18 PM, Mark Kirkwood
>>>  wrote:
 Yes. I ran into that as well - I used

 WSGIChunkedRequest On

 in the virtualhost config for the *keystone* server [1] as indicated in
 issue 7796.

 Cheers

 Mark

 [1] i.e, not the rgw.

 On 08/10/14 22:58, Ashish Chandra wrote:
>
> Hi Mark,
> Good you got the solution. But since you have already done
> authenticating RadosGW with Keystone, I am having one issue that you can
> help with. For me I get an error "411 Length Required" with Keystone
> token authentication.
> To fix this I use "WSGIChunkedRequest On" in rgw.conf as mentioned in
> http://tracker.ceph.com/issues/7796.
>
> Did you face the issue, if yes what was your solution.
>
>


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000

[Loic Dachary]

Hi Frederic,

Thanks for the additional information.

On 10/10/2014 10:58, SCHAER Frederic wrote:
> [root@ceph1 ~]# parted -s /dev/sdc mklabel gpt

What happens if you don't do that ? ceph-disk should be able to handle a disk 
without this step.

Other than this the preparation looks fine.

> It's funny : I retried and ceph-disk prepare worked this time. I tried on 
> another disk, and it failed.
> There is a difference in the output from ceph-disk : on the failing disk, I 
> have these extra lines after disks are prepared :

> (...)
> realtime =none   extsz=4096   blocks=0, rtextents=0
> Warning: The kernel is still using the old partition table.
> The new table will be used at the next reboot.
> The operation has completed successfully.
> partx: /dev/sdc: error adding partitions 1-2

Since it worked when you repeated the command prepare on the previous disk, 
maybe it will also work on this disk ? Could you try and send the output of

ceph-disk -v  prepare /dev/sdc

as well as the udev debug output when you try that ? 

Maybe it's related to the fact that blkid does not have ID_PART_ENTRY_ (could 
it be that the root cause it that udev itself does not support it or partially 
supports it) ? Maybe it's a race condition if it works the second time around. 

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000

[Loic Dachary]

Hi Frederic,

It looks like this is just because 
https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284 should call partx 
instead of partprobe. The udev debug output makes this quite clear 
http://tracker.ceph.com/issues/9721

I think 
https://github.com/dachary/ceph/commit/8d914001420e5bfc1e12df2d4882bfe2e1719a5c#diff-788c3cea6213c27f5fdb22f8337096d5R1285
 fixes it

Cheers

On 09/10/2014 16:29, SCHAER Frederic wrote:
> 
> 
> -Message d'origine-
> De : Loic Dachary [mailto:l...@dachary.org] 
> Envoyé : jeudi 9 octobre 2014 16:20
> À : SCHAER Frederic; ceph-users@lists.ceph.com
> Objet : Re: [ceph-users] ceph-dis prepare : 
> UUID=----
> 
> 
> 
> On 09/10/2014 16:04, SCHAER Frederic wrote:
>> Hi Loic,
>>
>> Back on sdb, as the sde output was from another machine on which I ran partx 
>> -u afterwards.
>> To reply your last question first : I think the SG_IO error comes from the 
>> fact that disks are exported as a single disks RAID0 on a PERC 6/E, which 
>> does not support JBOD - this is decommissioned hardware on which I'd like to 
>> test and validate we can use ceph for our use case...
>>
>> So back on the  UUID.
>> It's funny : I retried and ceph-disk prepare worked this time. I tried on 
>> another disk, and it failed.
>> There is a difference in the output from ceph-disk : on the failing disk, I 
>> have these extra lines after disks are prepared :
>>
>> (...)
>> realtime =none   extsz=4096   blocks=0, rtextents=0
>> Warning: The kernel is still using the old partition table.
>> The new table will be used at the next reboot.
>> The operation has completed successfully.
>> partx: /dev/sdc: error adding partitions 1-2
>>
>> I didn't have the warning about the old partition tables on the disk that 
>> worked. 
>> So on this new disk, I have :
>>
>> [root@ceph1 ~]# mount /dev/sdc1 /mnt
>> [root@ceph1 ~]# ll /mnt/
>> total 16
>> -rw-r--r-- 1 root root 37 Oct  9 15:58 ceph_fsid
>> -rw-r--r-- 1 root root 37 Oct  9 15:58 fsid
>> lrwxrwxrwx 1 root root 58 Oct  9 15:58 journal -> 
>> /dev/disk/by-partuuid/5e50bb8b-0b99-455f-af71-10815a32bfbc
>> -rw-r--r-- 1 root root 37 Oct  9 15:58 journal_uuid
>> -rw-r--r-- 1 root root 21 Oct  9 15:58 magic
>>
>> [root@ceph1 ~]# cat /mnt/journal_uuid
>> 5e50bb8b-0b99-455f-af71-10815a32bfbc
>>
>> [root@ceph1 ~]# sgdisk --info=1 /dev/sdc
>> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
>> Partition unique GUID: 244973DE-7472-421C-BB25-4B09D3F8D441
>> First sector: 10487808 (at 5.0 GiB)
>> Last sector: 1952448478 (at 931.0 GiB)
>> Partition size: 1941960671 sectors (926.0 GiB)
>> Attribute flags: 
>> Partition name: 'ceph data'
>>
>> [root@ceph1 ~]# sgdisk --info=2 /dev/sdc
>> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
>> Partition unique GUID: 5E50BB8B-0B99-455F-AF71-10815A32BFBC
>> First sector: 2048 (at 1024.0 KiB)
>> Last sector: 10485760 (at 5.0 GiB)
>> Partition size: 10483713 sectors (5.0 GiB)
>> Attribute flags: 
>> Partition name: 'ceph journal'
>>
>> Puzzling, isn't it ?
>>
>>
> 
> Yes :-) Just to be 100% sure, when you try to activate this /dev/sdc it shows 
> an error and complains that the journal uuid is -000* etc ? If so could 
> you copy your udev debug output ?
> 
> Cheers
> 
> [>- FS : -<]  
> 
> No, when I manually activate the disk instead of attempting to go the udev 
> way, it seems to work :
> [root@ceph1 ~]# ceph-disk activate /dev/sdc1
> got monmap epoch 1
> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 2014-10-09 16:21:43.286288 7f2be6a027c0 -1 journal check: ondisk fsid 
> ---- doesn't match expected 
> 244973de-7472-421c-bb25-4b09d3f8d441, invalid (someone else's?) journal
> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 2014-10-09 16:21:43.301957 7f2be6a027c0 -1 
> filestore(/var/lib/ceph/tmp/mnt.4lJlzP) could not find 
> 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
> 2014-10-09 16:21:43.305941 7f2be6a027c0 -1 created object store 
> /var/lib/ceph/tmp/mnt.4lJlzP journal /var/lib/ceph/tmp/mnt.4lJlzP/journal for 
> osd.47 fsid 70ac4a78-46c0-45e6-8ff9-878b37f50fa1
> 2014-10-09 16:21:43.305992 7f2be6a027c0 -1 auth: error reading file: 
> /var/lib/ceph/tmp/mnt.4lJlzP/keyring: can't open 
> /var/lib/ceph/tmp/mnt.4lJlzP/keyring: (2) No such file or directory
> 2014-10-09 16:21:43.306099 7f2be6a027c0 -1 created new key in keyring 
> /var/lib/ceph/tmp/mnt.4lJlzP/keyring
> added key for osd.47
> === osd

Re: [ceph-users] ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000

[Loic Dachary]

Hi Frederic,

To be 100% sure it would be great if you could manually patch your local 
ceph-disk script and change 'partprobe', into 'partx', '-a', in 
https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284

ceph-disk zap
ceph-disk prepare

and hopefully it will show up as it should. It works for me on centos7 but ...

Cheers

On 10/10/2014 14:33, Loic Dachary wrote:
> Hi Frederic,
> 
> It looks like this is just because 
> https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284 should call 
> partx instead of partprobe. The udev debug output makes this quite clear 
> http://tracker.ceph.com/issues/9721
> 
> I think 
> https://github.com/dachary/ceph/commit/8d914001420e5bfc1e12df2d4882bfe2e1719a5c#diff-788c3cea6213c27f5fdb22f8337096d5R1285
>  fixes it
> 
> Cheers
> 
> On 09/10/2014 16:29, SCHAER Frederic wrote:
>>
>>
>> -Message d'origine-
>> De : Loic Dachary [mailto:l...@dachary.org] 
>> Envoyé : jeudi 9 octobre 2014 16:20
>> À : SCHAER Frederic; ceph-users@lists.ceph.com
>> Objet : Re: [ceph-users] ceph-dis prepare : 
>> UUID=----
>>
>>
>>
>> On 09/10/2014 16:04, SCHAER Frederic wrote:
>>> Hi Loic,
>>>
>>> Back on sdb, as the sde output was from another machine on which I ran 
>>> partx -u afterwards.
>>> To reply your last question first : I think the SG_IO error comes from the 
>>> fact that disks are exported as a single disks RAID0 on a PERC 6/E, which 
>>> does not support JBOD - this is decommissioned hardware on which I'd like 
>>> to test and validate we can use ceph for our use case...
>>>
>>> So back on the  UUID.
>>> It's funny : I retried and ceph-disk prepare worked this time. I tried on 
>>> another disk, and it failed.
>>> There is a difference in the output from ceph-disk : on the failing disk, I 
>>> have these extra lines after disks are prepared :
>>>
>>> (...)
>>> realtime =none   extsz=4096   blocks=0, rtextents=0
>>> Warning: The kernel is still using the old partition table.
>>> The new table will be used at the next reboot.
>>> The operation has completed successfully.
>>> partx: /dev/sdc: error adding partitions 1-2
>>>
>>> I didn't have the warning about the old partition tables on the disk that 
>>> worked. 
>>> So on this new disk, I have :
>>>
>>> [root@ceph1 ~]# mount /dev/sdc1 /mnt
>>> [root@ceph1 ~]# ll /mnt/
>>> total 16
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 ceph_fsid
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 fsid
>>> lrwxrwxrwx 1 root root 58 Oct  9 15:58 journal -> 
>>> /dev/disk/by-partuuid/5e50bb8b-0b99-455f-af71-10815a32bfbc
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 journal_uuid
>>> -rw-r--r-- 1 root root 21 Oct  9 15:58 magic
>>>
>>> [root@ceph1 ~]# cat /mnt/journal_uuid
>>> 5e50bb8b-0b99-455f-af71-10815a32bfbc
>>>
>>> [root@ceph1 ~]# sgdisk --info=1 /dev/sdc
>>> Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
>>> Partition unique GUID: 244973DE-7472-421C-BB25-4B09D3F8D441
>>> First sector: 10487808 (at 5.0 GiB)
>>> Last sector: 1952448478 (at 931.0 GiB)
>>> Partition size: 1941960671 sectors (926.0 GiB)
>>> Attribute flags: 
>>> Partition name: 'ceph data'
>>>
>>> [root@ceph1 ~]# sgdisk --info=2 /dev/sdc
>>> Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
>>> Partition unique GUID: 5E50BB8B-0B99-455F-AF71-10815A32BFBC
>>> First sector: 2048 (at 1024.0 KiB)
>>> Last sector: 10485760 (at 5.0 GiB)
>>> Partition size: 10483713 sectors (5.0 GiB)
>>> Attribute flags: 
>>> Partition name: 'ceph journal'
>>>
>>> Puzzling, isn't it ?
>>>
>>>
>>
>> Yes :-) Just to be 100% sure, when you try to activate this /dev/sdc it 
>> shows an error and complains that the journal uuid is -000* etc ? If so 
>> could you copy your udev debug output ?
>>
>> Cheers
>>
>> [>- FS : -<]  
>>
>> No, when I manually activate the disk instead of attempting to go the udev 
>> way, it seems to work :
>> [root@ceph1 ~]# ceph-disk activate /dev/sdc1
>> got monmap epoch 1
>> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> 2014-10-09 16:21:43.286288 7f2be6a027c0 -1 journal check: ondisk fsid 
>> ---- doesn't match expected 
>> 244973de-7472-421c-bb25-4b09d3f8d441, invalid (someone else's?) journal
>> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 
>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> 2014-10-09 16:21:43.301957 7f2be6a027c0 -1 
>> filestore(/var/lib/ceph/tmp/mnt.4lJlzP) could not find 
>> 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
>> 2014-10-09 16:21:43.305941 7f2be6a027c

Re: [ceph-users] ceph-dis prepare : UUID=00000000-0000-0000-0000-000000000000

[SCHAER Frederic]

Hi Loic,

Patched, and still not working (sorry)...
I'm attaching the prepare output, and also a different a "real " udev debug 
output I captured using " udevadm monitor --environment " (udev.log file)

I added a "sync" command in ceph-disk-udev (this did not change a thing), and I 
noticed that udev script is called 3 times when adding one disk, and that the 
debug output was captured and then mixed all into one file.
This may lead to log mis-interpretation (race conditions ?)...
I changed a bit the logging in order to get one file per call and attached 
those logs to this mail.

File timestamps are as follows :
  File: '/var/log/udev_ceph.log.out.22706'
Change: 2014-10-10 15:48:09.136386306 +0200
  File: '/var/log/udev_ceph.log.out.22749'
Change: 2014-10-10 15:48:11.502425395 +0200
  File: '/var/log/udev_ceph.log.out.22750'
Change: 2014-10-10 15:48:11.606427113 +0200

Actually, I can reproduce the UUID=0 thing with this command :

[root@ceph1 ~]# /usr/sbin/ceph-disk -v activate-journal /dev/sdc2
INFO:ceph-disk:Running command: /usr/bin/ceph-osd -i 0 --get-journal-uuid 
--osd-journal /dev/sdc2
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0b 00 00 00 00 20 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
DEBUG:ceph-disk:Journal /dev/sdc2 has OSD UUID 
----
INFO:ceph-disk:Running command: /sbin/blkid -p -s TYPE -ovalue -- 
/dev/disk/by-partuuid/----
error: /dev/disk/by-partuuid/----: No such file 
or directory
ceph-disk: Cannot discover filesystem type: device 
/dev/disk/by-partuuid/----: Command 
'/sbin/blkid' returned non-zero exit status 2

Ah - to answer previous mails :
- I tried to manually create the gpt partition table to see if things would 
improve, but this was not the case (I also tried to zero out the start and end 
of disks, and also to add random data)
- running ceph-disk prepare twice does not work, it's just that once every 20 
(?) times it "surprisingly does not fail" on this hardware/os combination ;)

Regards

-Message d'origine-
De : Loic Dachary [mailto:l...@dachary.org] 
Envoyé : vendredi 10 octobre 2014 14:37
À : SCHAER Frederic; ceph-users@lists.ceph.com
Objet : Re: [ceph-users] ceph-dis prepare : 
UUID=----

Hi Frederic,

To be 100% sure it would be great if you could manually patch your local 
ceph-disk script and change 'partprobe', into 'partx', '-a', in 
https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284

ceph-disk zap
ceph-disk prepare

and hopefully it will show up as it should. It works for me on centos7 but ...

Cheers

On 10/10/2014 14:33, Loic Dachary wrote:
> Hi Frederic,
> 
> It looks like this is just because 
> https://github.com/ceph/ceph/blob/v0.80.6/src/ceph-disk#L1284 should call 
> partx instead of partprobe. The udev debug output makes this quite clear 
> http://tracker.ceph.com/issues/9721
> 
> I think 
> https://github.com/dachary/ceph/commit/8d914001420e5bfc1e12df2d4882bfe2e1719a5c#diff-788c3cea6213c27f5fdb22f8337096d5R1285
>  fixes it
> 
> Cheers
> 
> On 09/10/2014 16:29, SCHAER Frederic wrote:
>>
>>
>> -Message d'origine-
>> De : Loic Dachary [mailto:l...@dachary.org] 
>> Envoyé : jeudi 9 octobre 2014 16:20
>> À : SCHAER Frederic; ceph-users@lists.ceph.com
>> Objet : Re: [ceph-users] ceph-dis prepare : 
>> UUID=----
>>
>>
>>
>> On 09/10/2014 16:04, SCHAER Frederic wrote:
>>> Hi Loic,
>>>
>>> Back on sdb, as the sde output was from another machine on which I ran 
>>> partx -u afterwards.
>>> To reply your last question first : I think the SG_IO error comes from the 
>>> fact that disks are exported as a single disks RAID0 on a PERC 6/E, which 
>>> does not support JBOD - this is decommissioned hardware on which I'd like 
>>> to test and validate we can use ceph for our use case...
>>>
>>> So back on the  UUID.
>>> It's funny : I retried and ceph-disk prepare worked this time. I tried on 
>>> another disk, and it failed.
>>> There is a difference in the output from ceph-disk : on the failing disk, I 
>>> have these extra lines after disks are prepared :
>>>
>>> (...)
>>> realtime =none   extsz=4096   blocks=0, rtextents=0
>>> Warning: The kernel is still using the old partition table.
>>> The new table will be used at the next reboot.
>>> The operation has completed successfully.
>>> partx: /dev/sdc: error adding partitions 1-2
>>>
>>> I didn't have the warning about the old partition tables on the disk that 
>>> worked. 
>>> So on this new disk, I have :
>>>
>>> [root@ceph1 ~]# mount /dev/sdc1 /mnt
>>> [root@ceph1 ~]# ll /mnt/
>>> total 16
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 ceph_fsid
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 fsid
>>> lrwxrwxrwx 1 root root 58 Oct  9 15:58 journal -> 
>>> /dev/disk/by-partuuid/5e50bb8b-0b99-455f-af71-10815a32bfbc
>>> -rw-r--r-- 1 root root 37 Oct  9 15:58 journal

[ceph-users] ceph at "Universite de Lorraine"

[Stéphane DUGRAVOT]

Hi all, 

We (French University) plan to implement a storage platform (distributed of 
course ) for a volume of 750 TB. We are interesting in CEPH ... 

We wonder the availability of professional support in our project approach. Do 
you know a professional integrator that could assist us for : 


* early design , 
* deployment 
* and probably support ? 
I already know 2 of them. But ... 

Someone told me : " there is no need for professional support , just buy the 
equipment , install it and let it run ceph " 

-> This is probably true, but distributed storage technologies are so new to us 
that we should consider professional support in our project approach 
(especially at this point ) . 

Thanks. 

Stephane. 

-- 
Université de Lorraine 
Stéphane DUGRAVOT - Direction du numérique - Infrastructure 
Jabber : stephane.dugra...@univ-lorraine.fr 
Tél.: +33 3 83 68 20 98 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph at "Universite de Lorraine"

[Alexandre DERUMIER]

Hi Stéphane,

Inktank also provide support through ceph enterprise, and also early design.

http://www.inktank.com/enterprise/


>>Someone told me : " there is no need for professional support , just buy the 
>>equipment , install it and let it run ceph " 

I think it's depend if you have time or human ressources to learn how ceph is 
working.

(It's a little more complex than a classic san storage, mainly for deploy it 
(hardware/network design), tuning it).

But when it's running, it's running very good and without data loss.


Regards,

Alexandre


- Mail original - 

De: "Stéphane DUGRAVOT"  
À: ceph-users@lists.ceph.com 
Envoyé: Vendredi 10 Octobre 2014 16:58:57 
Objet: [ceph-users] ceph at "Universite de Lorraine" 



Hi all, 



We (French University) plan to implement a storage platform (distributed of 
course ) for a volume of 750 TB. We are interesting in CEPH ... 


We wonder the availability of professional support in our project approach. Do 
you know a professional integrator that could assist us for : 


• early design , 
• deployment 
• and probably support ? 

I already know 2 of them. But ... 



Someone told me : " there is no need for professional support , just buy the 
equipment , install it and let it run ceph " 








-> This is probably true, but distributed storage technologies are so new to us 
that we should consider professional support in our project approach 
(especially at this point ) . 

Thanks. 



Stephane. 




-- 
Université de Lorraine 
Stéphane DUGRAVOT - Direction du numérique - Infrastructure 
Jabber : stephane.dugra...@univ-lorraine.fr 

Tél.: +33 3 83 68 20 98 


___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph at "Universite de Lorraine"

[Loic Dachary]

Hi Stéphane,

It all depends on your use case. Red Hat is able to provide the best support 
there is if your use case is unique and complex. If this is for test purposes 
and teaching distributed software it can probably be self managed by students. 
Could you describe what it is going to be used for ?

Cheers

On 10/10/2014 16:58, Stéphane DUGRAVOT wrote:
> Hi all,
> 
> We (French University) plan to implement a storage platform (distributed of 
> course)for a volume of750 TB. We are interesting in CEPH ...
> 
> We wonder the availability of professional support in our project approach.Do 
> you know a professional integrator that could assist us for :
> 
>   * early design,
>   * deployment
>   * and probably support ?
> 
> I already know 2 of them. But ...
> 
> Someone told me : "/there is no need for professional support, just buy the 
> equipment, install it and let it run ceph/"
> 
> -> This is probably true, but distributed storage technologies are so new to 
> us that we should consider professional support in our project approach 
> (especially at this point).
> 
> Thanks.
> 
> Stephane.
> 
> -- 
> *Université de Lorraine**/
> /*Stéphane DUGRAVOT - Direction du numérique - Infrastructure
> Jabber : /stephane.dugra...@univ-lorraine.fr/
> Tél.: /+33 3 83 68 20 98/
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph at "Universite de Lorraine"

[Lionel Bouton]

Le 10/10/2014 16:58, Stéphane DUGRAVOT a écrit :
> Hi all,
>
> We (French University) plan to implement a storage platform
> (distributed of course)for a volume of750 TB. We are interesting in
> CEPH ...
>
> We wonder the availability of professional support in our project
> approach.Do you know a professional integrator that could assist us for :
>
>   * early design,
>   * deployment
>   * and probably support ?
>
> I already know 2 of them. But ...
>
> Someone told me : "/there is no need for professional support, just
> buy the equipment, install it and let it run ceph/"
>
> -> This is probably true, but distributed storage technologies are so
> new to us that we should consider professional support in our project
> approach (especially at this point).
>

It depends on your actual needs, experience with Ceph and the time you
will be able to spend testing then maintaining the platform.

Ceph could be the only component in your storage platform or you might
need to integrate it with other components (KVM, iSCSI targets, NFS,
...) this can generate a lot of questions (goals for service
availability, needs for support of these other components, ...).

You might have to plan for later increases in storage capacity which
might or might not have measurable impacts on performance during
transitions depending on your initial choices of hardware and software
configurations, is it relevant to your case?
You may have to deploy your platform incrementally for budget reasons
which makes the previous question automatically relevant.

If I were in your place and there aren't any confidential information
preventing you to do so, I'll describe your project in more details here
to get a first feedback from the community on the challenges you might
have deploying your platform. Then you'll have a better grasp on what
you'll have to ask an integrator if you really need one.

Best regards,

Lionel Bouton
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Aquino, Ben O]

Thank You Ilya!

Here's the output of dmesg during command execution:

rbd: loaded rbd (rados block device)
libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon1 192.168.101.43:6789 socket error on read
libceph: mon2 192.168.100.42:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon2 192.168.100.42:6789 socket error on read
libceph: mon0 192.168.101.41:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon0 192.168.101.41:6789 socket error on read
libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon1 192.168.101.43:6789 socket error on read
libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon1 192.168.101.43:6789 socket error on read
libceph: mon2 192.168.100.42:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon2 192.168.100.42:6789 socket error on read
libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon1 192.168.101.43:6789 socket error on read
libceph: mon2 192.168.100.42:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon2 192.168.100.42:6789 socket error on read
libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon1 192.168.101.43:6789 socket error on read
libceph: mon2 192.168.100.42:6789 feature set mismatch, my 4a042a42 < server's 
2404a042a42, missing 240
libceph: mon2 192.168.100.42:6789 socket error on read



-Original Message-
From: Ilya Dryomov [mailto:ilya.dryo...@inktank.com] 
Sent: Friday, October 10, 2014 1:10 AM
To: Aquino, Ben O
Cc: ceph-users@lists.ceph.com; Ferber, Dan; Barnes, Thomas J
Subject: Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add 
failed: (5) Input/output error

On Fri, Oct 10, 2014 at 12:48 AM, Aquino, Ben O  wrote:
> Hello Ceph Users:
>
>
>
> Ceph baremetal client attempting to map device volume via kernel RBD 
> Driver, resulting in unable to map device volume and outputs I/O error.
>
> This is Ceph client only, no MDS,OSD or MON running…see I/O error 
> output below.
>
>
>
>
>
> Client Host Linux Kernel Version :
>
> [root@root ceph]# uname -a
>
> Linux root 3.10.25-11.el6.centos.alt.x86_64 #1 SMP Fri Dec 27 21:44:15 
> UTC
> 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> Ceph Version:
>
> [root@root ceph]# ceph -v
>
> ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
>
>
>
> Check Kernel RBD driver:
>
> [root@root ceph]# locate rbd.ko
>
> /lib/modules/3.10.25-11.el6.centos.alt.x86_64/kernel/drivers/block/rbd
> .ko
>
> /lib/modules/3.10.25-11.el6.centos.alt.x86_64/kernel/drivers/block/drb
> d/drbd.ko
>
>
>
> Check Client to Ceph-Server Connections:
>
> [root@root ceph]# ceph osd lspools
>
> 0 data,1 metadata,2 rbd,3 vsmpool_hp1,4 vsmpool_perf1,5 
> vsmpool_vperf1,6
> openstack_hp1,7 openstack_perf1,8 openstack_vperf1,9 vsmpool_perf2,10
> vsmpool_hp2,11 vsmpool_vperf2,12 testopnstack,13 ec_perf_pool,14
> ec_perf_pool1,15 ec_perf_pool2,16 ec_hiperf_pool1,17 ec_valperf_pool1,
>
>
>
> Created RBD:
>
> [root@root ceph]# rbd create rbd9  --size 104800 --pool vsmpool_hp1 
> --id admin
>
>
>
> Check RBD:
>
> [root@root ceph]# rbd ls vsmpool_hp1
>
> rbd1
>
> rbd2
>
> rbd3
>
> rbd4
>
> rbd5
>
> rbd6
>
> rbd7
>
> rbd8
>
> rbd9
>
>
>
> Display RBD INFO:
>
> [root@root ceph]# rbd info vsmpool_hp1/rbd9
>
> rbd image 'rbd9':
>
> size 102 GB in 26200 objects
>
> order 22 (4096 kB objects)
>
> block_name_prefix: rb.0.227915.238e1f29
>
> format: 1
>
>
>
> Map RBD:
>
> [root@root ceph]# rbd map vsmpool_hp1/rbd9 --id admin
>
> rbd: add failed: (5) Input/output error

Is there anything in dmesg?  I'm pretty sure you'll see something like "feature 
set mismatch" in there if you look.  Please paste that.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Ilya Dryomov]

On Fri, Oct 10, 2014 at 8:33 PM, Ilya Dryomov  wrote:
> On Fri, Oct 10, 2014 at 8:11 PM, Aquino, Ben O  wrote:
>> Thank You Ilya!
>>
>> Here's the output of dmesg during command execution:
>>
>> rbd: loaded rbd (rados block device)
>> libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < 
>> server's 2404a042a42, missing 240
>> libceph: mon1 192.168.101.43:6789 socket error on read
>
> The kernel you are running is missing support for EC and TUNABLES3
> bits.  I believe if you upgrade to the latest firefly release EC bit
> won't be required - it's a bug in 0.80.1.  As for the TUNABLES3 bit,
> you'll either have to upgrade to 3.16.3 or above or disable
> chooseleaf_vary_r tunable in your crushmap.

See

http://ceph.com/docs/master/start/os-recommendations/#ceph-dependencies
http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables

for more details.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Openstack keystone with Radosgw

[lakshmi k s]

Mark, I am going no where with this. I am going to try with latest OpenStack 
build (build internal to my company) that has HA support. I will keep you 
posted.


On Thursday, October 9, 2014 10:46 PM, Mark Kirkwood 
 wrote:
 


Oh, I see. That complicates it a wee bit (looks back at your messages). 
I see you have:

rgw_keystone_url = http://192.0.8.2:5000

So you'll need to amend/create etc a



and put it in there. I suspect you might be better off changing your rgw 
kesytone url to use port 35357 (the public one). However I think that is 
a side issue.

Also just to double check - 192.0.8.2 *is* the server you are showing us 
the sites-available from?

Cheers

Mark

On 10/10/14 12:50, lakshmi k s wrote:
> Yes Mark, but there is no keystone.conf in this modified Openstack code.
> There is only horizon.conf under /etc/apache2/sites-available folder.
> And that has virtual host 80 only. Should I simply add :35357?
>
> root@overcloud-controller0-fjvtpqjip2hl
> :/etc/apache2/sites-available#
> ls
> 000-default.conf  default-ssl.conf  horizon.conf
>
>
>
>
> On Thursday, October 9, 2014 4:45 PM, Mark Kirkwood
>  wrote:
>
>
> Hmm - It looks to me like you added the chunked request into Horizon
> instead of Keystone. You want virtual host *:35357
>
>
> On 10/10/14 12:32, lakshmi k s wrote:
>  > Have done this too, but in vain. I made changes to Horizon.conf as shown
>  > below. I had only I do not see the user being validated in radosgw log
>  > at all.
>  >
>  > root@overcloud-controller0-fjvtpqjip2hl
> :/etc/apache2/sites-available#
> ls
>  > 000-default.conf  default-ssl.conf  horizon.conf
>  >
>  > 
>  > 
>  >  WSGIScriptAlias /
>  >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi
>  >  WSGIDaemonProcess horizon user=horizon group=horizon processes=3
>  > threads=10 home=/opt/stack/venvs/horizon
>  >
> python-path=/opt/stack/venvs/horizon:/opt/stack/venvs/horizon/lib/python2.7/site-packages/
>  > WSGIApplicationGroup %{GLOBAL}
>  >
>  >  SetEnv APACHE_RUN_USER horizon
>  >  SetEnv APACHE_RUN_GROUP horizon
>  >  WSGIProcessGroup horizon
>  >WSGIChunkedRequest On
>  >
>  >  DocumentRoot
>  >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
>  >  Alias /static
>  >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
>  >  Alias /media
>  >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
>  >
>  >  
>  >  Options FollowSymLinks
>  >  AllowOverride None
>  >  
>  >
>  >>
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static>
>  >  Options Indexes FollowSymLinks MultiViews
>  >  Require all granted
>  > AllowOverride None
>  >  Order allow,deny
>  >  allow from all
>  >  
>  >
>  >> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard>
>  >  Options Indexes FollowSymLinks MultiViews
>  >  Require all granted
>  >  AllowOverride None
>  >  Order allow,deny
>  > allow from all
>  >  
>  >
>  >  ErrorLog /var/log/httpd/horizon_error.log
>  >  LogLevel debug
>  >  CustomLog /var/log/httpd/horizon_access.log combined
>  > 
>  >
>  > WSGISocketPrefix /var/run/httpd
>  >
>  > --
>  >
>  >
>  >
>  >
>  > On Thursday, October 9, 2014 3:51 PM, Mark Kirkwood
>  >  > wrote:
>  >
>  >
>  > No, I don't have any explicit ssl enabled in the rgw site.
>  >
>  > Now you might be running into http://tracker.ceph.com/issues/7796
>  > . So
>  > check if you have enabled
>  >
>  > WSGIChunkedRequest On
>  >
>  > In your keystone virtualhost setup (explained in the issue).
>  >
>  > Cheers
>  >
>  > Mark
>  >
>  >
>  > On 10/10/14 11:03, lakshmi k s wrote:
>  >  > Right, I have these certs on both nodes - keystone node and rgw
> gateway
>  >  > node. Not sure where I am going wrong. And what about SSL? Should the
>  >  > following be in rgw.conf in gateway node? I am not using this as
> it was
>  >  > optional.
>  >  >
>  >  > SSLEngine on
>  >  > SSLCertificateFile /etc/apache2/ssl/apache.crt
>  >  > SSLCertificateKeyFile /etc/apache2/ssl/apache.key
>  >  > SetEnv SERVER_PORT_SECURE 443
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  > On Thursday, October 9, 2014 2:48 PM, Mark Kirkwood
>  >  > mailto:mark.kirkw...@catalyst.net.nz>
>  >  >> wrote:
>  >  >
>  >  >
>  >  > Almost - the converted certs need to be saved on your *rgw* host in
>  >  > nss_db_path (default is /var/ceph/nss but wherever you have it
>  >  > configured should be ok). Then restart the

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Ilya Dryomov]

On Fri, Oct 10, 2014 at 8:11 PM, Aquino, Ben O  wrote:
> Thank You Ilya!
>
> Here's the output of dmesg during command execution:
>
> rbd: loaded rbd (rados block device)
> libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < 
> server's 2404a042a42, missing 240
> libceph: mon1 192.168.101.43:6789 socket error on read

The kernel you are running is missing support for EC and TUNABLES3
bits.  I believe if you upgrade to the latest firefly release EC bit
won't be required - it's a bug in 0.80.1.  As for the TUNABLES3 bit,
you'll either have to upgrade to 3.16.3 or above or disable
chooseleaf_vary_r tunable in your crushmap.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph at "Universite de Lorraine"

[Lionel Bouton]

Le 10/10/2014 16:58, Stéphane DUGRAVOT a écrit :
> Hi all,
>
> We (French University) plan to implement a storage platform
> (distributed of course)for a volume of750 TB. We are interesting in
> CEPH ...
>
> We wonder the availability of professional support in our project
> approach.Do you know a professional integrator that could assist us for :
>
>   * early design,
>   * deployment
>   * and probably support ?
>
> I already know 2 of them. But ...
>
> Someone told me : "/there is no need for professional support, just
> buy the equipment, install it and let it run ceph/"
>
> -> This is probably true, but distributed storage technologies are so
> new to us that we should consider professional support in our project
> approach (especially at this point).
>

It depends on your actual needs, experience with Ceph and the time you
will be able to spend testing then maintaining the platform.

Ceph could be the only component in your storage platform or you might
need to integrate it with other components (KVM, iSCSI targets, NFS,
...) this can generate a lot of questions (goals for service
availability, needs for support of these other components, ...).

You might have to plan for later increases in storage capacity which
might or might not have measurable impacts on performance during
transitions depending on your initial choices of hardware and software
configurations, is it relevant to your case?
You may have to deploy your platform incrementally for budget reasons
which makes the previous question automatically relevant.

If I were in your place and there aren't any confidential information
preventing you to do so, I'll describe your project in more details here
to get a first feedback from the community on the challenges you might
have deploying your platform. Then you'll have a better grasp on what
you'll have to ask an integrator if you really need one.

Best regards,

Lionel Bouton
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Basic Ceph questions

[Craig Lewis]

>
>
> Just curious, what kind of applications use RBD? It cant be
> applications which need high speed SAN storage performance
> characteristics?
>

Most people seem to be using it as storage for OpenStack.

I've heard about people using RDB + Heartbeats to make an HA NFS, while
they wait for CephFS to be production ready.

People that are re-exporting images via iSCSI and Fiber Channel are
probably doing something different.  If I had to hazard a guess, I'd guess
that they're doing some sort of HA Clustered service, like a database.
That's the traditional use for shared storage.




>
> For VMs, I am trying to visualize how the RBD device would be exposed.
> Where does the driver live exactly? If its exposed via libvirt and
> QEMU, does the kernel driver run in the host OS, and communicate with
> a backend Ceph cluster? If yes, does libRBD provide a target (SCSI?)
> interface which the kernel driver connects to? Trying to visualize
> what the stack looks like, and the flow of IOs for block devices.
>

I'll have to leave that for others to answer.


>>
> >> b. If it is strongly consistent, is that the case across sites also?
> >> How can it be performant across geo sites if that is the case? If its
> >> choosing consistency over partitioning and availability...For object,
> >> I read somewhere that it is now eventually consistent(local CP,
> >> remotely AP) via DR. Gets a bit confusing with all the literature out
> >> there. If it is DR, isnt that slightly different from the Swift case?
> >
> >
> > If you're referring to RadosGW Federation, no.  That replication is
> async.
> > The replication has several delays built in, so the fastest you could to
> see
> > your data show up in the secondary is about a minute.  Longer if the file
> > takes a while to transfer, or you have a lot of activity to replicate.
> >
> > Each site is still CP.  There is just delay getting data from the
> primary to
> > the secondary.
> In that case, it is like Swift, only differently done. The async makes
> it eventually consistent across sites, no?


I'm not sure regarding Swift.  Also outside my experience.

But yes, the async replication is eventually consistent, with no
guarantee.  Problems during replication can cause the clusters to get out
of sync.  The replication agent will retry failures, but it doesn't store
the information anywhere.  If you restart the replication agent when it had
known failures, those failures won't be retried.  Every one of the errors
is logged, so I was able to manually download & re-upload the file to the
primary cluster, which triggered re-replication.

So far, all of the inconsistencies have shown up by comparing bucket
listing.  I'm in the process of manually verifying checksums (my
application stores a SHA256 for every object uploaded).  So far, I haven't
had any failures in files that were marked as successfully replicated.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Aquino, Ben O]

Thanks Ilya,
Will try to build client hosts with Centos7 running kernel 3.16.

_ben

-Original Message-
From: Ilya Dryomov [mailto:ilya.dryo...@inktank.com] 
Sent: Friday, October 10, 2014 9:34 AM
To: Aquino, Ben O
Cc: ceph-users@lists.ceph.com; Ferber, Dan; Barnes, Thomas J
Subject: Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add 
failed: (5) Input/output error

On Fri, Oct 10, 2014 at 8:11 PM, Aquino, Ben O  wrote:
> Thank You Ilya!
>
> Here's the output of dmesg during command execution:
>
> rbd: loaded rbd (rados block device)
> libceph: mon1 192.168.101.43:6789 feature set mismatch, my 4a042a42 < 
> server's 2404a042a42, missing 240
> libceph: mon1 192.168.101.43:6789 socket error on read

The kernel you are running is missing support for EC and TUNABLES3 bits.  I 
believe if you upgrade to the latest firefly release EC bit won't be required - 
it's a bug in 0.80.1.  As for the TUNABLES3 bit, you'll either have to upgrade 
to 3.16.3 or above or disable chooseleaf_vary_r tunable in your crushmap.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Regarding Primary affinity configuration

[Johnu George (johnugeo)]

Thanks for detailed post, Greg. I was trying to configure primary affinity
in my cluster but I didn’t see any expected results. As you said, I was
just looking into single pg and got wrong. I also had primary affinity
value configured for multiple osds in a pg, which makes the calculation
more complex. As in your eg: if osd0, osd1,osd2 has primary affinity value
of [1,0.5,0.1] and there are 600 pgs, the final distribution comes in
440:140:20 or 22:7:1 which is slighly skewed from expected.

Johnu

On 10/9/14, 4:51 PM, "Gregory Farnum"  wrote:

>On Thu, Oct 9, 2014 at 4:24 PM, Johnu George (johnugeo)
> wrote:
>> Hi Greg,
>>  Thanks for your extremely informative post. My related
>>questions
>> are posted inline
>>
>> On 10/9/14, 2:21 PM, "Gregory Farnum"  wrote:
>>
>>>On Thu, Oct 9, 2014 at 10:55 AM, Johnu George (johnugeo)
>>> wrote:
 Hi All,
   I have few questions regarding the Primary affinity.  In the
 original blueprint

(https://wiki.ceph.com/Planning/Blueprints/Firefly/osdmap%3A_primary_ro
le
_affinity
 ), one example has been given.

 For PG x, CRUSH returns [a, b, c]
 If a has primary_affinity of .5, b and c have 1 , with 50%
probability,
we
 will choose b or c instead of a. (25% for b, 25% for c)

 A) I was browsing through the code, but I could not find this logic of
 splitting the rest of configured primary affinity value between other
osds.
 How is this handled?

 if (a < CEPH_OSD_MAX_PRIMARY_AFFINITY &&
 (crush_hash32_2(CRUSH_HASH_RJENKINS1,
 seed, o) >> 16) >= a) {
   // we chose not to use this primary.  note it anyway as a
   // fallback in case we don't pick anyone else, but keep looking.
   if (pos < 0)
 pos = i;
 } else {
   pos = i;
   break;
 }
   }
>>>
>>>It's a fallback mechanism ‹ if the chosen primary for a PG has primary
>>>affinity less than the default (max), we (probabilistically) look for
>>>a different OSD to be the primary. We decide whether to offload by
>>>running a hash and discarding the OSD if the output value is greater
>>>than the OSDs affinity, and then we go through the list and run that
>>>calculation in order (obviously if the affinity is 1, then it passes
>>>without needing to run the hash).
>>>If no OSD in the list has a high enough hash value, we take the
>>>originally-chosen primary.
>>  As in example for [0.5,1,1], I got your point that with 50%
>>probability,
>> first osd will be chosen. But, how do we ensure that second and third
>>osd
>> will be having remaining 25% and 25% respectively?. I could see only
>> individual primary affinity values but not a sum value anywhere to
>>ensure
>> that.
>
>Well, for any given PG with that pattern, the second OSD in the list
>is going to be chosen. But *which* osd is listed second is random, so
>if you only have 3 OSDs 0,1,2 (with weights .5, 1, 1, respectively),
>then the PGs in total will work in a 1:2:2 ratio because OSDs 1 and 2
>will between themselves be first in half of the PG lists.
>
>>
>>>
 B) Since, primary affinity value is configured independently, there
can
be a
 situation with [0.1,0.1,0.1]  with total value that don¹t add to 1.
How is
 this taken care of?
>>>
>>>These primary affinity values are just compared against the hash
>>>output I mentioned, so the sum doesn't matter. In general we simply
>>>expect that OSDs which don't have the max weight value will be chosen
>>>as primary in proportion to their share of the total weight of their
>>>PG membership (ie, if they have a weight of .5 and everybody else has
>>>weight 1, they will be primary in half the normal number of PGs. If
>>>everybody has a weight of .5, they will be primary in the normal
>>>proportions. Etc).
>>
>> I got your idea but I couldn¹t figure out that from the code. You said
>> that max weight value will be chosen as primary in proportion to their
>> share of the total weight of their
>> PG membership. But, from what I understood from code, if it is
>> [0.1,0.1,0.1], first osd will be chosen always. (Probabilistically for
>>10%
>> reads, it will choose first osd. However,first osd will still be chosen
>> for rest of the reads as part of fallback mechanism which is the
>> originally chosen primary.) Am I wrong?
>
>If each OSD has affinity of 0.1, then the hash is run until its output
>is <0.1 for one of the OSDs in the list. If *none* of the OSDs in the
>list hashes out a number smaller than that, then the first one in the
>list (which would be the primary by default!) will be selected.
>
>>
>>>

 C) Slightly confused. What happens for a situation with [1,0.5,1] ? Is
osd.0
 always returned?
>>>
>>>If the first OSD in the PG list has primary affinity of 1 then it is
>>>always the primary for that OSD, yes. That's not osd.0, though; just
>>>the first OSD in the PG list. ;)
>>
>> Sorry. I meant the first OS

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Ilya Dryomov]

On Fri, Oct 10, 2014 at 9:22 PM, Aquino, Ben O  wrote:
> Thanks Ilya,
> Will try to build client hosts with Centos7 running kernel 3.16.

Make sure it's 3.16.3 or later.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Aquino, Ben O]

Thanks Ilya!
Will try building client hosts Fedora20 running version 
kernel-3.16.3-200.fc20.x86_64.rpm.
Then client hosts will used ceph 80.1 from used 
http://ceph.com/rpm-firefly/fc20/x86_64/.


Regards;
_ben

-Original Message-
From: Ilya Dryomov [mailto:ilya.dryo...@inktank.com] 
Sent: Friday, October 10, 2014 10:29 AM
To: Aquino, Ben O
Cc: ceph-users@lists.ceph.com; Ferber, Dan; Barnes, Thomas J
Subject: Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add 
failed: (5) Input/output error

On Fri, Oct 10, 2014 at 9:22 PM, Aquino, Ben O  wrote:
> Thanks Ilya,
> Will try to build client hosts with Centos7 running kernel 3.16.

Make sure it's 3.16.3 or later.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Basic Ceph questions

[John Spray]

On Fri, Oct 10, 2014 at 1:19 AM, Marcus White
 wrote:
> FUSE is probably for Ceph file system..

For avoidance of doubt: there are *two* fuse modules in ceph:
 * RBD: http://ceph.com/docs/master/man/8/rbd-fuse/
 * CephFS: http://ceph.com/docs/master/man/8/ceph-fuse/

Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph at "Universite de Lorraine"

[Serge van Ginderachter]

On 10 October 2014 16:58, Stéphane DUGRAVOT <
stephane.dugra...@univ-lorraine.fr> wrote:

> We wonder the availability of professional support in our project
> approach.


​We were happy to work with Wido Den Hollander https://www.42on.com/​

​​
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mds isn't working anymore after osd's running full

[Gregory Farnum]

Ugh, "debug journaler", not "debug journaled."

That said, the filer output tells me that you're missing an object out
of the MDS log. (200.08f5) I think this issue should be resolved
if you "dump" the journal to a file, "reset" it, and then "undump" it.
(These are commands you can invoke from ceph-mds.)
I haven't done this myself in a long time, so there may be some hard
edges around it. In particular, I'm not sure if the dumped journal
file will stop when the data stops, or if it will be a little too
long. If so, we can fix that by truncating the dumped file to the
proper length and resetting and undumping again.
(And just to harp on it, this journal manipulation is a lot simpler in
Giant... ;) )
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

On Wed, Oct 8, 2014 at 7:11 AM, Jasper Siero
 wrote:
> Hello Greg,
>
> No problem thanks for looking into the log. I attached the log to this email.
> I'm looking forward for the new release because it would be nice to have more 
> possibilities to diagnose problems.
>
> Kind regards,
>
> Jasper Siero
> 
> Van: Gregory Farnum [g...@inktank.com]
> Verzonden: dinsdag 7 oktober 2014 19:45
> Aan: Jasper Siero
> CC: ceph-users
> Onderwerp: Re: [ceph-users] mds isn't working anymore after osd's running full
>
> Sorry; I guess this fell off my radar.
>
> The issue here is not that it's waiting for an osdmap; it got the
> requested map and went into replay mode almost immediately. In fact
> the log looks good except that it seems to finish replaying the log
> and then simply fail to transition into active. Generate a new one,
> adding in "debug journaled = 20" and "debug filer = 20", and we can
> probably figure out how to fix it.
> (This diagnosis is much easier in the upcoming Giant!)
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
> On Tue, Oct 7, 2014 at 7:55 AM, Jasper Siero
>  wrote:
>> Hello Gregory,
>>
>> We still have the same problems with our test ceph cluster and didn't 
>> receive a reply from you after I send you the requested log files. Do you 
>> know if it's possible to get our cephfs filesystem working again or is it 
>> better to give up the files on cephfs and start over again?
>>
>> We restarted the cluster serveral times but it's still degraded:
>> [root@th1-mon001 ~]# ceph -w
>> cluster c78209f5-55ea-4c70-8968-2231d2b05560
>>  health HEALTH_WARN mds cluster is degraded
>>  monmap e3: 3 mons at 
>> {th1-mon001=10.1.2.21:6789/0,th1-mon002=10.1.2.22:6789/0,th1-mon003=10.1.2.23:6789/0},
>>  election epoch 432, quorum 0,1,2 th1-mon001,th1-mon002,th1-mon003
>>  mdsmap e190: 1/1/1 up {0=th1-mon001=up:replay}, 1 up:standby
>>  osdmap e2248: 12 osds: 12 up, 12 in
>>   pgmap v197548: 492 pgs, 4 pools, 60297 MB data, 470 kobjects
>> 124 GB used, 175 GB / 299 GB avail
>>  491 active+clean
>>1 active+clean+scrubbing+deep
>>
>> One placement group stays in the deep scrubbing fase.
>>
>> Kind regards,
>>
>> Jasper Siero
>>
>>
>> 
>> Van: Jasper Siero
>> Verzonden: donderdag 21 augustus 2014 16:43
>> Aan: Gregory Farnum
>> Onderwerp: RE: [ceph-users] mds isn't working anymore after osd's running 
>> full
>>
>> I did restart it but you are right about the epoch number which has changed 
>> but the situation looks the same.
>> 2014-08-21 16:33:06.032366 7f9b5f3cd700  1 mds.0.27  need osdmap epoch 1994, 
>> have 1993
>> 2014-08-21 16:33:06.032368 7f9b5f3cd700  1 mds.0.27  waiting for osdmap 1994 
>> (which blacklists
>> prior instance)
>> I started the mds with the debug options and attached the log.
>>
>> Thanks,
>>
>> Jasper
>> 
>> Van: Gregory Farnum [g...@inktank.com]
>> Verzonden: woensdag 20 augustus 2014 18:38
>> Aan: Jasper Siero
>> CC: ceph-users@lists.ceph.com
>> Onderwerp: Re: [ceph-users] mds isn't working anymore after osd's running 
>> full
>>
>> After restarting your MDS, it still says it has epoch 1832 and needs
>> epoch 1833? I think you didn't really restart it.
>> If the epoch numbers have changed, can you restart it with "debug mds
>> = 20", "debug objecter = 20", "debug ms = 1" in the ceph.conf and post
>> the resulting log file somewhere?
>> -Greg
>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>
>>
>> On Wed, Aug 20, 2014 at 12:49 AM, Jasper Siero
>>  wrote:
>>> Unfortunately that doesn't help. I restarted both the active and standby 
>>> mds but that doesn't change the state of the mds. Is there a way to force 
>>> the mds to look at the 1832 epoch (or earlier) instead of 1833 (need osdmap 
>>> epoch 1833, have 1832)?
>>>
>>> Thanks,
>>>
>>> Jasper
>>> 
>>> Van: Gregory Farnum [g...@inktank.com]
>>> Verzonden: dinsdag 19 augustus 2014 19:49
>>> Aan: Jasper Siero
>>> CC: ceph-users@lists.ceph.com
>>> Onderwerp: Re: [ceph-users] mds isn't working anymor

[ceph-users] Firefly v0.80.6 issues 9696 and 9732

[Samuel Just]

We've gotten some reports of a couple of issues on v0.80.6:

1) #9696: mixed clusters (or upgrading clusters) with v0.80.6 and
pre-firefly osds/mons can hit an assert in PG::choose_acting during
backfill.  The fix appears to be to remove the assert
(wip-9696[-firefly]).

2) #9731: there is a bug in PGLog::IndexedLog::trim() which can cause
a segfault.  The bug primarily triggers on OSDs which recently
upgraded from any version prior to v0.80.6 (wip-9731[-firefly])

You may want to hold off on v0.80.6, particularly on upgrades, until
we have a point release out with fixes.  The fixes appear to be
simple, we hope to have a .7 release out next week.
-Sam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Micro Ceph summit during the OpenStack summit

[Loic Dachary]

Hi Ceph,

TL;DR: please register at http://pad.ceph.com/p/kilo if you're attending the 
OpenStack summit

November 3 - 7 in Paris will be the OpenStack summit in Paris 
https://www.openstack.org/summit/openstack-paris-summit-2014/, an opportunity 
to meet with Ceph developers and users. We will have a conference room 
dedicated to Ceph (half a day, date to be determined).

Instead of preparing an abstract agenda, it is more interesting to find out who 
will be there and what topics we would like to talk about. 

In the spirit of the OpenStack summit it would make sense to primarily discuss 
the implementation proposals of various features and improvements scheduled for 
the next Ceph release, Hammer. The online Ceph Developer Summit 
http://ceph.com/community/ceph-developer-summit-hammer/ is scheduled the week 
before and we will have plenty of material.

If you're attending the OpenStack summit, please add yourself to 
http://pad.ceph.com/p/kilo and list the topics you'd like to discuss. Next week 
Josh Durgin and myself will spend some time to prepare this micro Ceph summit 
and make it a lively and informative experience :-)

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS priorities (survey!)

[Sage Weil]

Hi everyone,

In order to help us prioritize our efforts around CephFS, we'd very much 
appreciate it if anybody interested complete the below survey:

https://www.surveymonkey.com/s/VWYVSZ8

It's a single rank-order list of things we could be working on.  Any input 
you provide will be most helpful and appreciated!

Thanks-
sage

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add failed: (5) Input/output error

[Aquino, Ben O]

ThankYou Ilya!



I have built Centos6.5 kernel 3.6.3 using source linux-3.16.3.tar.gz  from 
www.kernel.org

[root@root ~]# uname -a

Linux root 3.16.3 #1 SMP Fri Oct 10 06:48:44 PDT 2014 x86_64 x86_64 x86_64 
GNU/Linux

The kernel modules (includes rbd.ko hopefully…are still compiling….



Since building a FedoraCore20 Client hosts is much faster that building a 
kernel and modules from sources.



I have built a FedoraCore20 Client hosts machine and below are the 
results…Enjoy!!:



[root@localhost ceph]# uname -a

Linux localhost.localdomain 3.16.3-200.fc20.x86_64 #1 SMP Wed Sep 17 22:34:21 
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux





Created RBD:

[root@root ceph]# rbd create rbd1  --size 40960 --pool vsmpool_hp1  -id admin

[root@root ceph]# rbd create rbd1  --size 40960 --pool vsmpool_perf1  -id admin



List the RBD’s:

[root@localhost ceph]# rbd ls vsmpool_hp1

rbd1

[root@localhost ceph]# rbd ls vsmpool_perf1

rbd1





[root@localhost ceph]# rbd map vsmpool_hp1/rbd1 --id admin

Successful…

Note: this is where we have an I/O error  in Centos6.5 kernel 3.10.25



[root@localhost ceph]# rbd showmapped

id poolimage snap device

0  vsmpool_hp1 rbd1  -/dev/rbd0



Format the RBD:



[root@localhost ceph]# mkfs.xfs /dev/rbd/vsmpool_hp1/rbd1

log stripe unit (4194304 bytes) is too large (maximum is 256KiB)

log stripe unit adjusted to 32KiB

meta-data=/dev/rbd/vsmpool_hp1/rbd1 isize=256agcount=17, agsize=654336 blks

 =   sectsz=512   attr=2, projid32bit=1

 =   crc=0finobt=0

data =   bsize=4096   blocks=10485760, imaxpct=25

 =   sunit=1024   swidth=1024 blks

naming   =version 2  bsize=4096   ascii-ci=0 ftype=0

log  =internal log   bsize=4096   blocks=5120, version=2

 =   sectsz=512   sunit=8 blks, lazy-count=1

realtime =none   extsz=4096   blocks=0, rtextents=0



Mount the RBD:

#mount /dev/rbd0 /media/test-target1/





Check Linux

[root@localhost ceph]# df -h

Filesystem   Size  Used Avail Use% Mounted on

/dev/mapper/fedora-root   41G  4.4G   35G  12% /

devtmpfs 995M 0  995M   0% /dev

tmpfs   1002M  136K 1001M   1% /dev/shm

tmpfs   1002M  760K 1001M   1% /run

tmpfs   1002M 0 1002M   0% /sys/fs/cgroup

tmpfs   1002M   12K 1002M   1% /tmp

/dev/sda1477M   97M  351M  22% /boot

/dev/mapper/fedora-home   20G   49M   19G   1% /home

/dev/rbd0 40G   33M   40G   1% /media/test-target1

[root@localhost ceph]#





I’ll share results of centos6.5 running kernel 3.16.3 once module finish 
compiling….



Enjoy!!!



Regards

Ben Aquino











-Original Message-
From: Ilya Dryomov [mailto:ilya.dryo...@inktank.com]
Sent: Friday, October 10, 2014 10:29 AM
To: Aquino, Ben O
Cc: ceph-users@lists.ceph.com; Ferber, Dan; Barnes, Thomas J
Subject: Re: [ceph-users] rbd map vsmpool_hp1/rbd9 --id admin -->rbd: add 
failed: (5) Input/output error



On Fri, Oct 10, 2014 at 9:22 PM, Aquino, Ben O 
mailto:ben.o.aqu...@intel.com>> wrote:

> Thanks Ilya,

> Will try to build client hosts with Centos7 running kernel 3.16.



Make sure it's 3.16.3 or later.



Thanks,



Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 回复: 回复: scrub error with keyvalue backend

[廖建锋]

I like keyvalue backend very much because it if a good performance
my request is simple: keep it running, now have another BUG which was fixed in 
0.85 :


014-10-11 08:42:01.165836 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:01.286205 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:01.286209 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:01.286243 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:01.286256 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:01.670037 7f8e6dcf7700 0 log [WRN] : 86 slow requests, 6 
included below; oldest blocked for > 76.874851 secs
2014-10-11 08:42:01.670046 7f8e6dcf7700 0 log [WRN] : slow request 76.867403 
seconds old, received at 2014-10-11 08:40:44.802551: osd_op(mds.0.1:76899860 
10001846283. [create 0~0,setxattr parent (235)] 0.3c8de802 RETRY=1 
ondisk+retry+write e158) v4 currently reached pg
2014-10-11 08:42:01.670049 7f8e6dcf7700 0 log [WRN] : slow request 76.844191 
seconds old, received at 2014-10-11 08:40:44.825763: 
osd_op(client.5190.0:25456631 10001846ea0. [write 0~5174] 0.bf079148 
RETRY=1 snapc 1=[] ondisk+retry+write e158) v4 currently reached pg
2014-10-11 08:42:01.670052 7f8e6dcf7700 0 log [WRN] : slow request 76.867380 
seconds old, received at 2014-10-11 08:40:44.802574: osd_op(mds.0.1:76899903 
100018462ae. [create 0~0,setxattr parent (235)] 0.847d393 RETRY=1 
ondisk+retry+write e158) v4 currently reached pg
2014-10-11 08:42:01.670055 7f8e6dcf7700 0 log [WRN] : slow request 76.844154 
seconds old, received at 2014-10-11 08:40:44.825800: 
osd_op(client.5190.0:25456733 10001846f06. [write 0~5386] 0.9e43e402 
RETRY=1 snapc 1=[] ondisk+retry+write e158) v4 currently reached pg
2014-10-11 08:42:01.670058 7f8e6dcf7700 0 log [WRN] : slow request 76.867329 
seconds old, received at 2014-10-11 08:40:44.802625: osd_op(mds.0.1:76899939 
100018462d2. [create 0~0,setxattr parent (231)] 0.554a3444 RETRY=1 
ondisk+retry+write e158) v4 currently reached pg
2014-10-11 08:42:02.099305 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.099308 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:02.099405 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.099407 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:02.415290 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.415293 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:02.415331 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.415333 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:02.599635 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.599639 7f8e3abb2700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60
2014-10-11 08:42:02.599806 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e64ca2700' had timed out after 60
2014-10-11 08:42:02.599809 7f8e39bb0700 1 heartbeat_map is_healthy 
'KeyValueStore::op_tp thread 0x7f8e644a1700' had timed out after 60


发件人： ceph-users
发送时间： 2014-10-10 16:09
收件人： ceph-users; 
ceph-users
主题： [ceph-users] 回复: scrub error with keyvalue backend
is there anybody can help ?


发件人： ceph-users
发送时间： 2014-10-10 13:34
收件人： ceph-users
主题： [ceph-users] scrub error with keyvalue backend
Dear ceph,

 # ceph -s
cluster e1f18421-5d20-4c3e-83be-a74b77468d61
health HEALTH_ERR 4 pgs inconsistent; 4 scrub errors
monmap e2: 3 mons at 
{storage-1-213=10.1.0.213:6789/0,storage-1-214=10.1.0.214:6789/0,storage-1-215=10.1.0.215:6789/0},
 election epoch 16, quorum 0,1,2 storage-1-213,storage-1-214,storage-1-215
mdsmap e7: 1/1/1 up {0=storage-1-213=up:active}, 2 up:standby
osdmap e135: 18 osds: 18 up, 18 in
pgmap v84135: 1164 pgs, 3 pools, 801 GB data, 15264 kobjects
1853 GB used, 34919 GB / 36772 GB avail
1159 active+clean
4 active+clean+inco

Re: [ceph-users] Openstack keystone with Radosgw

[lakshmi k s]

With latest HA build, I found keystone_modwsgi.conf in 
/etc/apache2/sites-available and added the chunking like below. We have many 
controller nodes, but single virtual IP - 192.0.2.21 for which keystone is 
configured. I have verified keystone setup by executing other services like 
nova list, cinder list, etc. They work fine. It is swift pointing to ceph 
object gateway that is not working. 


Listen 192.0.2.24:35357
Listen 192.0.2.24:5000


WSGIScriptAlias / /etc/keystone/admin
WSGIDaemonProcess keystoneadmin user=keystone group=keystone processes=2 
threads=1 home=/opt/stack/venvs/openstack 
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
WSGIApplicationGroup keystoneadmin

WSGIProcessGroup keystoneadmin


Options FollowSymLinks
Require all granted
WSGIChunkedRequest On


ErrorLog /var/log/keystone/keystone_modwsgi.log
LogLevel info
CustomLog /var/log/keystone/keystone_apache_access.log combined



WSGIScriptAlias / /etc/keystone/main
WSGIDaemonProcess keystonemain user=keystone group=keystone processes=2 
threads=1 home=/opt/stack/venvs/openstack 
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
WSGIApplicationGroup keystonemain

WSGIProcessGroup keystonemain


Options FollowSymLinks
WSGIChunkedRequest On
Require all granted


ErrorLog /var/log/keystone/keystone_modwsgi.log
LogLevel info
CustomLog /var/log/keystone/keystone_apache_access.log combined


root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone 
service-list
+--+--+---+---+
|id|   name   |  type |
description|
+--+--+---+---+
| 642251f08a93444da1aa457c2a0ae9f3 |  cinder  | volume|   Cinder Volume 
Service   |
| c909ea43c9244f7c8296e870986c5fc1 |  glance  | image |Glance Image 
Service   |
| bf80fcba3aec45a6988262b31b7ae12a |   heat   | orchestration |Heat 
Service   |
| 3a1cf21dd3974313ba833e807b3ff997 | keystone |identity   | Keystone 
Identity Service |
| 8abff3ea4bba41f4b9cc9a77a29191fe | neutron  |network|  Neutron 
Service  |
| d87e2f24576a459495f1e08439bae238 |   nova   |compute|Nova Compute 
Service   |
| 77434bc194a3495793b5b4c943248e16 |  swift   |  object-store | 
  |
+--+--+---+---+


root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone 
endpoint-list
+--+---+---+---+-+--+
|id|   region  | publicurl  
   |internalurl|
 adminurl|service_id|
+--+---+---+---+-+--+
| 09159f243eb6457581e01af56e32bf18 | regionOne | 
http://192.0.2.21:8774/v3 | http://192.0.2.21:8774/v3   
  |http://192.0.2.21:8774/v3| 
9b431dae0ff642629ae8f5bfd006e578 |
| 0dda582955934dc0af898ec3db2c5fbc | regionOne |  
http://192.0.2.21:8776/v1/%(tenant_id)s  |  
http://192.0.2.21:8776/v1/%(tenant_id)s  | 
http://192.0.2.21:8776/v1/%(tenant_id)s | 642251f08a93444da1aa457c2a0ae9f3 |
| 2ccd8523954c4491b08b648cfd42ae6c | regionOne | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |  
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |
| 30ca33f2f84242c2a6ad8a91446d265b | regionOne |   
http://192.0.2.21:8773/services/Cloud   |   
http://192.0.2.21:8773/services/Cloud   |  
http://192.0.2.21:8773/services/Admin  | 389b4dec8c9c479dbf46622c22da12d1 |
| 9caad71ea7144f4283509cb60faff864 | regionOne |  
http://192.0.2.21:8774/v2/$(tenant_id)s  |  
http://192.0.2.21:8774/v2/$(tenant_id)s  | 
http://192.0.2.21:8774/v2/$(tenant_id)s | d87e2f24576a459495f1e08439bae238 |
| d3a87ad4fd1c4626a499f0491cfb054a | regionOne |  
http://192.0.2.21:9292/  |  http://192.0.2.21:9292/ 
 | http://192.0.2.21:9292/ | 
c909ea43c9244f7c8296e870986c5fc1 |
| e10b562bb4b646c8a90b6a4255d7efd7 | regionOne | 
http://192.0.2.21:21131/v1| http://19

Re: [ceph-users] Openstack keystone with Radosgw

[Mark Kirkwood]

Hi,

I think your swift endpoint:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | 
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |


is the issue. It should be:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne | 
http://gateway.ex.com/swift/v1 | http://gateway.ex.com/swift/v1 | 
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |


i.e no AUTH_%(tenantid)s in there 
http://ceph.com/docs/master/radosgw/keystone/.


Regards

Mark

On 11/10/14 14:28, lakshmi k s wrote:

With latest HA build, I found keystone_modwsgi.conf in
/etc/apache2/sites-available and added the chunking like below. We have
many controller nodes, but single virtual IP - 192.0.2.21 for which
keystone is configured. I have verified keystone setup by executing
other services like nova list, cinder list, etc. They work fine. It is
swift pointing to ceph object gateway that is not working.

Listen 192.0.2.24:35357
Listen 192.0.2.24:5000


 WSGIScriptAlias / /etc/keystone/admin
 WSGIDaemonProcess keystoneadmin user=keystone group=keystone
processes=2 threads=1 home=/opt/stack/venvs/openstack
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
 WSGIApplicationGroup keystoneadmin

 WSGIProcessGroup keystoneadmin

 
 Options FollowSymLinks
 Require all granted
 WSGIChunkedRequest On
 

 ErrorLog /var/log/keystone/keystone_modwsgi.log
 LogLevel info
 CustomLog /var/log/keystone/keystone_apache_access.log combined



 WSGIScriptAlias / /etc/keystone/main
 WSGIDaemonProcess keystonemain user=keystone group=keystone
processes=2 threads=1 home=/opt/stack/venvs/openstack
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
 WSGIApplicationGroup keystonemain

 WSGIProcessGroup keystonemain

 
 Options FollowSymLinks
 WSGIChunkedRequest On
 Require all granted
 

 ErrorLog /var/log/keystone/keystone_modwsgi.log
 LogLevel info
 CustomLog /var/log/keystone/keystone_apache_access.log combined


root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone
service-list
+--+--+---+---+
|id|   name   |  type |
description|
+--+--+---+---+
| 642251f08a93444da1aa457c2a0ae9f3 |  cinder  | volume|   Cinder
Volume Service   |
| c909ea43c9244f7c8296e870986c5fc1 |  glance  | image |
Glance Image Service   |
| bf80fcba3aec45a6988262b31b7ae12a |   heat   | orchestration |
Heat Service   |
| 3a1cf21dd3974313ba833e807b3ff997 | keystone |identity   | Keystone
Identity Service |
| 8abff3ea4bba41f4b9cc9a77a29191fe | neutron  |network|
Neutron Service  |
| d87e2f24576a459495f1e08439bae238 |   nova   |compute|Nova
Compute Service   |
| 77434bc194a3495793b5b4c943248e16 |  swift   |  object-store
|   |
+--+--+---+---+


root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone
endpoint-list
+--+---+---+---+-+--+
|id|   region  |
publicurl |
internalurl| adminurl
|service_id|
+--+---+---+---+-+--+
| 09159f243eb6457581e01af56e32bf18 | regionOne |
http://192.0.2.21:8774/v3 |
http://192.0.2.21:8774/v3 |
http://192.0.2.21:8774/v3| 9b431dae0ff642629ae8f5bfd006e578 |
| 0dda582955934dc0af898ec3db2c5fbc | regionOne |
http://192.0.2.21:8776/v1/%(tenant_id)s  |
http://192.0.2.21:8776/v1/%(tenant_id)s  |
http://192.0.2.21:8776/v1/%(tenant_id)s | 642251f08a93444da1aa457c2a0ae9f3 |
| 2ccd8523954c4491b08b648cfd42ae6c | regionOne |
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |
| 30ca33f2f84242c2a6ad8a91446d265b | regionOne |
http://192.0.2.21:8773/services/Cloud   |
http://192.0.2.21:8773/services/Cloud   |
http://192.0.2.21:8773/services/Admin  | 389b4dec8c9c479dbf46622c22da12d1 |
| 9caad71ea7144f4283509cb60faff864 | regionOne |
http://192.0.2.21:8774/v2/$(tenant_id)s  |
http://192.0.2.

Re: [ceph-users] Openstack keystone with Radosgw

[lakshmi k s]

Hello Mark - I tried that as well, but in vain. In fact, that is how I created 
the endpoint to begin with. Since, that didn't work, I followed Openstack 
standard which was to include %tenant-id.

-Lakshmi.



On Friday, October 10, 2014 6:49 PM, Mark Kirkwood 
 wrote:
 


Hi,

I think your swift endpoint:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | 
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | 
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |

is the issue. It should be:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne | 
http://gateway.ex.com/swift/v1 | http://gateway.ex.com/swift/v1 | 
http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 |

i.e no AUTH_%(tenantid)s in there 
http://ceph.com/docs/master/radosgw/keystone/.

Regards

Mark

On 11/10/14 14:28, lakshmi k s wrote:
> With latest HA build, I found keystone_modwsgi.conf in
> /etc/apache2/sites-available and added the chunking like below. We have
> many controller nodes, but single virtual IP - 192.0.2.21 for which
> keystone is configured. I have verified keystone setup by executing
> other services like nova list, cinder list, etc. They work fine. It is
> swift pointing to ceph object gateway that is not working.
>
> Listen 192.0.2.24:35357
> Listen 192.0.2.24:5000
>
> 
>  WSGIScriptAlias / /etc/keystone/admin
>  WSGIDaemonProcess keystoneadmin user=keystone group=keystone
> processes=2 threads=1 home=/opt/stack/venvs/openstack
> python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
>  WSGIApplicationGroup keystoneadmin
>
>  WSGIProcessGroup keystoneadmin
>
>  
>  Options FollowSymLinks
>  Require all granted
>  WSGIChunkedRequest On
>  
>
>  ErrorLog /var/log/keystone/keystone_modwsgi.log
>  LogLevel info
>  CustomLog /var/log/keystone/keystone_apache_access.log combined
> 
>
> 
>  WSGIScriptAlias / /etc/keystone/main
>  WSGIDaemonProcess keystonemain user=keystone group=keystone
> processes=2 threads=1 home=/opt/stack/venvs/openstack
> python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
>  WSGIApplicationGroup keystonemain
>
>  WSGIProcessGroup keystonemain
>
>  
>  Options FollowSymLinks
>  WSGIChunkedRequest On
>  Require all granted
>  
>
>  ErrorLog /var/log/keystone/keystone_modwsgi.log
>  LogLevel info
>  CustomLog /var/log/keystone/keystone_apache_access.log combined
> 
>
> root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone
> service-list
> +--+--+---+---+
> |id|   name   |  type |
> description|
> +--+--+---+---+
> | 642251f08a93444da1aa457c2a0ae9f3 |  cinder  | volume|   Cinder
> Volume Service   |
> | c909ea43c9244f7c8296e870986c5fc1 |  glance  | image |
> Glance Image Service   |
> | bf80fcba3aec45a6988262b31b7ae12a |   heat   | orchestration |
> Heat Service   |
> | 3a1cf21dd3974313ba833e807b3ff997 | keystone |identity   | Keystone
> Identity Service |
> | 8abff3ea4bba41f4b9cc9a77a29191fe | neutron  |network|
> Neutron Service  |
> | d87e2f24576a459495f1e08439bae238 |   nova   |compute|Nova
> Compute Service   |
> | 77434bc194a3495793b5b4c943248e16 |  swift   |  object-store
> |   |
> +--+--+---+---+
>
>
> root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone
> endpoint-list
> +--+---+---+---+-+--+
> |id|   region  |
> publicurl |
> internalurl| adminurl
> |service_id|
> +--+---+---+---+-+--+
> | 09159f243eb6457581e01af56e32bf18 | regionOne |
> http://192.0.2.21:8774/v3 |
> http://192.0.2.21:8774/v3 |
> http://192.0.2.21:8774/v3| 9b431dae0ff642629ae8f5bfd006e578 |
> | 0dda582955934dc0af898ec3db2c5fbc | regionOne |
> http://192.0.2.21:8776/v1/%(tenant_id)s  |
> http://192.0.2.21:8776/v1/%(tenant_id)s  |
> http://192.0.2.21:8776/v1/%(tenant_id)s | 642251f08a93444da1aa457c2a0ae9f3 |
> | 2ccd8523954c4491b08b648cfd42ae6c | regionOne |
> http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |
> http://gat

Re: [ceph-users] Openstack keystone with Radosgw

[Mark Kirkwood]

Right, well I suggest changing it back, and adding

debug rgw = 20

in the [client.radosgw...] section of ceph.conf and capture the 
resulting log when you try 'swift stat'. It might reveal the next thing 
to check.


Regards

Mark

On 11/10/14 16:02, lakshmi k s wrote:

Hello Mark - I tried that as well, but in vain. In fact, that is how I
created the endpoint to begin with. Since, that didn't work, I followed
Openstack standard which was to include %tenant-id.

-Lakshmi.


On Friday, October 10, 2014 6:49 PM, Mark Kirkwood
 wrote:


Hi,

I think your swift endpoint:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne |
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |
http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s |
http://gateway.ex.com/swift/v1 |
77434bc194a3495793b5b4c943248e16 |

is the issue. It should be:

| 2ccd8523954c4491b08b648cfd42ae6c | regionOne |
http://gateway.ex.com/swift/v1 |
http://gateway.ex.com/swift/v1 |
http://gateway.ex.com/swift/v1 |
77434bc194a3495793b5b4c943248e16 |

i.e no AUTH_%(tenantid)s in there
http://ceph.com/docs/master/radosgw/keystone/.

Regards

Mark

On 11/10/14 14:28, lakshmi k s wrote:
 > With latest HA build, I found keystone_modwsgi.conf in
 > /etc/apache2/sites-available and added the chunking like below. We have
 > many controller nodes, but single virtual IP - 192.0.2.21 for which
 > keystone is configured. I have verified keystone setup by executing
 > other services like nova list, cinder list, etc. They work fine. It is
 > swift pointing to ceph object gateway that is not working.
 >
 > Listen 192.0.2.24:35357
 > Listen 192.0.2.24:5000
 >
 > 
 >  WSGIScriptAlias / /etc/keystone/admin
 >  WSGIDaemonProcess keystoneadmin user=keystone group=keystone
 > processes=2 threads=1 home=/opt/stack/venvs/openstack
 >
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
 >  WSGIApplicationGroup keystoneadmin
 >
 >  WSGIProcessGroup keystoneadmin
 >
 >  
 >  Options FollowSymLinks
 >  Require all granted
 >  WSGIChunkedRequest On
 >  
 >
 >  ErrorLog /var/log/keystone/keystone_modwsgi.log
 > LogLevel info
 >  CustomLog /var/log/keystone/keystone_apache_access.log combined
 > 
 >
 > 
 >  WSGIScriptAlias / /etc/keystone/main
 >  WSGIDaemonProcess keystonemain user=keystone group=keystone
 > processes=2 threads=1 home=/opt/stack/venvs/openstack
 >
python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/
 >  WSGIApplicationGroup keystonemain
 >
 >  WSGIProcessGroup keystonemain
 >
 >  
 >  Options FollowSymLinks
 >  WSGIChunkedRequest On
 >  Require all granted
 >  
 >
 > ErrorLog /var/log/keystone/keystone_modwsgi.log
 >  LogLevel info
 >  CustomLog /var/log/keystone/keystone_apache_access.log combined
 > 
 >
 > root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5
:~#
keystone
 > service-list
 >
+--+--+---+---+
 > |id|  name  |  type|
 > description|
 >
+--+--+---+---+
 > | 642251f08a93444da1aa457c2a0ae9f3 | cinder  |volume|  Cinder
 > Volume Service  |
 > | c909ea43c9244f7c8296e870986c5fc1 |  glance  |image|
 > Glance Image Service  |
 > | bf80fcba3aec45a6988262b31b7ae12a |  heat  | orchestration |
 > Heat Service  |
 > | 3a1cf21dd3974313ba833e807b3ff997 | keystone |identity  | Keystone
 > Identity Service |
 > | 8abff3ea4bba41f4b9cc9a77a29191fe | neutron  |network|
 > Neutron Service  |
 > | d87e2f24576a459495f1e08439bae238 |  nova  |compute|Nova
 > Compute Service  |
 > | 77434bc194a3495793b5b4c943248e16 |  swift  |  object-store
 > | |
 >
+--+--+---+---+
 >
 >
 > root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5
:~#
keystone
 > endpoint-list
 >
+--+---+---+---+-+--+
 > |id|  region  |
 > publicurl|
 > internalurl| adminurl
 > |service_id|
 >
+--+---+---+---+-+--+
 > | 09159f2