Re: [ceph-users] os recommendations

[Christoph Hellwig]

On Tue, Nov 26, 2013 at 06:50:33AM -0800, Sage Weil wrote:
> If syncfs(2) is not present, we have to use sync(2).  That means you have 
> N daemons calling sync(2) to force a commit on a single fs, but all other 
> mounted fs's are also synced... which means N times the sync(2) calls.
> 
> Fortunately syncfs(2) has been around for a while now, so this only 
> affects really old distros.  And even when glibc does not have a syscall 
> wrapper for it, we try to call the syscall directly.

And for btrfs you were/are using magic ioctls, right.

Looks like the page reference in the last post has already been updated,
thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Container size via s3api

[Mihály Árva-Tóth]

2013/11/26 Derek Yarnell 

> On 11/26/13, 4:04 AM, Mihály Árva-Tóth wrote:
> > Hello,
> >
> > Is there any idea? I don't know this is s3api limitation or missing
> feature?
> >
> > Thank you,
> > Mihaly
>
> Hi Mihaly,
>
> If all you are looking for is the current size of the bucket this can be
> found from the adminops api[1] or when you get do the GET bucket[2]
> operation you will receive a list of keys that you can loop through and
> total their sizes.
>
> [1] - http://ceph.com/docs/master/radosgw/adminops/
> [2] - http://ceph.com/docs/master/radosgw/s3/bucketops/#get-bucket
>

Hi Derek,

Thank you for your answer. Okay so I see I can't do that with s3api. I
would not like to list - loop - sum() because containers holds 5-10
millions of objects and I have fifty containers at the moment. Adminops api
is much interesting thanks I will make some research.

Have a nice day!
Mihaly
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] getting problem in OSD prepare : [ceph_deploy.osd][ERROR ] OSError: [Errno 18] Invalid cross-device link

[upendrayadav.u]

Thanks a lot... after update with ceph-deploy 1.3.3, everything is working fine...Regards,Upendra YadavDFS On Wed, 27 Nov 2013 02:22:00 +0530 Alfredo Deza wrote  ceph-deploy 1.3.3 just got released and you should not see this with the new version.On Tue, Nov 26, 2013 at 9:56 AM, Alfredo Deza  wrote: On Tue, Nov 26, 2013 at 9:19 AM, upendrayadav.u  wrote: Dear Team After executing : ceph-deploy -v osd prepare ceph-node2:/home/ceph/osd1i'm getting some error :[ceph-node2][DEBUG ] connected to host: ceph-node2  [ceph-node2][DEBUG ] detect platform information from remote host[ceph-node2][DEBUG ] detect machine type[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.4 Final[ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node2 [ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf[ceph-node2][WARNIN] osd keyring does not exist yet, creating one[ceph-node2][DEBUG ] create a keyring file [ceph_deploy.osd][ERROR ] OSError: [Errno 18] Invalid cross-device link[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs  You are hitting a bug in ceph-deploy where it fails to copy files across different file systems. This is fixed and shouldbe released soon: http://tracker.ceph.com/issues/6701   and same error for ceph-deploy -v osd prepare ceph-node3:/home/ceph/osd2===1st osd successfully prepared : ceph-deploy -v osd prepare ceph-node1:/home/ceph/osd0 [ceph-node1][DEBUG ] connected to host: ceph-node1 [ceph-node1][DEBUG ] detect platform information from remote host[ceph-node1][DEBUG ] detect machine type[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node1[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf[ceph-node1][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host ceph-node1 disk /home/ceph/osd0 journal None activate False[ceph-node1][INFO  ] Running command: sudo ceph-disk-prepare --fs-type xfs --cluster ceph -- /home/ceph/osd0 [ceph_deploy.osd][DEBUG ] Host ceph-node1 is now ready for osd use.*I have 1 mon and 3 osd. where monitor and 1st osd sharing same machine...  mon and osd0 -       ceph-node1osd1 -                     ceph-node2osd2 -                     ceph-node3ceph-deploy - admin-node Please help me to solve this problem thanks for your precious time and kind attention...Regards,Upendra Yadav DFS___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com   ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw setting puplic ACLs fails.

[Micha Krause]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

No solution so far, but I also asked in IRC and linuxkidd told me they
where looking for a workaround.


Micha Krause
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKVrncACgkQfAR45tA28LhUqQCeMcR430bhaYFncB2/NFTcJIM1
zmcAoICqWwjkMfNjP2yolxBeKI0IvDgJ
=rNFL
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of threads for osd processes

[Jens-Christian Fischer]

> The largest group of threads is those from the network messenger — in
> the current implementation it creates two threads per process the
> daemon is communicating with. That's two threads for each OSD it
> shares PGs with, and two threads for each client which is accessing
> any data on that OSD.

If I read your statement right, then 1000 threads still seem excessive, no? 
(with 24 OSD, there's only max 2 * 23 threads to the other OSDs + some threads 
to the clients)...

/jc
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] can not get rbd cache perf counter

[Shu, Xinxin]

Recently,  I want to test performance benefit of rbd cache, i cannot get 
obvious performance benefit at my setup, then I  try to make sure rbd cache is 
enabled, but I cannot get rbd cache perf counter. In order to identify how to 
enable rbd cache perf counter, I setup a simple setup(one client hosted vms, 
one ceph cluster with two OSDs, each osd has a SSD partition for journal.), 
then build ceph-0.67.4.

My ceph.conf shows as bellows:


[global]
debug default = 0
log file = /var/log/ceph/$name.log

max open files = 131072

auth cluster required = none
auth service required = none
auth client required = none
rbd cache = true
[mon.a]
host = {monitor_host_name}
mon addr = {monitor_addr}

[osd.0]
host = {osd.0_hostname}
public addr = {public_addr}
cluster addr = {cluster_addr}
osd mkfs type = xfs
devs = /dev/sdb1
osd journal = /dev/sdd5
[osd.1]
host = {osd.1_hostname}
public addr = {public_addr}
cluster addr = {cluster_addr}
osd mkfs type = xfs
devs = /dev/sdc1
osd journal = /dev/sdd6


after ceph cluster is built, I create a rbd image with rbd create -size 10240 
-new-format test

then use virsh to start a vm, below is my vm xml file


  test
  524288
  524288
  1
  
hvm

  
  


  
  

  
  


  
  destroy
  restart
  destroy
  
/usr/bin/qemu-system-x86_64

  
  
  
  

 
  
  
  
  i


  

  


Then I add a rbd admin socket in ceph.conf on my client, below is the config

[global]
auth cluster required = none
auth service required = none
auth client required = none
rbd cache = true
rbd cache writethrough until flush = true
[client]
admin socket=/var/run/ceph/rbd-$pid.asok
[mon.a]
host = {monitor_host_name}
mon addr = {monitor_host_addr}


then I checked rbd cache perf counter by this socket, but the output did not 
get any rbd cache statistics

ceph --admin-daemon /var/run/ceph/rbd-3526.asok perf dump output

{ "objecter": { "op_active": 0,
  "op_laggy": 0,
  "op_send": 0,
  "op_send_bytes": 0,
  "op_resend": 0,
  "op_ack": 0,
  "op_commit": 0,
  "op": 0,
  "op_r": 0,
  "op_w": 0,
  "op_rmw": 0,
  "op_pg": 0,
  "osdop_stat": 0,
  "osdop_create": 0,
  "osdop_read": 0,
  "osdop_write": 0,
  "osdop_writefull": 0,
  "osdop_append": 0,
  "osdop_zero": 0,
  "osdop_truncate": 0,
  "osdop_delete": 0,
  "osdop_mapext": 0,
  "osdop_sparse_read": 0,
  "osdop_clonerange": 0,
  "osdop_getxattr": 0,
  "osdop_setxattr": 0,
  "osdop_cmpxattr": 0,
  "osdop_rmxattr": 0,
  "osdop_resetxattrs": 0,
  "osdop_tmap_up": 0,
  "osdop_tmap_put": 0,
  "osdop_tmap_get": 0,
  "osdop_call": 0,
  "osdop_watch": 0,
  "osdop_notify": 0,
  "osdop_src_cmpxattr": 0,
  "osdop_pgls": 0,
  "osdop_pgls_filter": 0,
  "osdop_other": 0,
  "linger_active": 0,
  "linger_send": 0,
  "linger_resend": 0,
  "poolop_active": 0,
  "poolop_send": 0,
  "poolop_resend": 0,
  "poolstat_active": 0,
  "poolstat_send": 0,
  "poolstat_resend": 0,
  "statfs_active": 0,
  "statfs_send": 0,
  "statfs_resend": 0,
  "command_active": 0,
  "command_send": 0,
  "command_resend": 0,
  "map_epoch": 0,
  "map_full": 0,
  "map_inc": 0,
  "osd_sessions": 0,
  "osd_session_open": 0,
  "osd_session_close": 0,
  "osd_laggy": 0},
  "throttle-msgr_dispatch_throttler-radosclient": { "val": 0,
  "max": 104857600,
  "get": 11,
  "get_sum": 5655,
  "get_or_fail_fail": 0,
  "get_or_fail_success": 0,
  "take": 0,
  "take_sum": 0,
  "put": 11,
  "put_sum": 5655,
  "wait": { "avgcount": 0,
  "sum": 0.0}},
  "throttle-objecter_bytes": { "val": 0,
  "max": 104857600,
  "get": 0,
  "get_sum": 0,
  "get_or_fail_fail": 0,
  "get_or_fail_success": 0,
  "take": 0,
  "take_sum": 0,
  "put": 0,
  "put_sum": 0,
  "wait": { "avgcount": 0,
  "sum": 0.0}},
  "throttle-objecter_ops": { "val": 0,
  "max": 1024,
  "get": 0,
  "get_sum": 0,
  "get_or_fail_fail": 0,
  "get_or_fail_success": 0,
  "take": 0,
  "take_sum": 0,
  "put": 0,
  "put_sum": 0,
  "wait": { "avgcount": 0,
  "sum": 0.0}}}

Qemu version:  qemu-system-x86_64 --version
QEMU emulator version 1.2.0 (qemu-kvm-1.2.0+noroms-0ubuntu2.12.10.5, Debian), 
Copyright (c) 2003-2008 Fabrice Bellard

Can anybody help me, any hints will be appreciated ?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Openstack Havana, boot from volume fails

[Jens-Christian Fischer]

> Thanks a lot, Jens. Do I have to have cephx authentication enabled? Did you 
> enable it? Which user from the node that contains cinder-api or glance-api 
> are you using to create volumes and images? The documentation at  
> http://ceph.com/docs/master/rbd/rbd-openstack/ mentions creating new users 
> client.volumes and client.images for cinder and glance respectively. Did you 
> do that?


we have cephx authentication enabled: Here's the /etc/ceph/ceph.conf file that 
our cluster has (we have OSDs on our compute nodes - we shouldn't, but this is 
a test cluster only)

root@h1:~# cat /etc/ceph/ceph.conf
[global]
fsid = 6b3bd327-2f97-44f6-a8fc-
mon_initial_members = hxs, h0s, h1s
mon_host = :yyy:0:6::11c,:yyy:0:6::11e,:yyy:0:6::11d
auth_supported = cephx
osd_journal_size = 1024
filestore_xattr_use_omap = true
ms_bind_ipv6 = true
rgw_print_continue = false

[client]
rbd cache = true


[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

[client.radosgw.gateway]
host = hxs
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log


Make sure that /etc/ceph/ceph.conf is readable by other processes - ceph-deploy 
sets it to 0600 or 0400 (which makes nova really really unhappy)

root@h1:~# ls -l /etc/ceph/ceph.conf
-rw-r--r-- 1 root root 592 Nov  8 16:32 /etc/ceph/ceph.conf

We have a volumes and an images user as you can see (with the necessary rights 
on the volumes and images pool, as described in the ceph-openstack 
documentation)


A really good overview over the current state of ceph and OpenStack Havana was 
posted by Sebastien Hen yesterday: 
http://techs.enovance.com/6424/back-from-the-summit-cephopenstack-integration - 
it cleared a bunch of things for me

cheers
jc


>  
> Thanks again!
> Narendra
>  
> From: Jens-Christian Fischer [mailto:jens-christian.fisc...@switch.ch] 
> Sent: Monday, November 25, 2013 8:19 AM
> To: Trivedi, Narendra
> Cc: ceph-users@lists.ceph.com; Rüdiger Rissmann
> Subject: Re: [ceph-users] Openstack Havana, boot from volume fails
>  
> Hi Narendra
>  
> rbd for cinder and glance are according to the ceph documentation here: 
> http://ceph.com/docs/master/rbd/rbd-openstack/
>  
> rbd for VM images configured like so: https://review.openstack.org/#/c/36042/
>  
> config sample (nova.conf):
>  
> --- cut ---
>  
> volume_driver=nova.volume.driver.RBDDriver
> rbd_pool=volumes
> rbd_user=volumes
> rbd_secret_uuid=--
>  
>  
> libvirt_images_type=rbd
> # the RADOS pool in which rbd volumes are stored (string value)
> libvirt_images_rbd_pool=volumes
> # path to the ceph configuration file to use (string value)
> libvirt_images_rbd_ceph_conf=/etc/ceph/ceph.conf
>  
>  
> # dont inject stuff into partions, RBD backed partitions don't work that way
> libvirt_inject_partition = -2
>  
> --- cut ---
>  
> and finally, used the following files from this repository: 
> https://github.com/jdurgin/nova/tree/havana-ephemeral-rbd
>  
> image/glance.py
> virt/images.py
> virt/libvirt/driver.py
> virt/libvirt/imagebackend.py
> virt/libvirt/utils.py
>  
> good luck :)
>  
> cheers
> jc
>  
> -- 
> SWITCH
> Jens-Christian Fischer, Peta Solutions
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> phone +41 44 268 15 15, direct +41 44 268 15 71
> jens-christian.fisc...@switch.ch
> http://www.switch.ch
> 
> http://www.switch.ch/socialmedia
>  
> On 22.11.2013, at 17:41, "Trivedi, Narendra"  
> wrote:
> 
> 
> Hi Jean,
>  
> Could you please tell me which link you followed to install RBD etc. for 
> Havana?
>  
> Thanks!
> Narendra
>  
> From: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jens-Christian Fischer
> Sent: Thursday, November 21, 2013 8:06 AM
> To: ceph-users@lists.ceph.com
> Cc: Rüdiger Rissmann
> Subject: [ceph-users] Openstack Havana, boot from volume fails
>  
> Hi all
>  
> I'm playing with the boot from volume options in Havana and have run into 
> problems:
>  
> (Openstack Havana, Ceph Dumpling (0.67.4), rbd for glance, cinder and 
> experimental ephemeral disk support)
>  
> The following things do work:
> - glance images are in rbd
> - cinder volumes are in rbd
> - creating a VM from an image works
> - creating a VM from a snapshot works
>  
>  
> However, the booting from volume fails:
>  
> Steps to reproduce:
>  
> Boot from image
> Create snapshot from running instance
> Create volume from this snapshot
> Start a new instance with "boot from volume" and the volume just created:
>  
> The boot process hangs after around 3 seconds, and the console.log of the 
> instance shows this:
>  
> [0.00] Linux version 3.11.0-12-generic (buildd@allspice) (gcc version 
> 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7) ) #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 
> 2013 (Ubuntu 3.11.0-12.19-generic 3.11.3)
> [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-3.11.0-12-gen

Re: [ceph-users] how to Testing cinder and glance with CEPH

[Jens-Christian Fischer]

Hi Karan

your cinder.conf looks sensible to me, I have posted mine here:

--- cut ---

[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes

volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
glance_api_version=2

rbd_user=volumes
rbd_secret_uuid=e1915277-e3a5-4547-bc9e-xxx

rpc_backend = cinder.openstack.common.rpc.impl_kombu
rabbit_host = xxx.yyy.cc
rabbit_port = 5672

quota_volumes=20
quota_snapshots=20

debug = False
use_syslog = True
syslog_log_facility = LOG_LOCAL0



[database]
connection = mysql://cinder:x...@xxx.yyy.cc/cinder


[keystone_authtoken]
# keystone public API
auth_protocol = https
auth_host = xxx.yyy.cc
auth_port = 5000
admin_tenant_name = service
admin_user = cinder
admin_password =xxx

--- cut ---

what are the different cinder*.log files telling you?

Is /etc/ceph/ceph.conf readable for other processes? (chmod 644 
/etc/ceph/ceph.conf)
Are the key rings available and readable?

good luck
jc


-- 
SWITCH
Jens-Christian Fischer, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fisc...@switch.ch
http://www.switch.ch

http://www.switch.ch/socialmedia

On 27.11.2013, at 08:51, Karan Singh  wrote:

> Hello Sebastien / Community
> 
> 
> I tried the commands mentioned in below email.
> 
> 
> [root@rdo ~]#
> [root@rdo ~]# cinder create 1
> +-+--+
> |   Property  |Value |
> +-+--+
> | attachments |  []  |
> |  availability_zone  | nova |
> |   bootable  |false |
> |  created_at |  2013-11-27T07:40:54.161478  |
> | display_description | None |
> | display_name| None |
> |  id | ae8cd686-5f1d-4c05-8c42-cb7622122a3e |
> |   metadata  |  {}  |
> | size|  1   |
> | snapshot_id | None |
> | source_volid| None |
> |status   |   creating   |
> | volume_type | None |
> +-+--+
> [root@rdo ~]#
> [root@rdo ~]# cinder list
> +--++--+--+-+--+-+
> |  ID  | Status | Display Name | Size | 
> Volume Type | Bootable | Attached to |
> +--++--+--+-+--+-+
> | ae8cd686-5f1d-4c05-8c42-cb7622122a3e | error  | None |  1   | 
> None|  false   | |
> +--++--+--+-+--+-+
> [root@rdo ~]#
> [root@rdo ~]#
> [root@rdo ~]#
> [root@rdo ~]# rbd -p ceph-volumes ls
> rbd: pool ceph-volumes doesn't contain rbd images
> [root@rdo ~]#
> [root@rdo ~]#
> [root@rdo ~]# rados lspools
> data
> metadata
> rbd
> ceph-images
> ceph-volumes
> [root@rdo ~]# rbd -p rbd ls
> [root@rdo ~]# rbd -p data ls
> foo
> foo1
> [root@rdo ~]#
> 
> 
> 
> 
> I checked in cinder.log and got the below errors.
> 
> 
> 2013-11-27 09:44:14.830 3273 INFO cinder.volume.manager [-] Updating volume 
> status
> 2013-11-27 09:44:14.830 3273 WARNING cinder.volume.manager [-] Unable to 
> update stats, driver is uninitialized
> 2013-11-27 09:44:42.407 12007 INFO cinder.volume.manager [-] Updating volume 
> status
> 2013-11-27 09:44:42.408 12007 WARNING cinder.volume.manager [-] Unable to 
> update stats, driver is uninitialized
> 2013-11-27 09:44:51.799 4943 INFO cinder.volume.manager [-] Updating volume 
> status
> 2013-11-27 09:44:51.799 4943 WARNING cinder.volume.manager [-] Unable to 
> update stats, driver is uninitialized
> 2013-11-27 09:45:14.834 3273 INFO cinder.volume.manager [-] Updating volume 
> status
> 2013-11-27 09:45:14.834 3273 WARNING cinder.volume.manager [-] Unable to 
> update stats, driver is uninitialized
> [root@rdo cinder]#
> 
> 
> 
> 
> Output from my cinder.conf file
> 
> 
> 
> # Options defined in cinder.volume.utils
> #
> 
> # The default block size used when copying/clearing volumes
> # (string value)
> #volume_dd_blocksize=1M
> 
> 
> # Total option count: 382
> volume_driver=cinder.volume.drivers.rbd.RBDDriver
> rbd_pool=ceph-volumes
> glance_api_version=2
> rbd_user=volumes
> rbd_secret_uuid=801a42

[ceph-users] rocksdb Seen today - replacement for leveldb?

[Stefan Priebe - Profihost AG]

Hi,

while googles leveldb was too slow for facebook they created rocksdb
(http://rocksdb.org/) may be interesting for Ceph? It's already
production quality.

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Minimum requirements for ceph monitors?

[Johannes Klarenbeek]

Dear Ceph-users,

I was wondering if I could use a raspberry PI exclusively for ceph mons only. 
Well, it doesn't have to be a raspberry pi of course, but some cheap, no fan, 
Athlon like server with 100Mps Ethernet connection in a 10 node 4Gb NIC per 
server ceph cluster. Just a thought.

What are the requirements for a ceph-mon only node anyway? Can I run them 
without harddisks (PXE boot and a lot of ram for example)? And what happens if 
I would add 2 monitors in my cluster, that totals up to 3?

Does a ceph-mon node need a second NIC team for cluster traffic only as well?

I hope there are some guide lines on this. Thanks in advance!

Regards,
Johannes




__ Informatie van ESET Endpoint Antivirus, versie van database 
viruskenmerken 9100 (20131127) __

Het bericht is gecontroleerd door ESET Endpoint Antivirus.

http://www.eset.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw daemon stalls on download of some files

[Sebastian]

Hi,

we have a setup of 4 Servers running ceph and radosgw. We use it as an internal 
S3 service for our files. The Servers run Debian Squeeze with Ceph 0.67.4. 

The cluster has been running smoothly for quite a while, but we are currently 
experiencing issues with the radosgw. For some files the HTTP Download just 
stalls at around 500kb. 

The Apache error log just says:
[error] [client ] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle 
timeout (30 sec)
[error] [client ] Handler for fastcgi-script returned invalid result code 1

radosgw logging:
7f00bc66a700  1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 
0x7f00934bb700' had timed out after 600
7f00bc66a700  1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 
0x7f00ab4eb700' had timed out after 600

The interesting thing is that the cluster health is fine an only some files are 
not working properly. Most of them just work fine. A restart of radosgw fixes 
the issue. The other ceph logs are also clean.

Any idea why this happens?

Sebastian


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to monitor osd?

[Ugis]

I can recommend zabbix for it, I use it myself.
You just install zabbix agent on OSD node - it will automatically
discover mounted file systems and report usage on those(osd mounts as
well), nice GUI available if needed.

Sure, you need to set up zabbix server before, but it is easy and worth it!
Zabbix is free & open source.
http://www.zabbix.com/download.php

Good luck!

Ugis

2013/11/21 John Kinsella :
> As an OSD is just a partition, you could use any of the monitoring packages 
> out there? (I like opsview…)
>
> We use the check-ceph-status nagios plugin[1] to monitor overall cluster 
> status, but I'm planning on adding/finding more monitoring functionality soon 
> (e.g. ceph df)
>
> John
> 1: https://github.com/dreamhost/ceph-nagios-plugin
>
> On Nov 21, 2013, at 1:59 AM, tianqing lee 
>  wrote:
>
>> hello,
>>is there some methods to monitor osd nodes? for example the free size of 
>> one osd node.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to Testing cinder and glance with CEPH

[Karan Singh]

Thanks Jens / Sebastien 

It worked for me now , Thanks a lot for your suggestions , they were worth. 

Many Thanks 
Karan Singh 


- Original Message -

From: "Jens-Christian Fischer"  
To: "Karan Singh"  
Cc: "Sebastien Han" , ceph-users@lists.ceph.com 
Sent: Wednesday, 27 November, 2013 11:57:53 AM 
Subject: Re: [ceph-users] how to Testing cinder and glance with CEPH 

Hi Karan 

your cinder.conf looks sensible to me, I have posted mine here: 

--- cut --- 

[DEFAULT] 
rootwrap_config = /etc/cinder/rootwrap.conf 
api_paste_confg = /etc/cinder/api-paste.ini 
iscsi_helper = tgtadm 
volume_name_template = volume-%s 
volume_group = cinder-volumes 
verbose = True 
auth_strategy = keystone 
state_path = /var/lib/cinder 
lock_path = /var/lock/cinder 
volumes_dir = /var/lib/cinder/volumes 

volume_driver=cinder.volume.drivers.rbd.RBDDriver 
rbd_pool=volumes 
glance_api_version=2 

rbd_user=volumes 
rbd_secret_uuid=e1915277-e3a5-4547-bc9e-xxx 

rpc_backend = cinder.openstack.common.rpc.impl_kombu 
rabbit_host = xxx.yyy.cc 
rabbit_port = 5672 

quota_volumes=20 
quota_snapshots=20 

debug = False 
use_syslog = True 
syslog_log_facility = LOG_LOCAL0 



[database] 
connection = mysql://cinder:x...@xxx.yyy.cc/cinder 


[keystone_authtoken] 
# keystone public API 
auth_protocol = https 
auth_host = xxx.yyy.cc 
auth_port = 5000 
admin_tenant_name = service 
admin_user = cinder 
admin_password =xxx 

--- cut --- 

what are the different cinder*.log files telling you? 

Is /etc/ceph/ceph.conf readable for other processes? (chmod 644 
/etc/ceph/ceph.conf) 
Are the key rings available and readable? 

good luck 
jc 


-- 
SWITCH 
Jens-Christian Fischer, Peta Solutions 
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland 
phone +41 44 268 15 15, direct +41 44 268 15 71 
jens-christian.fisc...@switch.ch 
http://www.switch.ch 

http://www.switch.ch/socialmedia 

On 27.11.2013, at 08:51, Karan Singh < ksi...@csc.fi > wrote: 



Hello Sebastien / Community 


I tried the commands mentioned in below email. 


[root@rdo ~]# 
[root@rdo ~]# cinder create 1 
+-+--+ 
| Property | Value | 
+-+--+ 
| attachments | [] | 
| availability_zone | nova | 
| bootable | false | 
| created_at | 2013-11-27T07:40:54.161478 | 
| display_description | None | 
| display_name | None | 
| id | ae8cd686-5f1d-4c05-8c42-cb7622122a3e | 
| metadata | {} | 
| size | 1 | 
| snapshot_id | None | 
| source_volid | None | 
| status | creating | 
| volume_type | None | 
+-+--+ 
[root@rdo ~]# 
[root@rdo ~]# cinder list 
+--++--+--+-+--+-+
 
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | 
+--++--+--+-+--+-+
 
| ae8cd686-5f1d-4c05-8c42-cb7622122a3e | error | None | 1 | None | false | | 
+--++--+--+-+--+-+
 
[root@rdo ~]# 
[root@rdo ~]# 
[root@rdo ~]# 
[root@rdo ~]# rbd -p ceph-volumes ls 
rbd: pool ceph-volumes doesn't contain rbd images 
[root@rdo ~]# 
[root@rdo ~]# 
[root@rdo ~]# rados lspools 
data 
metadata 
rbd 
ceph-images 
ceph-volumes 
[root@rdo ~]# rbd -p rbd ls 
[root@rdo ~]# rbd -p data ls 
foo 
foo1 
[root@rdo ~]# 




I checked in cinder.log and got the below errors. 


2013-11-27 09:44:14.830 3273 INFO cinder.volume.manager [-] Updating volume 
status 
2013-11-27 09:44:14.830 3273 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized 
2013-11-27 09:44:42.407 12007 INFO cinder.volume.manager [-] Updating volume 
status 
2013-11-27 09:44:42.408 12007 WARNING cinder.volume.manager [-] Unable to 
update stats, driver is uninitialized 
2013-11-27 09:44:51.799 4943 INFO cinder.volume.manager [-] Updating volume 
status 
2013-11-27 09:44:51.799 4943 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized 
2013-11-27 09:45:14.834 3273 INFO cinder.volume.manager [-] Updating volume 
status 
2013-11-27 09:45:14.834 3273 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized 
[root@rdo cinder]# 




Output from my cinder.conf file 



# Options defined in cinder.volume.utils 
# 

# The default block size used when copying/clearing volumes 
# (string value) 
#volume_dd_blocksize=1M 


# Total option count: 382 
volume_driver=cinder.volume.drivers.rbd.RBDDriver 
rbd_pool=ceph-volumes 
glance_api_version=2 
rbd_user=volumes 
rbd_secret_uuid=801a42ec-aec1-3ea8-d869-823c2de56b83 

rootwrap_config=/etc/cinder/rootwrap.conf 
sql_connection = mysql://cinder:root@localhost/cinder 
api_paste_config = /etc/cinder/api-paste.ini 

iscsi_helper=tgtadm 
volume_name_template = volume-%s 
volume_group = cinder-volumes 
v

Re: [ceph-users] Minimum requirements for ceph monitors?

[Mark Nelson]

On 11/27/2013 05:35 AM, Johannes Klarenbeek wrote:

Dear Ceph-users,

I was wondering if I could use a raspberry PI exclusively for ceph mons
only. Well, it doesn’t have to be a raspberry pi of course, but some
cheap, no fan, Athlon like server with 100Mps Ethernet connection in a
10 node 4Gb NIC per server ceph cluster. Just a thought.

What are the requirements for a ceph-mon only node anyway? Can I run
them without harddisks (PXE boot and a lot of ram for example)? And what
happens if I would add 2 monitors in my cluster, that totals up to 3?

Does a ceph-mon node need a second NIC team for cluster traffic only as
well?

I hope there are some guide lines on this. Thanks in advance!


Hi,

The lowest specced machines we've done relatively extensive mon 
performance testing on were 4-core ARM9 nodes with 4GB of RAM back for 
the cuttlefish release.  After a number of fixes and optimizations we 
could comfortably support around 200 OSDs with 3 ARM mons.  That was 
only with a limited number of clients though so your mileage may vary. 
A pi would be tough as it's a lot slower and has far less memory. 
Having a local SSD or HD is probably a good idea for leveldb.  Multiple 
network links are a nice to have, but not strictly necessary in this 
price class.


If you are willing to spend a little more, there are other options. 
Some of the thumbstick PCs have dualcore ARM9 CPUs, 2GB of RAM, and can 
run Ubuntu for about $60-70.  That might be good enough as mon nodes for 
smallish clusters (especially if they are clocked at 2GHz+).


If you are willing to pay a little more, this is the board that I just 
picked up for cheap Ceph testing:


http://www.amazon.com/ECS-Elitegroup-Motherboard-NM70-I2-1-0/dp/B00G237CYQ/ref=sr_1_1?ie=UTF8&qid=1385561872&sr=8-1&keywords=ecs+1037u

For ~$67 you get a mini-itx motherboard with a soldered on 17W dual core 
1.8GHz ivy-bridge based Celeron (supports SSE4.2 CRC32 instructions!). 
It has 2 standard dimm slots so no compromising on memory, on-board 
gigabit eithernet, 3 3Gb/s + 1 6Gb/s SATA, and a single PCIE slot for an 
additional NIC.  This has the potential to make a very competent low 
cost, lowish power OSD or mon server.  The biggest downside is that it 
doesn't appear to support ECC memory.  Some of the newer Atoms appear 
to, so that might be an option as well.


Good luck!




Regards,

Johannes



__ Informatie van ESET Endpoint Antivirus, versie van database
viruskenmerken 9100 (20131127) __

Het bericht is gecontroleerd door ESET Endpoint Antivirus.

http://www.eset.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] tracker.ceph.com - public email address visibility?

[James Pearce]

I was going to add something to the bug tracker, but it looks to me 
that contributor email addresses all have public (unauthenticated) 
visibility?  Can this be set in user preferences?


Many thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of threads for osd processes

[Gregory Farnum]

On Wed, Nov 27, 2013 at 1:31 AM, Jens-Christian Fischer
 wrote:
>> The largest group of threads is those from the network messenger — in
>> the current implementation it creates two threads per process the
>> daemon is communicating with. That's two threads for each OSD it
>> shares PGs with, and two threads for each client which is accessing
>> any data on that OSD.
>
> If I read your statement right, then 1000 threads still seem excessive, no? 
> (with 24 OSD, there's only max 2 * 23 threads to the other OSDs + some 
> threads to the clients)...

Well, it depends on how many clients you have. ;) I think the default
settings also have ~12 internal working threads (but I don't recall
exactly). The thread count definitely is not related to the number of
PGs it hosts (directly, anyway — more PGs can lead to more OSD peers
and so more messenger threads). Keep in mind that if you have clients
connecting and then disconnecting repeatedly (eg, the rados tool),
each instance counts as a client and the connection has to time out
(15 minutes) before its threads get cleaned up.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of threads for osd processes

[Mark Nelson]

On 11/27/2013 09:25 AM, Gregory Farnum wrote:

On Wed, Nov 27, 2013 at 1:31 AM, Jens-Christian Fischer
 wrote:

The largest group of threads is those from the network messenger — in
the current implementation it creates two threads per process the
daemon is communicating with. That's two threads for each OSD it
shares PGs with, and two threads for each client which is accessing
any data on that OSD.


If I read your statement right, then 1000 threads still seem excessive, no? 
(with 24 OSD, there's only max 2 * 23 threads to the other OSDs + some threads 
to the clients)...


Well, it depends on how many clients you have. ;) I think the default
settings also have ~12 internal working threads (but I don't recall
exactly). The thread count definitely is not related to the number of
PGs it hosts (directly, anyway — more PGs can lead to more OSD peers
and so more messenger threads). Keep in mind that if you have clients
connecting and then disconnecting repeatedly (eg, the rados tool),
each instance counts as a client and the connection has to time out
(15 minutes) before its threads get cleaned up.


So I am woefully ignorant as to why/how we are doing things here, but is 
there any reason we are spawning new threads for each client connection 
rather than using a thread pool like we do in other areas?



-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of threads for osd processes

[Gregory Farnum]

On Wed, Nov 27, 2013 at 7:28 AM, Mark Nelson  wrote:
> On 11/27/2013 09:25 AM, Gregory Farnum wrote:
>>
>> On Wed, Nov 27, 2013 at 1:31 AM, Jens-Christian Fischer
>>  wrote:

 The largest group of threads is those from the network messenger — in
 the current implementation it creates two threads per process the
 daemon is communicating with. That's two threads for each OSD it
 shares PGs with, and two threads for each client which is accessing
 any data on that OSD.
>>>
>>>
>>> If I read your statement right, then 1000 threads still seem excessive,
>>> no? (with 24 OSD, there's only max 2 * 23 threads to the other OSDs + some
>>> threads to the clients)...
>>
>>
>> Well, it depends on how many clients you have. ;) I think the default
>> settings also have ~12 internal working threads (but I don't recall
>> exactly). The thread count definitely is not related to the number of
>> PGs it hosts (directly, anyway — more PGs can lead to more OSD peers
>> and so more messenger threads). Keep in mind that if you have clients
>> connecting and then disconnecting repeatedly (eg, the rados tool),
>> each instance counts as a client and the connection has to time out
>> (15 minutes) before its threads get cleaned up.
>
>
> So I am woefully ignorant as to why/how we are doing things here, but is
> there any reason we are spawning new threads for each client connection
> rather than using a thread pool like we do in other areas?

Because it's harder and scales a bajillion times farther than people
think it does. Rather spend the dev time on new features and things,
but we will have to address it eventually.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] librados: client.bootstrap-osd authentication error (1) Operation not permitted

[upendrayadav.u]

On Activating cluster ceph disks by using command ceph-deploy osd activate ceph-node3:/home/ceph/osd2i am gettingceph-node3][DEBUG ] connected to host: ceph-node3 [ceph-node3][DEBUG ] detect platform information from remote host[ceph-node3][DEBUG ] detect machine type[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.4 Final[ceph_deploy.osd][DEBUG ] activating host ceph-node3 disk /home/ceph/osd2[ceph_deploy.osd][DEBUG ] will use init type: sysvinit[ceph-node3][INFO  ] Running command: sudo ceph-disk-activate --mark-init sysvinit --mount /home/ceph/osd2[ceph-node3][WARNIN] 2013-11-27 20:56:02.002853 7fe016a60700  0 librados: client.bootstrap-osd authentication error (1) Operation not permitted[ceph-node3][WARNIN] Error connecting to cluster: PermissionError1st OSD that sharing monitor is activated successfully...I am getting this error only for 2nd and 3rd OSD.I have 1 mon and 3 osd. where monitor and 1st osd sharing same machine...mon and osd0 -       ceph-node1osd1 -                     ceph-node2osd2 -                     ceph-node3ceph-deploy - admin-nodeRegards,Upendra YadavDFS___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Number of threads for osd processes

[Kasper Dieter]

On Wed, Nov 27, 2013 at 04:34:00PM +0100, Gregory Farnum wrote:
> On Wed, Nov 27, 2013 at 7:28 AM, Mark Nelson  wrote:
> > On 11/27/2013 09:25 AM, Gregory Farnum wrote:
> >>
> >> On Wed, Nov 27, 2013 at 1:31 AM, Jens-Christian Fischer
> >>  wrote:
> 
>  The largest group of threads is those from the network messenger ? in
>  the current implementation it creates two threads per process the
>  daemon is communicating with. That's two threads for each OSD it
>  shares PGs with, and two threads for each client which is accessing
>  any data on that OSD.
> >>>
> >>>
> >>> If I read your statement right, then 1000 threads still seem excessive,
> >>> no? (with 24 OSD, there's only max 2 * 23 threads to the other OSDs + some
> >>> threads to the clients)...
> >>
> >>
> >> Well, it depends on how many clients you have. ;) I think the default
> >> settings also have ~12 internal working threads (but I don't recall
> >> exactly). The thread count definitely is not related to the number of
> >> PGs it hosts (directly, anyway ? more PGs can lead to more OSD peers
> >> and so more messenger threads). Keep in mind that if you have clients
> >> connecting and then disconnecting repeatedly (eg, the rados tool),
> >> each instance counts as a client and the connection has to time out
> >> (15 minutes) before its threads get cleaned up.
> >
> >
> > So I am woefully ignorant as to why/how we are doing things here, but is
> > there any reason we are spawning new threads for each client connection
> > rather than using a thread pool like we do in other areas?
> 
> Because it's harder and scales a bajillion times farther than people
> think it does. 
It may scale 'farther', but not faster.

1000s of threads talking to each other, managing messages, managing queues, 
managing locks ...
... this takes time: 100s of micro seconds, 100s of systems calls for _ONE_ 
single client-write
(Bug #6366 / long TAT - due too long residence time in Ceph code)

Regards,
-Dieter


> Rather spend the dev time on new features and things,
> but we will have to address it eventually.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw setting puplic ACLs fails.

[Derek Yarnell]

On 11/26/13, 3:31 PM, Shain Miley wrote:
> Micha,
> 
> Did you ever figure out a work around for this issue?
> 
> I also had plans of using s3cmd to put, and recursively set acl's on a 
> nightly basis...however we are getting the 403 errors as well during our 
> testing.
> 
> I was just wondering if you were able to find another solution.

Hi,

There is code[1] in the master branch (I am not sure but I hope it will
make it into the next stable release, it is not in 0.72.x) that allows
you defer to the bucket ACLs.  defer_to_bucket_acls is the configurable
which allows for two different modes.  Recurse just propagates the
specific bucket acls to all the keys, it does fall through to the key
ACL if the bucket ACL doesn't apply.  Full_control allows someone with
FULL_CONTROL at the bucket level to do whatever they want to the keys
(including replace the whole ACL), and again falls through to the key ACL.

Note this breaks AWS S3 compatibility and is why it is a configurable.

[1] - https://github.com/ceph/ceph/pull/672

Thanks,
derek

-- 
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSD on an external, shared device

[kevin horan]

I am working with a small test cluster, but the problems described 
here will remain in production. I have an external fiber channel storage 
array and have exported 2 3TB disks (just as JBODs). I can use 
ceph-deploy to create an OSD for each of these disks on a node named 
Vashti. So far everything is fine. The problem is that I have another 
machine, named Zadok (also will be part of the ceph cluster), which is 
on the same fiber channel network and so can see the same two disks. 
This on its own is still not a problem. But the ceph init script now 
seems to scan all devices it can see and if it finds an OSD on any of 
them it just starts it. So now both machines will find both disks and 
mount/start both of them, which will lead to corruption. I have seen 
this happen already.  So how can I prevent this from happening?  Ideally 
I would want one OSD running on each machine.  I cannot use fiber 
channel zoning to make one disk invisible to one machine because that 
only works on the FC port level, but both disks come from the same 
storage array and thus share the same FC port.
Is there any way to manually configure which OSDs are started on 
which machines? The osd configuration block includes the osd name and 
host, so is there a way to say that, say, osd.0 should only be started 
on host vashti and osd.1 should only be started on host zadok?  I tried 
using this configuration:


   [osd.0]
   host=vashti
   dev=/dev/disk/by-uuid/b1fd5281-1dc6-41dd-a48a-b7ee29843e38

   [osd.1]
   host=zadok
   dev=/dev/disk/by-uuid/ee079c7b-6f24-46d7-ba3a-c371367bdaea

But the init script still starts both of them. Is there any way to 
disable the automatic scanning of disks?


I'm stuck with this hardware so hopefully there is a way to make it 
work. Thanks for any help.


Kevin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw setting puplic ACLs fails.

[Shain Miley]

Derek,
That's great...I am hopeful it makes it into the next release too...it will 
solve several issues we are having, trying to working around radosgw bucket and 
object permissions when there are multiple users writing files to our buckets.

And with the 's3cmd setacl' failing...at this point I don't see too many other 
alternatives for us.

Thanks again,

Shain

Shain Miley | Manager of Systems and Infrastructure, Digital Media | 
smi...@npr.org | 202.513.3649


From: Derek Yarnell [de...@umiacs.umd.edu]
Sent: Wednesday, November 27, 2013 11:21 AM
To: Shain Miley
Cc: de...@umiacs.umd.edu; ceph-users
Subject: Re: [ceph-users] radosgw setting puplic ACLs fails.

On 11/26/13, 3:31 PM, Shain Miley wrote:
> Micha,
>
> Did you ever figure out a work around for this issue?
>
> I also had plans of using s3cmd to put, and recursively set acl's on a 
> nightly basis...however we are getting the 403 errors as well during our 
> testing.
>
> I was just wondering if you were able to find another solution.

Hi,

There is code[1] in the master branch (I am not sure but I hope it will
make it into the next stable release, it is not in 0.72.x) that allows
you defer to the bucket ACLs.  defer_to_bucket_acls is the configurable
which allows for two different modes.  Recurse just propagates the
specific bucket acls to all the keys, it does fall through to the key
ACL if the bucket ACL doesn't apply.  Full_control allows someone with
FULL_CONTROL at the bucket level to do whatever they want to the keys
(including replace the whole ACL), and again falls through to the key ACL.

Note this breaks AWS S3 compatibility and is why it is a configurable.

[1] - https://github.com/ceph/ceph/pull/672

Thanks,
derek

--
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Minimum requirements for ceph monitors?

[Gruher, Joseph R]

>For ~$67 you get a mini-itx motherboard with a soldered on 17W dual core
>1.8GHz ivy-bridge based Celeron (supports SSE4.2 CRC32 instructions!).
>It has 2 standard dimm slots so no compromising on memory, on-board gigabit
>eithernet, 3 3Gb/s + 1 6Gb/s SATA, and a single PCIE slot for an additional 
>NIC.
>This has the potential to make a very competent low cost, lowish power OSD
>or mon server.  The biggest downside is that it doesn't appear to support ECC
>memory.  Some of the newer Atoms appear to, so that might be an option as
>well.

Yup, the server and storage purposed Atoms do support ECC.  I think Atom sounds 
like an interesting fit for OSD servers, the new Avoton SoCs are quite fast, 
can host up to 64GB ECC RAM on two channels, and have 4x1GbE or 1x10GbE 
onboard.  Plus six SATA lanes onboard which would be a nice fit for an OS disk, 
a journal SSD and four OSD disks.  I have been hoping to track down a few 
boards and do some testing with Atom myself.

http://ark.intel.com/products/77987/Intel-Atom-Processor-C2750-4M-Cache-2_40-GHz
 

Would be interested to hear if anyone else has tried such an experiment.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD on an external, shared device

[Kevin Horan]

Thanks. I may have to go this route, but it seems awfully fragile. One 
stray  command could destroy the entire cluster, replicas and all. Since 
all disks are visible to all nodes, any one of them could mount 
everything, corrupting all OSDs at once.
Surly other people are using external FC drives, how do you limit 
the visibility of the drives? Am I missing something here? Could there 
be a configuration option or something added to ceph to ensure that it 
never tries to mount things on its own?


Thanks.

Kevin
On 11/26/2013 05:14 PM, Kyle Bader wrote:

 Is there any way to manually configure which OSDs are started on which
machines? The osd configuration block includes the osd name and host, so is
there a way to say that, say, osd.0 should only be started on host vashti
and osd.1 should only be started on host zadok?  I tried using this
configuration:

The ceph udev rules are going to automatically mount disks that match
the ceph "magic" guids, to dig through the full logic you need to
inspect these files:

/lib/udev/rules.d/60-ceph-partuuid-workaround.rules
/lib/udev/rules.d/95-ceph-osd.rules

The upstart scripts look to see what is mounted at /var/lib/ceph/osd/
and starts osd daemons as appropriate:

/etc/init/ceph-osd-all-starter.conf

In theory you should be able to remove the udev scripts and mount the
osds in /var/lib/ceph/osd if your using upstart. You will want to make
sure that upgrades to the ceph package don't replace the files, maybe
that means making a null rule and using "-o
Dpkg::Options::='--force-confold" in ceph-deploy/chef/puppet/whatever.
You will also want to avoid putting the mounts in fstab because it
could render your node unbootable if the device or filesystem fails.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD on an external, shared device

[LaSalle, Jurvis]

Is LUN masking an option in your SAN?

On 11/27/13, 2:34 PM, "Kevin Horan"  wrote:

>Thanks. I may have to go this route, but it seems awfully fragile. One
>stray  command could destroy the entire cluster, replicas and all. Since
>all disks are visible to all nodes, any one of them could mount
>everything, corrupting all OSDs at once.
> Surly other people are using external FC drives, how do you limit
>the visibility of the drives? Am I missing something here? Could there
>be a configuration option or something added to ceph to ensure that it
>never tries to mount things on its own?
>
>Thanks.
>
>Kevin
>On 11/26/2013 05:14 PM, Kyle Bader wrote:
>>>  Is there any way to manually configure which OSDs are started on
>>>which
>>> machines? The osd configuration block includes the osd name and host,
>>>so is
>>> there a way to say that, say, osd.0 should only be started on host
>>>vashti
>>> and osd.1 should only be started on host zadok?  I tried using this
>>> configuration:
>> The ceph udev rules are going to automatically mount disks that match
>> the ceph "magic" guids, to dig through the full logic you need to
>> inspect these files:
>>
>> /lib/udev/rules.d/60-ceph-partuuid-workaround.rules
>> /lib/udev/rules.d/95-ceph-osd.rules
>>
>> The upstart scripts look to see what is mounted at /var/lib/ceph/osd/
>> and starts osd daemons as appropriate:
>>
>> /etc/init/ceph-osd-all-starter.conf
>>
>> In theory you should be able to remove the udev scripts and mount the
>> osds in /var/lib/ceph/osd if your using upstart. You will want to make
>> sure that upgrades to the ceph package don't replace the files, maybe
>> that means making a null rule and using "-o
>> Dpkg::Options::='--force-confold" in ceph-deploy/chef/puppet/whatever.
>> You will also want to avoid putting the mounts in fstab because it
>> could render your node unbootable if the device or filesystem fails.
>>
>
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD on an external, shared device

[Kevin Horan]

Ah, that sounds like what I want. I'll look into that, thanks.

Kevin

On 11/27/2013 11:37 AM, LaSalle, Jurvis wrote:

Is LUN masking an option in your SAN?

On 11/27/13, 2:34 PM, "Kevin Horan"  wrote:


Thanks. I may have to go this route, but it seems awfully fragile. One
stray  command could destroy the entire cluster, replicas and all. Since
all disks are visible to all nodes, any one of them could mount
everything, corrupting all OSDs at once.
 Surly other people are using external FC drives, how do you limit
the visibility of the drives? Am I missing something here? Could there
be a configuration option or something added to ceph to ensure that it
never tries to mount things on its own?

Thanks.

Kevin
On 11/26/2013 05:14 PM, Kyle Bader wrote:

  Is there any way to manually configure which OSDs are started on
which
machines? The osd configuration block includes the osd name and host,
so is
there a way to say that, say, osd.0 should only be started on host
vashti
and osd.1 should only be started on host zadok?  I tried using this
configuration:

The ceph udev rules are going to automatically mount disks that match
the ceph "magic" guids, to dig through the full logic you need to
inspect these files:

/lib/udev/rules.d/60-ceph-partuuid-workaround.rules
/lib/udev/rules.d/95-ceph-osd.rules

The upstart scripts look to see what is mounted at /var/lib/ceph/osd/
and starts osd daemons as appropriate:

/etc/init/ceph-osd-all-starter.conf

In theory you should be able to remove the udev scripts and mount the
osds in /var/lib/ceph/osd if your using upstart. You will want to make
sure that upgrades to the ceph package don't replace the files, maybe
that means making a null rule and using "-o
Dpkg::Options::='--force-confold" in ceph-deploy/chef/puppet/whatever.
You will also want to avoid putting the mounts in fstab because it
could render your node unbootable if the device or filesystem fails.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Constant slow / blocked requests with otherwise healthy cluster

[Oliver Schulz]

Dear Ceph Experts,

our Ceph cluster suddenly went into a state of OSDs constantly having
blocked or slow requests, rendering the cluster unusable. This happened
during normal use, there were no updates, etc.

All disks seem to be healthy (smartctl, iostat, etc.). A complete
hardware reboot including system update on all nodes has not helped.
The network equipment also shows no trouble.

We'd be glad for any advice on how to diagnose and solve this, as
the cluster is basically at a standstill and we urgently need
to get it back into operation.

Cluster structure: 6 Nodes, 6x 3TB disks plus 1x System/Journal SSD
per node, one OSD per disk. We're running ceph version 0.67.4-1precise
on Ubuntu 12.04.3 with kernel 3.8.0-33-generic (x86_64).

"ceph status" shows something like (it varies):

cluster 899509fe-afe4-42f4-a555-bb044ca0f52d
 health HEALTH_WARN 77 requests are blocked > 32 sec
 monmap e1: 3 mons at 
{a=134.107.24.179:6789/0,b=134.107.24.181:6789/0,c=134.107.24.183:6789/0}, 
election epoch 312, quorum 0,1,2 a,b,c
 osdmap e32600: 36 osds: 36 up, 36 in
  pgmap v16404527: 14304 pgs: 14304 active+clean; 20153 GB data, 60630 GB 
used, 39923 GB / 100553 GB avail; 1506KB/s rd, 21246B/s wr, 545op/s
 mdsmap e478: 1/1/1 up {0=c=up:active}, 1 up:standby-replay

"ceph health detail" shows something like (it varies):

HEALTH_WARN 363 requests are blocked > 32 sec; 22 osds have slow requests
363 ops are blocked > 32.768 sec
1 ops are blocked > 32.768 sec on osd.0
8 ops are blocked > 32.768 sec on osd.3
37 ops are blocked > 32.768 sec on osd.12
[...]
11 ops are blocked > 32.768 sec on osd.62
45 ops are blocked > 32.768 sec on osd.65
22 osds have slow requests

The number and identity of affected OSDs constantly changes
(sometimes health even goes to OK for a moment).


Cheers and thanks for any ideas,

Oliver
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw daemon stalls on download of some files

[Yehuda Sadeh]

On Wed, Nov 27, 2013 at 4:46 AM, Sebastian  wrote:
> Hi,
>
> we have a setup of 4 Servers running ceph and radosgw. We use it as an 
> internal S3 service for our files. The Servers run Debian Squeeze with Ceph 
> 0.67.4.
>
> The cluster has been running smoothly for quite a while, but we are currently 
> experiencing issues with the radosgw. For some files the HTTP Download just 
> stalls at around 500kb.
>
> The Apache error log just says:
> [error] [client ] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: 
> idle timeout (30 sec)
> [error] [client ] Handler for fastcgi-script returned invalid result code 1
>
> radosgw logging:
> 7f00bc66a700  1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 
> 0x7f00934bb700' had timed out after 600
> 7f00bc66a700  1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 
> 0x7f00ab4eb700' had timed out after 600
>
> The interesting thing is that the cluster health is fine an only some files 
> are not working properly. Most of them just work fine. A restart of radosgw 
> fixes the issue. The other ceph logs are also clean.
>
> Any idea why this happens?
>

No, but you can turn on 'debug ms = 1' on your gateway ceph.conf, and
that might give some better indication.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Container size via s3api

[Yehuda Sadeh]

On Wed, Nov 27, 2013 at 12:24 AM, Mihály Árva-Tóth
 wrote:
> 2013/11/26 Derek Yarnell 
>>
>> On 11/26/13, 4:04 AM, Mihály Árva-Tóth wrote:
>> > Hello,
>> >
>> > Is there any idea? I don't know this is s3api limitation or missing
>> > feature?
>> >
>> > Thank you,
>> > Mihaly
>>
>> Hi Mihaly,
>>
>> If all you are looking for is the current size of the bucket this can be
>> found from the adminops api[1] or when you get do the GET bucket[2]
>> operation you will receive a list of keys that you can loop through and
>> total their sizes.
>>
>> [1] - http://ceph.com/docs/master/radosgw/adminops/
>> [2] - http://ceph.com/docs/master/radosgw/s3/bucketops/#get-bucket
>
>
> Hi Derek,
>
> Thank you for your answer. Okay so I see I can't do that with s3api. I would
> not like to list - loop - sum() because containers holds 5-10 millions of
> objects and I have fifty containers at the moment. Adminops api is much
> interesting thanks I will make some research.
>


The S3 api does not provide that info. As was mentioned above you can
try using the admin api.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Constant slow / blocked requests with otherwise healthy cluster

[Andrey Korolyov]

Hey,

What number do you have for a replication factor? As for three, 1.5k
IOPS may be a little bit high for 36 disks, and your OSD ids looks a bit
suspicious - there should not be 60+ OSDs based on calculation from
numbers below.

On 11/28/2013 12:45 AM, Oliver Schulz wrote:
> Dear Ceph Experts,
> 
> our Ceph cluster suddenly went into a state of OSDs constantly having
> blocked or slow requests, rendering the cluster unusable. This happened
> during normal use, there were no updates, etc.
> 
> All disks seem to be healthy (smartctl, iostat, etc.). A complete
> hardware reboot including system update on all nodes has not helped.
> The network equipment also shows no trouble.
> 
> We'd be glad for any advice on how to diagnose and solve this, as
> the cluster is basically at a standstill and we urgently need
> to get it back into operation.
> 
> Cluster structure: 6 Nodes, 6x 3TB disks plus 1x System/Journal SSD
> per node, one OSD per disk. We're running ceph version 0.67.4-1precise
> on Ubuntu 12.04.3 with kernel 3.8.0-33-generic (x86_64).
> 
> "ceph status" shows something like (it varies):
> 
> cluster 899509fe-afe4-42f4-a555-bb044ca0f52d
>  health HEALTH_WARN 77 requests are blocked > 32 sec
>  monmap e1: 3 mons at
> {a=134.107.24.179:6789/0,b=134.107.24.181:6789/0,c=134.107.24.183:6789/0},
> election epoch 312, quorum 0,1,2 a,b,c
>  osdmap e32600: 36 osds: 36 up, 36 in
>   pgmap v16404527: 14304 pgs: 14304 active+clean; 20153 GB data,
> 60630 GB used, 39923 GB / 100553 GB avail; 1506KB/s rd, 21246B/s wr,
> 545op/s
>  mdsmap e478: 1/1/1 up {0=c=up:active}, 1 up:standby-replay
> 
> "ceph health detail" shows something like (it varies):
> 
> HEALTH_WARN 363 requests are blocked > 32 sec; 22 osds have slow
> requests
> 363 ops are blocked > 32.768 sec
> 1 ops are blocked > 32.768 sec on osd.0
> 8 ops are blocked > 32.768 sec on osd.3
> 37 ops are blocked > 32.768 sec on osd.12
> [...]
> 11 ops are blocked > 32.768 sec on osd.62
> 45 ops are blocked > 32.768 sec on osd.65
> 22 osds have slow requests
> 
> The number and identity of affected OSDs constantly changes
> (sometimes health even goes to OK for a moment).
> 
> 
> Cheers and thanks for any ideas,
> 
> Oliver
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Constant slow / blocked requests with otherwise healthy cluster

[Michael]

Sounds like what I was having starting a couple of days ago, played 
around with the conf, taking in/out suspect osd and doing full smart 
tests on them that came back perfectly fine, doing network tests that 
came back 110MB/s on all channels, doing OSD benches that reported all 
OSD managing 80+ MB/s happily while still under use.
Yet doing a RBD bench for only 10 seconds would report terrible write 
speed quickly dropping to 0MB/s then the whole cluster would report slow 
writes and block reads for up to a couple of minutes >>or until osd's 
were restarted<< (Makes me concerned there's a bug somewhere).


Updated to 3.12 kernel and restarted all of the ceph nodes and it's now 
happily churning through a rados -p rbd bench 300 write -t 120 that 
would have killed it in seconds for the full 5 minute duration at 
100MB/s average without any of the VM's running of it responding 
noticeably slower.


Wish I knew what actually caused it. :/

What version of ceph are you on?

-Michael

On 27/11/2013 21:00, Andrey Korolyov wrote:

Hey,

What number do you have for a replication factor? As for three, 1.5k
IOPS may be a little bit high for 36 disks, and your OSD ids looks a bit
suspicious - there should not be 60+ OSDs based on calculation from
numbers below.

On 11/28/2013 12:45 AM, Oliver Schulz wrote:

Dear Ceph Experts,

our Ceph cluster suddenly went into a state of OSDs constantly having
blocked or slow requests, rendering the cluster unusable. This happened
during normal use, there were no updates, etc.

All disks seem to be healthy (smartctl, iostat, etc.). A complete
hardware reboot including system update on all nodes has not helped.
The network equipment also shows no trouble.

We'd be glad for any advice on how to diagnose and solve this, as
the cluster is basically at a standstill and we urgently need
to get it back into operation.

Cluster structure: 6 Nodes, 6x 3TB disks plus 1x System/Journal SSD
per node, one OSD per disk. We're running ceph version 0.67.4-1precise
on Ubuntu 12.04.3 with kernel 3.8.0-33-generic (x86_64).

"ceph status" shows something like (it varies):

 cluster 899509fe-afe4-42f4-a555-bb044ca0f52d
  health HEALTH_WARN 77 requests are blocked > 32 sec
  monmap e1: 3 mons at
{a=134.107.24.179:6789/0,b=134.107.24.181:6789/0,c=134.107.24.183:6789/0},
election epoch 312, quorum 0,1,2 a,b,c
  osdmap e32600: 36 osds: 36 up, 36 in
   pgmap v16404527: 14304 pgs: 14304 active+clean; 20153 GB data,
60630 GB used, 39923 GB / 100553 GB avail; 1506KB/s rd, 21246B/s wr,
545op/s
  mdsmap e478: 1/1/1 up {0=c=up:active}, 1 up:standby-replay

"ceph health detail" shows something like (it varies):

 HEALTH_WARN 363 requests are blocked > 32 sec; 22 osds have slow
requests
 363 ops are blocked > 32.768 sec
 1 ops are blocked > 32.768 sec on osd.0
 8 ops are blocked > 32.768 sec on osd.3
 37 ops are blocked > 32.768 sec on osd.12
 [...]
 11 ops are blocked > 32.768 sec on osd.62
 45 ops are blocked > 32.768 sec on osd.65
 22 osds have slow requests

The number and identity of affected OSDs constantly changes
(sometimes health even goes to OK for a moment).


Cheers and thanks for any ideas,

Oliver
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 回复：Re: testing ceph performance issue

[Kyle Bader]

> How much performance can be improved if use SSDs  to storage journals?

You will see roughly twice the throughput unless you are using btrfs
(still improved but not as dramatic). You will also see lower latency
because the disk head doesn't have to seek back and forth between
journal and data partitions.

>   Kernel RBD Driver  ,  what is this ?

There are several RBD implementations, one is the kernel RBD driver in
upstream Linux, another is built into Qemu/KVM.

> and we want to know the RBD if  support XEN virual  ?

It is possible, but not nearly as well tested and not prevalent as RBD
via Qemu/KVM. This might be a starting point if your interested in
testing Xen/RBD integration:

http://wiki.xenproject.org/wiki/Ceph_and_libvirt_technology_preview

Hope that helps!

-- 

Kyle
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw setting puplic ACLs fails.

[Yehuda Sadeh]

I just pushed a fix for review for the s3cmd --setacl issue. It should
land a stable release soonish.

Thanks,
Yehuda

On Wed, Nov 27, 2013 at 10:12 AM, Shain Miley  wrote:
> Derek,
> That's great...I am hopeful it makes it into the next release too...it will 
> solve several issues we are having, trying to working around radosgw bucket 
> and object permissions when there are multiple users writing files to our 
> buckets.
>
> And with the 's3cmd setacl' failing...at this point I don't see too many 
> other alternatives for us.
>
> Thanks again,
>
> Shain
>
> Shain Miley | Manager of Systems and Infrastructure, Digital Media | 
> smi...@npr.org | 202.513.3649
>
> 
> From: Derek Yarnell [de...@umiacs.umd.edu]
> Sent: Wednesday, November 27, 2013 11:21 AM
> To: Shain Miley
> Cc: de...@umiacs.umd.edu; ceph-users
> Subject: Re: [ceph-users] radosgw setting puplic ACLs fails.
>
> On 11/26/13, 3:31 PM, Shain Miley wrote:
>> Micha,
>>
>> Did you ever figure out a work around for this issue?
>>
>> I also had plans of using s3cmd to put, and recursively set acl's on a 
>> nightly basis...however we are getting the 403 errors as well during our 
>> testing.
>>
>> I was just wondering if you were able to find another solution.
>
> Hi,
>
> There is code[1] in the master branch (I am not sure but I hope it will
> make it into the next stable release, it is not in 0.72.x) that allows
> you defer to the bucket ACLs.  defer_to_bucket_acls is the configurable
> which allows for two different modes.  Recurse just propagates the
> specific bucket acls to all the keys, it does fall through to the key
> ACL if the bucket ACL doesn't apply.  Full_control allows someone with
> FULL_CONTROL at the bucket level to do whatever they want to the keys
> (including replace the whole ACL), and again falls through to the key ACL.
>
> Note this breaks AWS S3 compatibility and is why it is a configurable.
>
> [1] - https://github.com/ceph/ceph/pull/672
>
> Thanks,
> derek
>
> --
> Derek T. Yarnell
> University of Maryland
> Institute for Advanced Computer Studies
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph as offline S3 substitute and peer-to-peer fileshare?

[Alek Storm]

Hi all,

I'd like to use Ceph to solve two problems at my company: to be an S3 mock
for testing our application, and for sharing test artifacts in a
peer-to-peer fashion between developers.

We currently store immutable binary blobs ranging from a few kB to several
hundred MB in S3, which means both integration testing and most normal
development is impossible when offline. Separately, we share these binary
blobs with each other through Dropbox (for local tweaking; we like
Dropbox's filesystem emulation) - unfortunately, as we grow, this is
becoming prohibitively expensive.

I'd like to set up Ceph as follows: each developer runs a single OSD on
their personal machine. Each OSD has a complete replica of all data; we'll
keep the total size to a few gigs at most. To sync with other boxes, the
developer can connect to a Ceph monitor on our corporate VPN whenever they
happen to be online, in order to initiate peering. Each developer, when
testing, points their local stack at their own OSD instance, instead of the
AWS S3 endpoint.

I know what I've described is a bit unusual - is Ceph the right tool for
this? Can you give any pointers to particular config settings or places in
the documentation I should be looking? I'm only just diving in now; Ceph's
flexibility can be a bit overwhelming.

Thanks in advance,
Alek
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] tracker.ceph.com - public email address visibility?

[Josh Durgin]

On 11/27/2013 07:21 AM, James Pearce wrote:

I was going to add something to the bug tracker, but it looks to me that
contributor email addresses all have public (unauthenticated)
visibility?  Can this be set in user preferences?


Yes, it can be hidden here: http://tracker.ceph.com/my/account
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Real size of rbd image

[Josh Durgin]

On 11/26/2013 02:22 PM, Stephen Taylor wrote:

 From ceph-users archive 08/27/2013:

On 08/27/2013 01:39 PM, Timofey Koolin wrote:


/Is way to know real size of rbd image and rbd snapshots?/



/rbd ls -l write declared size of image, but I want to know real size./


You can sum the sizes of the extents reported by:

  rbd diff pool/image[@snap] [--format json]

That's the difference since the beginning of time, so it reports all

used extents.

Josh

I don’t seem to be able to find any documentation supporting the [@snap]
parameter for this call, but it seems to work, at least in part. I have
a requirement to find the size of a snapshot relative to another
snapshot. Here is what I’ve used:

 rbd diff pool/image@snap2 --from-snap snap1


Most rbd commands work on snapshots too. The help text could certainly
be improved - suggestions welcome!


The returned list of extents seems to include all changes since snap1,
not just those up to snap2, but those that have been written after snap2
are labeled “zero” rather than as “data” extents. If I ignore the “zero”
extents and sum the lengths of the “data” extents, it seems to give me
an accurate relative snapshot size. Is this expected behavior and the
correct way to calculate the size I’m looking for?


Do you have discard/trim enabled for whatever's using the image?
The diff will include discarded extents as "zero". For calculating
size, it's fine to ignore them. It would be unexpected if these
aren't listed when you leave out the @snap2 portion though.

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] can not get rbd cache perf counter

[Josh Durgin]

On 11/27/2013 01:31 AM, Shu, Xinxin wrote:

Recently,  I want to test performance benefit of rbd cache, i cannot get
obvious performance benefit at my setup, then I  try to make sure rbd
cache is enabled, but I cannot get rbd cache perf counter. In order to
identify how to enable rbd cache perf counter, I setup a simple
setup(one client hosted vms, one ceph cluster with two OSDs, each osd
has a SSD partition for journal.), then build ceph-0.67.4.

My ceph.conf shows as bellows:

[global]

 debug default = 0

 log file = /var/log/ceph/$name.log

 max open files = 131072

 auth cluster required = none

 auth service required = none

 auth client required = none

 rbd cache = true

[mon.a]

 host = {monitor_host_name}

mon addr = {monitor_addr}

[osd.0]

 host = {osd.0_hostname}

 public addr = {public_addr}

 cluster addr = {cluster_addr}

 osd mkfs type = xfs

 devs = /dev/sdb1

 osd journal = /dev/sdd5

[osd.1]

 host = {osd.1_hostname}

 public addr = {public_addr}

 cluster addr = {cluster_addr}

 osd mkfs type = xfs

 devs = /dev/sdc1

 osd journal = /dev/sdd6

after ceph cluster is built, I create a rbd image with rbd create –size
10240 –new-format test

then use virsh to start a vm, below is my vm xml file



   test

   524288

   524288

   1

   

 hvm

 

   

   

 

 

   

   

 

   

   

 

 

   

   destroy

   restart

   destroy

   

 /usr/bin/qemu-system-x86_64

 

   

   

   

   

 

  

   

   

   

   i

 

 

   

 

   



Then I add a rbd admin socket in ceph.conf on my client, below is the config

[global]

 auth cluster required = none

 auth service required = none

 auth client required = none

 rbd cache = true

 rbd cache writethrough until flush = true

[client]

 admin socket=/var/run/ceph/rbd-$pid.asok

[mon.a]

 host = {monitor_host_name}

 mon addr = {monitor_host_addr}

then I checked rbd cache perf counter by this socket, but the output did
not get any rbd cache statistics

ceph --admin-daemon /var/run/ceph/rbd-3526.asok perf dump output

{ "objecter": { "op_active": 0,

   "op_laggy": 0,

   "op_send": 0,

   "op_send_bytes": 0,

   "op_resend": 0,

   "op_ack": 0,

   "op_commit": 0,

   "op": 0,

   "op_r": 0,

   "op_w": 0,

   "op_rmw": 0,

   "op_pg": 0,

   "osdop_stat": 0,

   "osdop_create": 0,

   "osdop_read": 0,

   "osdop_write": 0,

   "osdop_writefull": 0,

   "osdop_append": 0,

   "osdop_zero": 0,

   "osdop_truncate": 0,

   "osdop_delete": 0,

   "osdop_mapext": 0,

   "osdop_sparse_read": 0,

   "osdop_clonerange": 0,

   "osdop_getxattr": 0,

   "osdop_setxattr": 0,

   "osdop_cmpxattr": 0,

   "osdop_rmxattr": 0,

   "osdop_resetxattrs": 0,

   "osdop_tmap_up": 0,

   "osdop_tmap_put": 0,

   "osdop_tmap_get": 0,

   "osdop_call": 0,

   "osdop_watch": 0,

   "osdop_notify": 0,

   "osdop_src_cmpxattr": 0,

   "osdop_pgls": 0,

   "osdop_pgls_filter": 0,

   "osdop_other": 0,

   "linger_active": 0,

   "linger_send": 0,

   "linger_resend": 0,

   "poolop_active": 0,

   "poolop_send": 0,

   "poolop_resend": 0,

   "poolstat_active": 0,

   "poolstat_send": 0,

   "poolstat_resend": 0,

   "statfs_active": 0,

   "statfs_send": 0,

   "statfs_resend": 0,

   "command_active": 0,

   "command_send": 0,

   "command_resend": 0,

   "map_epoch": 0,

   "map_full": 0,

   "map_inc": 0,

   "osd_sessions": 0,

   "osd_session_open": 0,

   "osd_session_close": 0,

   "osd_laggy": 0},

   "throttle-msgr_dispatch_throttler-radosclient": { "val": 0,

   "max": 104857600,

   "get": 11,

   "get_sum": 5655,

   "get_or_fail_fail": 0,

   "get_or_fail_success": 0,

   "take": 0,

   "take_sum": 0,

   "put": 11,

   "put_sum": 5655,

   "wait": { "avgcount": 0,

   "sum": 0.0}},

   "throttle-objecter_bytes": { "val": 0,

   "max": 104857600,

   "get": 0,

   "get_sum": 0,

   "get_or_fail_fail": 0,

   "get_or_fail_success": 0,

   "take": 0,

   "take_sum": 0,

   "put": 0,

   "put_sum": 0,

   "wait": { "avgcount": 0,

   "sum": 0.0}},

   "throttle-objecter_ops": { "val": 0,

   "max": 1024,

   "get": 0,

   "get_sum": 0,

   "get_or_fail_fail": 0,

   "get_or_fail_success": 0,

   "take": 0,

   "take_sum": 0,

   "put": 0,

   "put_sum": 0,

   "wait": { "avgcount": 0,

   "sum": 0.0}}}

Qemu version:  qemu-system-x86_64 --version

QEMU emulator version 1.2.0 (qemu-kvm-1.2.0+noroms-0ubuntu2.12.10.5,
Debian), Copyright (c) 2003-2008 Fabric

Re: [ceph-users] [Big Problem?] Why not using Device'UUID in ceph.conf

[Josh Durgin]

On 11/26/2013 01:14 AM, Ta Ba Tuan wrote:

Hi James,

Proplem is why the Ceph not  recommend using Device'UUID in Ceph.conf,
when, above error can be occur?


I think with the newer-style configuration, where your disks have
partition ids setup by ceph-disk instead of entries in ceph.conf, it
doesn't matter if they change names, as long as they mount point stays
the same.

Josh


--
TuanTaBa


On 11/26/2013 04:04 PM, James Harper wrote:

Hi all

I have 3 OSDs, named sdb, sdc, sdd.
Suppose, one OSD with device /dev/sdc die => My server have only sdb,
sdc
at the moment.
Because device /dev/sdc replaced by /dev/sdd

Can you just use one of the /dev/disk/by-/
symlinks?

Eg
/dev/disk/by-uuid/153cf32b-e46b-4d31-95ef-749db3a88d02
/dev/disk/by-id/scsi-SATA_WDC_WD10EACS-00D_WD-WCAU66606660

Your distribution should allow for such things automatically, and if
not you should be able to add some udev rules to do it.

James


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] can not get rbd cache perf counter

[Josh Durgin]

[re-adding the list]

It's not related to the version of qemu. When qemu starts up, it
creates the admin socket file, but it needs write access to do that.

Does the user running qemu (libvirt-qemu on ubuntu) have write access
to /var/run/ceph? It may be unix permissions blocking it, or apparmor
or selinux if those are enabled.

On 11/27/2013 07:20 PM, Shu, Xinxin wrote:

Hi josh,
   Thanks for your reply,  the pid in the filename did not match  kvm process, 
since I add option in ceph.conf for rbd admin socket, why not qemu create this 
admin socket, is this due to qemu is not installed correctly or this rbd admin 
socket depends on secified qemu package.

-Original Message-
From: Josh Durgin [mailto:josh.dur...@inktank.com]
Sent: Thursday, November 28, 2013 11:01 AM
To: Shu, Xinxin; ceph-us...@ceph.com
Subject: Re: [ceph-users] can not get rbd cache perf counter

On 11/27/2013 01:31 AM, Shu, Xinxin wrote:

Recently,  I want to test performance benefit of rbd cache, i cannot
get obvious performance benefit at my setup, then I  try to make sure
rbd cache is enabled, but I cannot get rbd cache perf counter. In
order to identify how to enable rbd cache perf counter, I setup a
simple setup(one client hosted vms, one ceph cluster with two OSDs,
each osd has a SSD partition for journal.), then build ceph-0.67.4.

My ceph.conf shows as bellows:

[global]

  debug default = 0

  log file = /var/log/ceph/$name.log

  max open files = 131072

  auth cluster required = none

  auth service required = none

  auth client required = none

  rbd cache = true

[mon.a]

  host = {monitor_host_name}

mon addr = {monitor_addr}

[osd.0]

  host = {osd.0_hostname}

  public addr = {public_addr}

  cluster addr = {cluster_addr}

  osd mkfs type = xfs

  devs = /dev/sdb1

  osd journal = /dev/sdd5

[osd.1]

  host = {osd.1_hostname}

  public addr = {public_addr}

  cluster addr = {cluster_addr}

  osd mkfs type = xfs

  devs = /dev/sdc1

  osd journal = /dev/sdd6

after ceph cluster is built, I create a rbd image with rbd create
-size
10240 -new-format test

then use virsh to start a vm, below is my vm xml file



test

524288

524288

1



  hvm

  





  

  





  





  

  



destroy

restart

destroy



  /usr/bin/qemu-system-x86_64

  









  

   







i

  

  



  





Then I add a rbd admin socket in ceph.conf on my client, below is the
config

[global]

  auth cluster required = none

  auth service required = none

  auth client required = none

  rbd cache = true

  rbd cache writethrough until flush = true

[client]

  admin socket=/var/run/ceph/rbd-$pid.asok

[mon.a]

  host = {monitor_host_name}

  mon addr = {monitor_host_addr}

then I checked rbd cache perf counter by this socket, but the output
did not get any rbd cache statistics

ceph --admin-daemon /var/run/ceph/rbd-3526.asok perf dump output

{ "objecter": { "op_active": 0,

"op_laggy": 0,

"op_send": 0,

"op_send_bytes": 0,

"op_resend": 0,

"op_ack": 0,

"op_commit": 0,

"op": 0,

"op_r": 0,

"op_w": 0,

"op_rmw": 0,

"op_pg": 0,

"osdop_stat": 0,

"osdop_create": 0,

"osdop_read": 0,

"osdop_write": 0,

"osdop_writefull": 0,

"osdop_append": 0,

"osdop_zero": 0,

"osdop_truncate": 0,

"osdop_delete": 0,

"osdop_mapext": 0,

"osdop_sparse_read": 0,

"osdop_clonerange": 0,

"osdop_getxattr": 0,

"osdop_setxattr": 0,

"osdop_cmpxattr": 0,

"osdop_rmxattr": 0,

"osdop_resetxattrs": 0,

"osdop_tmap_up": 0,

"osdop_tmap_put": 0,

"osdop_tmap_get": 0,

"osdop_call": 0,

"osdop_watch": 0,

"osdop_notify": 0,

"osdop_src_cmpxattr": 0,

"osdop_pgls": 0,

"osdop_pgls_filter": 0,

"osdop_other": 0,

"linger_active": 0,

"linger_send": 0,

"linger_resend": 0,

"poolop_active": 0,

"poolop_send": 0,

"poolop_resend": 0,

"poolstat_active": 0,

"poolstat_send": 0,

"poolstat_resend": 0,

"statfs_active": 0,

"statfs_send": 0,

"statfs_resend": 0,

"command_active": 0,

"command_send": 0,

"command_resend": 0,

"map_epoch": 0,

"map_full": 0,

"map_inc": 0,

"osd_sessions": 0,

"osd_session_open": 0,

"osd_session_close": 0,

"osd_laggy": 0},

"throttle-msgr_dispatch_throttler-radosclient": { "val": 0,

"max": 104857600,

"get": 11,

"get_sum":

Re: [ceph-users] Container size via s3api

[Mihály Árva-Tóth]

2013/11/27 Yehuda Sadeh 

> On Wed, Nov 27, 2013 at 12:24 AM, Mihály Árva-Tóth
>  wrote:
> > 2013/11/26 Derek Yarnell 
> >>
> >> On 11/26/13, 4:04 AM, Mihály Árva-Tóth wrote:
> >> > Hello,
> >> >
> >> > Is there any idea? I don't know this is s3api limitation or missing
> >> > feature?
> >> >
> >> > Thank you,
> >> > Mihaly
> >>
> >> Hi Mihaly,
> >>
> >> If all you are looking for is the current size of the bucket this can be
> >> found from the adminops api[1] or when you get do the GET bucket[2]
> >> operation you will receive a list of keys that you can loop through and
> >> total their sizes.
> >>
> >> [1] - http://ceph.com/docs/master/radosgw/adminops/
> >> [2] - http://ceph.com/docs/master/radosgw/s3/bucketops/#get-bucket
> >
> >
> > Hi Derek,
> >
> > Thank you for your answer. Okay so I see I can't do that with s3api. I
> would
> > not like to list - loop - sum() because containers holds 5-10 millions of
> > objects and I have fifty containers at the moment. Adminops api is much
> > interesting thanks I will make some research.
> >
>
>
> The S3 api does not provide that info. As was mentioned above you can
> try using the admin api.
>
> Yehuda
>

Hi Yehuda,

Thank you, so this is an s3api limitation. Adminapi is great for me.

Regards,
Mihaly
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com