[ceph-users] Recommended OSD size

Hi All, We need some clarification on CEPH OSD and MON and MDS. It will be very helpful and better understand to know below details. Per OSD Recommended SIZE ( Both scsi and ssd ). Which is recommended one (per machine = per OSD) or (Per machine = many OSD.) Do we need run sepa

Re: [ceph-users] PGS stuck inactive and osd down

> On 12 May 2016, at 19:27, Vincenzo Pii wrote: > > I have installed a new ceph cluster with ceph-ansible (using the same version > and playbook that had worked before, with some necessary changes to > variables). > > The only major difference is that now an osd (osd3) has a disk twice as big

Re: [ceph-users] How do ceph clients determine a monitor's address (and esp. port) for initial connection?

Hi Greg, Thanks again and good guess! Amending testcluster.conf as follows: mon host = 192.168.10.201:6788 mon addr = 192.168.10.201:6788 ... gets around the problem. having "mon host = mona:6788" also works. Should I raise a defect or is this workaround good enough? Cheers, Christian On 1

Re: [ceph-users] Recommended OSD size

On Fri, 13 May 2016 12:38:05 +0530 gjprabu wrote: Hello, > Hi All, > > > > We need some clarification on CEPH OSD and MON and MDS. It will > be very helpful and better understand to know below details. > You will want to spend more time reading the documentation and hardware guides, as

Re: [ceph-users] rbd resize option

Thank you.now its working"resize2fs" and then "rbd resize" Thanks Swami On Thu, May 12, 2016 at 7:40 PM, M Ranga Swami Reddy wrote: > sure...checking the resize2fs before using the "rbd resize"... > > Thanks > Swami > > On Thu, May 12, 2016 at 7:17 PM, Eneko Lacunza wrote: >> You have t

[ceph-users] Ceph Recovery

Hi, After disaster and restarting for automatic recovery, I found following ceph status. Some OSDs cannot be restarted due to file system corruption (it seem that xfs is fragile). [root@management-b ~]# ceph status cluster 3810e9eb-9ece-4804-8c56-b986e7bb5627 health HEALTH_WARN

Re: [ceph-users] Ceph Recovery

> Op 13 mei 2016 om 11:34 schreef Lazuardi Nasution : > > > Hi, > > After disaster and restarting for automatic recovery, I found following > ceph status. Some OSDs cannot be restarted due to file system corruption > (it seem that xfs is fragile). > > [root@management-b ~]# ceph status > c

Re: [ceph-users] ceph-mon not starting on boot with systemd and Ubuntu 16.04

No luck either. After a reboot only the Ceph OSD starts, but the monitor not. I have checked: - service is enabled - tried to re-enable the service - check the MON logs to see if it was started, it wasn't - systemd log to see if it wants to start the MON, it doesn't My systemd-foo isn't that good

Re: [ceph-users] Ceph Recovery

Hi Wido, The status is same after 24 hour running. It seem that the status will not go to fully active+clean until all down OSDs back again. The only way to make down OSDs to go back again is reformating or replace if HDDs has hardware issue. Do you think that it is safe way to do? Best regards,

Re: [ceph-users] Ceph Recovery

> Op 13 mei 2016 om 11:55 schreef Lazuardi Nasution : > > > Hi Wido, > > The status is same after 24 hour running. It seem that the status will not > go to fully active+clean until all down OSDs back again. The only way to > make down OSDs to go back again is reformating or replace if HDDs has

[ceph-users] Ceph-Disk Prepare Bug

Hi, It seem there is bug on Infernalis "ceph-disk prepare" command to whole disk. Following are some combination of "parted" result after doing that command. Some time the data partition is not created, some time data partition is not formated and prepared well. Bad result: Number Start End

Re: [ceph-users] Recommended OSD size

Hello! On Fri, May 13, 2016 at 04:53:56PM +0900, chibi wrote: >> We need some clarification on CEPH OSD and MON and MDS. It will >> be very helpful and better understand to know below details. >> > You will want to spend more time reading the documentation and hardware > guides, as well

Re: [ceph-users] Try to find the right way to enable rbd-mirror.

Hi Dillaman, Thank you for getting back to me. My system is ubuntu, so I using "sudo rbd-mirror --cluster=local --log-file=mirror.log --debug-rbd-mirror=20/5" instead. I was read your reply but still confused. The image journaling is enable. ---rbd info start--- $ rbd in

Re: [ceph-users] Recommended OSD size

> >> And there is not good answer, it depends on your needs and use case. >> For example if your main goal is space and not performance, fewer but >> larger HDDs will be a better fit. > >In my deployment, I have slow requests when starting OSD with 2.5+ TB used on >it. >Due to slowdowns on start, I

Re: [ceph-users] ceph-mon not starting on boot with systemd and Ubuntu 16.04

This is starting to sound like a xenial systemd issue to me. Maybe poke the canonical folks? You might edit the unit file and make it touch something in /tmp instead of starting Ceph just to rule out ceph... sage On Fri, 13 May 2016, Wido den Hollander wrote: > No luck either. After a reboo

Re: [ceph-users] Try to find the right way to enable rbd-mirror.

On Fri, May 13, 2016 at 6:39 AM, Mika c wrote: > Hi Dillaman, > Thank you for getting back to me. > My system is ubuntu, so I using "sudo rbd-mirror --cluster=local > --log-file=mirror.log --debug-rbd-mirror=20/5" instead. I was read your > reply but still confused. For upstart systems, you can r

[ceph-users] PG active+clean+inconsistent unexpected clone errors in OSD log

Hi all, We have been hit by http://tracker.ceph.com/issues/12954 which caused two OSDs to crash during scrub operations. I have upgraded to 0.94.7 from 0.94.6 to apply a fix for this bug, and everything has been stable so far. However, since this morning 17 scrub errors appeared (which was to be e

Re: [ceph-users] ceph-mon not starting on boot with systemd and Ubuntu 16.04

Can you check that the dependencies have started? Anything about those in the logs? network-online.target local-fs.target ceph-create-keys@%i.service Jan > On 13 May 2016, at 14:30, Sage Weil wrote: > > This is starting to sound like a xenial systemd issue to me. Maybe poke > the canonical

Re: [ceph-users] ceph-mon not starting on boot with systemd and Ubuntu 16.04

> Op 13 mei 2016 om 14:56 schreef Jan Schermer : > > > Can you check that the dependencies have started? Anything about those in the > logs? > > network-online.target local-fs.target ceph-create-keys@%i.service > May 13 16:59:15 alpha systemd[1]: Reached target Local File Systems (Pre). May

[ceph-users] Segfault in libtcmalloc.so.4.2.2

Hi, Been getting some segfaults in our newest ceph cluster running ceph 9.2.1-1 on Debian 8.3 segfault at 0 ip 7f27e85120f7 sp 7f27cff9e860 error 4 in libtcmalloc.so.4.2.2 I saw there’s already a bug up there on the tracker: http://tracker.ceph.com/issues/15628

Re: [ceph-users] Multiple backend pools on the same cacher tier pool ?

On Fri, May 13, 2016 at 8:11 PM, Florent B wrote: > Hi everyone, > > I would like to setup Ceph cache tiering and I would like to know if I > can have a single cache tier pool, used as "hot storage" for multiple > backend pools ? no, we can't. I think it's too complexity to implement this in curr

Re: [ceph-users] Segfault in libtcmalloc.so.4.2.2

What is the exact kernel version ? Ubuntu has a new tcmalloc incorporated from 3.16.0.50 kernel onwards. If you are using older kernel than this better to upgrade kernel or try building latest tcmalloc and try to see if this is happening there. Ceph is not packaging tcmalloc it is using the tcmal

Re: [ceph-users] Weighted Priority Queue testing

Thanks Christian for the input. I will start digging the code and look for possible explanation. Regards Somnath -Original Message- From: Christian Balzer [mailto:ch...@gol.com] Sent: Thursday, May 12, 2016 11:52 PM To: Somnath Roy Cc: Scottix; ceph-users@lists.ceph.com; Nick Fisk Subject

[ceph-users] v0.94.7 Hammer released

This Hammer point release fixes several minor bugs. It also includes a backport of an improved ‘ceph osd reweight-by-utilization’ command for handling OSDs with higher-than-average utilizations. We recommend that all hammer v0.94.x users upgrade. For more detailed information, see the release a

Re: [ceph-users] Multiple backend pools on the same cacher tier pool ?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Haomai Wang > Sent: 13 May 2016 15:00 > To: Florent B > Cc: Ceph Users > Subject: Re: [ceph-users] Multiple backend pools on the same cacher tier > pool ? > > On Fri, May 13, 2016 at 8:11 PM

Re: [ceph-users] Segfault in libtcmalloc.so.4.2.2

Linux osd11.storage 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux apt-show-versions linux-image-3.16.0-4-amd64 linux-image-3.16.0-4-amd64:amd64/jessie-updates 3.16.7-ckt20-1+deb8u3 upgradeable to 3.16.7-ckt25-2 apt-show-versions libtcmalloc-minimal4 libtcmallo

Re: [ceph-users] Segfault in libtcmalloc.so.4.2.2

I am not sure about debian , but, for Ubuntu latest tcmalloc is not incorporated till 3.16.0.50.. You can use the attached program to detect if your tcmalloc is okay or not. Do this.. $ g++ -o gperftest tcmalloc_test.c -ltcmalloc $ TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=67108864 ./gperftest B

[ceph-users] Steps for Adding Cache Tier

I have been reading a lot of information about cache-tiers, and I wanted to know how best to go about adding the cache-tier to a production environment. Our current setup is Infernalis (9.2.1) 4 nodes with 8 x 4TB SATA drives per node and 2 x 400GB NVMe acting as journals (1:4 ratio). There is

[ceph-users] Mount Jewel CephFS to CentOS6

So I see that support for RHEL6 and derivatives was dropped in Jewel (http://ceph.com/releases/v10-2-0-jewel-released/) But is there backward compatibility to mount it using hammer on a node? Doesn't seem to be and that makes some sense, but how can I mount CephFS from a CentOS7-Jewel server to

Re: [ceph-users] Mount Jewel CephFS to CentOS6

Hi, ceph-fuse will be yours. Or, if you can do kernel > 2.6.32 ( or when ever ceph was introduced into the kernel ) then you can also have the kernel mount with hammer. -- Mit freundlichen Gruessen / Best regards Oliver Dzombic IP-Interactive mailto:i...@ip-interactive.de Anschrift: IP In

[ceph-users] Upgrade to Jewel... More interesting things...

So I'm updating a trusty cluster to Jewel and updating the kernel at the same time. Got around some mon issues, and that seems ok, but after upgrading one of my OSD nodes, I'm getting these errors in the old log on that node. *0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8

Re: [ceph-users] Mount Jewel CephFS to CentOS6

On Friday, May 13, 2016, Andrus, Brian Contractor wrote: > So I see that support for RHEL6 and derivatives was dropped in Jewel ( > http://ceph.com/releases/v10-2-0-jewel-released/) > > > > But is there backward compatibility to mount it using hammer on a node? > Doesn’t seem to be and that makes

Re: [ceph-users] Mount Jewel CephFS to CentOS6

On Fri, May 13, 2016 at 8:02 PM, Gregory Farnum wrote: > > > On Friday, May 13, 2016, Andrus, Brian Contractor wrote: >> >> So I see that support for RHEL6 and derivatives was dropped in Jewel >> (http://ceph.com/releases/v10-2-0-jewel-released/) >> >> >> >> But is there backward compatibility to

Re: [ceph-users] Upgrade to Jewel... More interesting things...

Hi Tu, from what version were you upgrading from? In Jewel, all Ceph processes run with the ceph user and not the root user and may be you should investigate into the permissions of the /var/lib/ceph/osd subdirectories? So if you have upgraded from hammer it could likely be the problem. Regar

Re: [ceph-users] Upgrade to Jewel... More interesting things...

Did you check the permissions of those directories? Part of the steps in the upgrade process mentions the following: chown -R ceph:ceph /var/lib/ceph Tom Walsh https://expresshosting.net/ 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 2240

Re: [ceph-users] Upgrade to Jewel... More interesting things...

That is most likely exactly what my issue is. I must have missed that step. Thanks. Will report back. On Fri, May 13, 2016 at 11:18 AM MailingLists - EWS < mailingli...@expresswebsystems.com> wrote: > Did you check the permissions of those directories? > > > > Part of the steps in the upgrade

Re: [ceph-users] How do ceph clients determine a monitor's address (and esp. port) for initial connection?

On Fri, May 13, 2016 at 12:51 AM, Christian Sarrasin wrote: > Hi Greg, > > Thanks again and good guess! Amending testcluster.conf as follows: > > mon host = 192.168.10.201:6788 > mon addr = 192.168.10.201:6788 > > ... gets around the problem. > > having "mon host = mona:6788" also works. > > Shou

[ceph-users] Starting a cluster with one OSD node

Hello, I have a quick and probably dumb question… We would like to use Ceph for our storage, I was thinking of a cluster with 3 Monitor and OSD nodes. I was wondering if it was a bad idea to start a Ceph cluster with just one OSD node (10 OSDs, 2 SSDs), then add more nodes as our budget allows

[ceph-users] Mounting format 2 rbd images (created in Jewel) on CentOS 7 clients

Hi, I’m playing with Jewel and discovered format 1 images have been deprecated. Since the rbd kernel module in CentOS/RHEL 7 does not yet support format 2 images, how do I access RBD images created in Jewel from CentOS/RHEL 7 clients? Thanks! Steven ___

Re: [ceph-users] Mounting format 2 rbd images (created in Jewel) on CentOS 7 clients

On Fri, May 13, 2016 at 10:11 PM, Steven Hsiao-Ting Lee wrote: > Hi, > > I’m playing with Jewel and discovered format 1 images have been deprecated. > Since the rbd kernel module in CentOS/RHEL 7 does not yet support format 2 > images, how do I access RBD images created in Jewel from CentOS/RHEL

Re: [ceph-users] What's the minimal version of "ceph" client side the current "jewel" release would support?

Yang, We've got some proxmox hosts which are still running firefly and appear to be working fine with Jewel. We did have a problem where the firefly clients wouldn't communicate with the ceph cluster due to mismatched capabilities flags but this was resolved by setting "ceph osd crush tunables leg

[ceph-users] ceph-hammer - problem adding / removing monitors

Hi everyone. We’re running ceph-hammer, and I was trying to rename our monitor servers. I tried following the procedure for removing a monitor, and adding a monitor. Removing seems to have work ok, as now I have 2 monitors up. When I try to add the 3rd monitor, and the ceph-deploy completes,

[ceph-users] straw_calc_version

Hello again Cephers... As I'm learning more and breaking more things, I'm finding more things I don't know. So currently, with all of the other things since I started upgrading to Jewel, I'm seeing this in my logs. crush map has straw_calc_version=0 Now, yes, I understand the general crush map a

Re: [ceph-users] straw_calc_version

On Fri, May 13, 2016 at 5:02 PM, Tu Holmes wrote: > Hello again Cephers... As I'm learning more and breaking more things, I'm > finding more things I don't know. > > So currently, with all of the other things since I started upgrading to > Jewel, I'm seeing this in my logs. > > crush map has straw

Re: [ceph-users] straw_calc_version

Thank you for the info. Basically I should just set it to 1. On Fri, May 13, 2016 at 5:12 PM Gregory Farnum wrote: > On Fri, May 13, 2016 at 5:02 PM, Tu Holmes wrote: > > Hello again Cephers... As I'm learning more and breaking more things, I'm > > finding more things I don't know. > > > > So

Re: [ceph-users] Starting a cluster with one OSD node

On Friday, May 13, 2016, Mike Jacobacci wrote: > Hello, > > I have a quick and probably dumb question… We would like to use Ceph for > our storage, I was thinking of a cluster with 3 Monitor and OSD nodes. I > was wondering if it was a bad idea to start a Ceph cluster with just one > OSD node (1

Re: [ceph-users] Weighted Priority Queue testing

Hello again, On Fri, 13 May 2016 14:17:22 + Somnath Roy wrote: > Thanks Christian for the input. > I will start digging the code and look for possible explanation. > To be fair, after a while more PGs become involved, up to to a backfill count of 18 (that's 9 actually backfill operations a

Re: [ceph-users] Steps for Adding Cache Tier

Hello, On Fri, 13 May 2016 11:57:24 -0400 MailingLists - EWS wrote: > I have been reading a lot of information about cache-tiers, and I wanted > to know how best to go about adding the cache-tier to a production > environment. > Did you read my thread titled "Cache tier operation clarifications