Re: [ceph-users] Migrating files from ceph fs from cluster a to cluster b without low downtime

[Eneko Lacunza]

El 06/06/16 a las 20:53, Oliver Dzombic escribió:

Hi,

thank you for your suggestion.

Rsync will copy the whole file new, if the size is different.

Since we talk about raw image files of virtual servers, rsync is no option.

We need something which will inside of a file just copy the delta's.

Something like lvmsync ( which is just working with LVM ).

So i am looking for a tool which can do that on a file-base level.


Have you tried rsync --inplace? It works quite well for us, no whole 
file copying. We use ir for raw VM disc files.


--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943493611
  943324914
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

Hello,
I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
with >85% of data and few OSDs filled with ~60%-70% of data.

Any reason why the unevenly OSDs filling happned? do I need to any
tweaks on configuration to fix the above? Please advise.

PS: Ceph version is - 0.80.7

Thanks
Swami
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] no osds in jewel

[Jaemyoun Lee]

Thanks for the feedback.

I removed "ceph-deploy mon create + ceph-deploy gatherkeys."
And my system disk is sde.

As your opinion, the disk cannot be umounted when purgedata was run.
Is it bug on Ubuntu 16.04?

*$ ssh csAnt lsblk*
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda  8:00   3.7T  0 disk
├─sda1   8:10   3.6T  0 part /var/lib/ceph/osd/ceph-0
└─sda2   8:20 5G  0 part
sdb  8:16   0   3.7T  0 disk
sdc  8:32   0   3.7T  0 disk
sdd  8:48   0   3.7T  0 disk
sde  8:64   0 238.5G  0 disk
├─sde1   8:65   0   512M  0 part /boot/efi
├─sde2   8:66   0   234G  0 part /
└─sde3   8:67   0 4G  0 part [SWAP]
sdf  8:80   0   3.7T  0 disk
sdg  8:96   1   3.7T  0 disk
sdh  8:112  1   3.7T  0 disk
sdi  8:128  0   3.7T  0 disk
sdj  8:144  0   3.7T  0 disk
sdk  8:160  0   3.7T  0 disk

*$ ceph-deploy purgedata csElsa csAnt csBull csCat*
...
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 16.04 xenial
[csAnt][INFO  ] purging data on csAnt
[csAnt][INFO  ] Running command: sudo rm -rf --one-file-system --
/var/lib/ceph
[csAnt][*WARNIN*] OSDs may still be mounted, trying to unmount them
[csAnt][INFO  ] Running command: sudo find /var/lib/ceph -mindepth 1
-maxdepth 2 -type d -exec umount {} ;
[csAnt][*WARNIN*] umount: /var/lib/ceph/osd: not mounted
[csAnt][*WARNIN*] umount: /var/lib/ceph/osd/ceph-0: target is busy
[csAnt][*WARNIN*] (In some cases useful info about processes that
[csAnt][*WARNIN*]  use the device is found by lsof(8) or fuser(1).)
[csAnt][INFO  ] Running command: sudo rm -rf --one-file-system --
/var/lib/ceph
[csAnt][*WARNIN*] rm: skipping '/var/lib/ceph/osd/ceph-0', since it's on a
different device
[csAnt][*ERROR* ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][*ERROR* ] RuntimeError: Failed to execute command: rm -rf
--one-file-system -- /var/lib/ceph


*$ ssh csAnt sudo umount -f /dev/sda1*
umount: /var/lib/ceph/osd/ceph-0: target is busy
(In some cases useful info about processes that
 use the device is found by lsof(8) or fuser(1).)

Best regards,
Jae

On Tue, Jun 7, 2016 at 12:55 AM LOPEZ Jean-Charles 
wrote:

> Hi,
>
> first I’f have one remark. You run both a ceph-deploy mon create-initial
> then a "ceph-deploy mon create + ceph-deploy gatherkeys". Choose one or the
> other not both.
>
> Then, I notice that you are zapping and deploying using drive /dev/sda
> which is usually the system disks. So next question is: Is /dev/sda really
> empty and available for OSDs
>
> I guess that what happened is that because sda is not free and empty, the
> zap failed as it does not have the force option and then the old create
> failed because sda could not be prepared/formatted.
>
> Regards
> JC
>
> On Jun 5, 2016, at 10:22, Jaemyoun Lee  wrote:
>
> Hi,
>
> When I run below script to install Ceph (10.2.0), I met an error "no osds".
> Hammer was installed by the script.
> So I think I miss new thing, which was released since Hammer.
>
> Do you know what I miss?
>
> --- The script ---
> #!/bin/sh
>
> set -x
>
> ceph-deploy new csElsa
> echo "osd pool default size = 1" >> ceph.conf
> ceph-deploy install csElsa csAnt csBull csCat
> ceph-deploy mon create-initial
> ceph-deploy mon create csElsa
> ceph-deploy gatherkeys csElsa
> ceph-deploy disk zap csAnt:sda
> ceph-deploy disk zap csBull:sda
> ceph-deploy disk zap csCat:sda
> ceph-deploy osd create csAnt:sda csBull:sda csCat:sda
> ceph-deploy admin csElsa csElsa csAnt csBull csCat
> sudo chmod +r /etc/ceph/ceph.client.admin.keyring
> ceph health
> --- end ---
>
> --- The result of "ceph -w" ---
> # I blocked the IP
> jae@csElsa:~/git/ceph$ ceph -w
>
> cluster 8b2816e9-1953-4157-aaf7-95e9e668fe46
>  health HEALTH_ERR
> 64 pgs are stuck inactive for more than 300 seconds
> 64 pgs stuck inactive
> no osds
>  monmap e1: 1 mons at {csElsa=1xx.1xx.2xx.1:6789/0}
> election epoch 3, quorum 0 csElsa
>  osdmap e1: 0 osds: 0 up, 0 in
> flags sortbitwise
>   pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
> 0 kB used, 0 kB / 0 kB avail
>   64 creating
>
> 2016-06-06 01:59:08.054985 mon.0 [INF] from='client.?
> 1xx.1xx.2xx.1:0/115687' entity='client.admin' cmd='[{"prefix": "auth
> get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow
> profile bootstrap-mds"]}]': finished
> --- end ---
>
> Best regards,
> Jae
>
> --
>   Jaemyoun Lee
>
>   CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University)
>   E-mail : jaemy...@hanyang.ac.kr
>   Website : http://cpslab.hanyang.ac.kr
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
  Jaemyoun Lee

  CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University)
  E-mail : jaemy...@hanyang.ac.kr
  Website : http://cpslab.hanyang.ac.kr
__

[ceph-users] CephFS mount via internet

[João Castro]

Hello guys,
Some information:

ceph version 10.2.1 
72 OSD (24x per machine)
3 monitor
2 MDS

I have a few outside servers I need to connect CephFS to. My monitors have 2 
interfaces, one private and one public (eth0 and eth1).
I am trying to mount CephFS via eth1 on monitor01 from an outside server on the 
internet. Happens, I cannot. I get connection timed out.
Is this normal?

I have a feeling that in the past I was able to do this.. We are also using a 
Cisco ASA Firewall in between, but I am routing all the traffic to this monitor 
to a different gateway just to avoid any issues.
I would surely appreciate your help.

Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS mount via internet

[Wido den Hollander]

> Op 7 juni 2016 om 10:59 schreef João Castro :
> 
> 
> Hello guys,
> Some information:
> 
> ceph version 10.2.1 
> 72 OSD (24x per machine)
> 3 monitor
> 2 MDS
> 
> I have a few outside servers I need to connect CephFS to. My monitors have 2 
> interfaces, one private and one public (eth0 and eth1).
> I am trying to mount CephFS via eth1 on monitor01 from an outside server on 
> the 
> internet. Happens, I cannot. I get connection timed out.
> Is this normal?

Yes, the monitor will only bind to a single IP-address. Check the output of 
netstat -nap on the monitor machines.

> 
> I have a feeling that in the past I was able to do this.. We are also using a 
> Cisco ASA Firewall in between, but I am routing all the traffic to this 
> monitor 
> to a different gateway just to avoid any issues.
> I would surely appreciate your help.
> 
> Thank you.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS mount via internet

[João Castro]

Thank you, Wido!
Anyway, it is the same story, if it cannot see the OSD's I cannot mount it :( 
argh.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Sage Weil]

On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> Hello,
> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
> with >85% of data and few OSDs filled with ~60%-70% of data.
> 
> Any reason why the unevenly OSDs filling happned? do I need to any
> tweaks on configuration to fix the above? Please advise.
> 
> PS: Ceph version is - 0.80.7

Jewel and the latest hammer point release have an improved 
reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry 
run) to correct this.

sage

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

Hi Sage,
>Jewel and the latest hammer point release have an improved
>reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.

Thank youBut not planning to upgrade the cluster soon.
So, in this case - are there any tunable options will help? like
"crush tunable optimal" or so?
OR any other configuration options change will help?


Thanks
Swami


On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> Hello,
>> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
>> with >85% of data and few OSDs filled with ~60%-70% of data.
>>
>> Any reason why the unevenly OSDs filling happned? do I need to any
>> tweaks on configuration to fix the above? Please advise.
>>
>> PS: Ceph version is - 0.80.7
>
> Jewel and the latest hammer point release have an improved
> reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.
>
> sage
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Sage Weil]

On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> Hi Sage,
> >Jewel and the latest hammer point release have an improved
> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> > run) to correct this.
> 
> Thank youBut not planning to upgrade the cluster soon.
> So, in this case - are there any tunable options will help? like
> "crush tunable optimal" or so?
> OR any other configuration options change will help?

Firefly also has reweight-by-utilization... it's just a bit less friendly 
than the newer versions.  CRUSH tunables don't generally help here unless 
you have lots of OSDs that are down+out.

Note that firefly is no longer supported.

sage


> 
> 
> Thanks
> Swami
> 
> 
> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> Hello,
> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
> >> with >85% of data and few OSDs filled with ~60%-70% of data.
> >>
> >> Any reason why the unevenly OSDs filling happned? do I need to any
> >> tweaks on configuration to fix the above? Please advise.
> >>
> >> PS: Ceph version is - 0.80.7
> >
> > Jewel and the latest hammer point release have an improved
> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> > run) to correct this.
> >
> > sage
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

OK, understood...
To fix the nearfull warn, I am reducing the weight of a specific OSD,
which filled >85%..
Is this work-around advisable?

Thanks
Swami

On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> Hi Sage,
>> >Jewel and the latest hammer point release have an improved
>> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
>> > run) to correct this.
>>
>> Thank youBut not planning to upgrade the cluster soon.
>> So, in this case - are there any tunable options will help? like
>> "crush tunable optimal" or so?
>> OR any other configuration options change will help?
>
> Firefly also has reweight-by-utilization... it's just a bit less friendly
> than the newer versions.  CRUSH tunables don't generally help here unless
> you have lots of OSDs that are down+out.
>
> Note that firefly is no longer supported.
>
> sage
>
>
>>
>>
>> Thanks
>> Swami
>>
>>
>> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
>> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> >> Hello,
>> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
>> >> with >85% of data and few OSDs filled with ~60%-70% of data.
>> >>
>> >> Any reason why the unevenly OSDs filling happned? do I need to any
>> >> tweaks on configuration to fix the above? Please advise.
>> >>
>> >> PS: Ceph version is - 0.80.7
>> >
>> > Jewel and the latest hammer point release have an improved
>> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
>> > run) to correct this.
>> >
>> > sage
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Sage Weil]

On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> OK, understood...
> To fix the nearfull warn, I am reducing the weight of a specific OSD,
> which filled >85%..
> Is this work-around advisable?

Sure.  This is what reweight-by-utilization does for you, but 
automatically.

sage

> 
> Thanks
> Swami
> 
> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> Hi Sage,
> >> >Jewel and the latest hammer point release have an improved
> >> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> >> > run) to correct this.
> >>
> >> Thank youBut not planning to upgrade the cluster soon.
> >> So, in this case - are there any tunable options will help? like
> >> "crush tunable optimal" or so?
> >> OR any other configuration options change will help?
> >
> > Firefly also has reweight-by-utilization... it's just a bit less friendly
> > than the newer versions.  CRUSH tunables don't generally help here unless
> > you have lots of OSDs that are down+out.
> >
> > Note that firefly is no longer supported.
> >
> > sage
> >
> >
> >>
> >>
> >> Thanks
> >> Swami
> >>
> >>
> >> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> >> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> >> Hello,
> >> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
> >> >> with >85% of data and few OSDs filled with ~60%-70% of data.
> >> >>
> >> >> Any reason why the unevenly OSDs filling happned? do I need to any
> >> >> tweaks on configuration to fix the above? Please advise.
> >> >>
> >> >> PS: Ceph version is - 0.80.7
> >> >
> >> > Jewel and the latest hammer point release have an improved
> >> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> >> > run) to correct this.
> >> >
> >> > sage
> >> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majord...@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Corentin Bonneton]

Hello,
You how much your PG pools since he first saw you have left too big.

--
Cordialement,
Corentin BONNETON


> Le 7 juin 2016 à 15:21, Sage Weil  a écrit :
> 
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> OK, understood...
>> To fix the nearfull warn, I am reducing the weight of a specific OSD,
>> which filled >85%..
>> Is this work-around advisable?
> 
> Sure.  This is what reweight-by-utilization does for you, but 
> automatically.
> 
> sage
> 
>> 
>> Thanks
>> Swami
>> 
>> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
>>> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
 Hi Sage,
> Jewel and the latest hammer point release have an improved
> reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.
 
 Thank youBut not planning to upgrade the cluster soon.
 So, in this case - are there any tunable options will help? like
 "crush tunable optimal" or so?
 OR any other configuration options change will help?
>>> 
>>> Firefly also has reweight-by-utilization... it's just a bit less friendly
>>> than the newer versions.  CRUSH tunables don't generally help here unless
>>> you have lots of OSDs that are down+out.
>>> 
>>> Note that firefly is no longer supported.
>>> 
>>> sage
>>> 
>>> 
 
 
 Thanks
 Swami
 
 
 On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> Hello,
>> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
>> with >85% of data and few OSDs filled with ~60%-70% of data.
>> 
>> Any reason why the unevenly OSDs filling happned? do I need to any
>> tweaks on configuration to fix the above? Please advise.
>> 
>> PS: Ceph version is - 0.80.7
> 
> Jewel and the latest hammer point release have an improved
> reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.
> 
> sage
> 
 --
 To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 
 
>> 
>> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Markus Blank-Burian]

Hello Sage,

are there any development plans to improve PG distribution to OSDs? 

We use CephFS on Infernalis and objects are distributed very well across the
PGs. But the automatic PG distribution creates large fluctuations, even in the
simplest case for same-sized OSDs and a flat hierarchy using 3x replication.
Testing with crushtool, we would need an unreasonable high number of PG/OSD to
have a nearly-flat distribution.
Reweighting the PGs with "ceph osd reweight" helped for our cache-pool since it
has a flat hierarchy (one OSD/host). The EC pool on the other hand has a
hierarchy with around 2-4 OSDs (1-4TB / OSD) per host and redundancy is
distributed across hosts (6-8TB / host). I assume, there would be no way around
tuning the host weights in the crush map in this case?

Markus

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage
Weil
Sent: Dienstag, 7. Juni 2016 15:22
To: M Ranga Swami Reddy 
Cc: ceph-devel ; ceph-users

Subject: Re: [ceph-users] un-even data filled on OSDs

On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> OK, understood...
> To fix the nearfull warn, I am reducing the weight of a specific OSD, 
> which filled >85%..
> Is this work-around advisable?

Sure.  This is what reweight-by-utilization does for you, but automatically.

sage

> 
> Thanks
> Swami
> 
> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> Hi Sage,
> >> >Jewel and the latest hammer point release have an improved 
> >> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... 
> >> >to dry
> >> > run) to correct this.
> >>
> >> Thank youBut not planning to upgrade the cluster soon.
> >> So, in this case - are there any tunable options will help? like 
> >> "crush tunable optimal" or so?
> >> OR any other configuration options change will help?
> >
> > Firefly also has reweight-by-utilization... it's just a bit less 
> > friendly than the newer versions.  CRUSH tunables don't generally 
> > help here unless you have lots of OSDs that are down+out.
> >
> > Note that firefly is no longer supported.
> >
> > sage
> >
> >
> >>
> >>
> >> Thanks
> >> Swami
> >>
> >>
> >> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> >> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> >> Hello,
> >> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs 
> >> >> filled with >85% of data and few OSDs filled with ~60%-70% of data.
> >> >>
> >> >> Any reason why the unevenly OSDs filling happned? do I need to 
> >> >> any tweaks on configuration to fix the above? Please advise.
> >> >>
> >> >> PS: Ceph version is - 0.80.7
> >> >
> >> > Jewel and the latest hammer point release have an improved 
> >> > reweight-by-utilization (ceph osd test-reweight-by-utilization 
> >> > ... to dry
> >> > run) to correct this.
> >> >
> >> > sage
> >> >
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe 
> >> ceph-devel" in the body of a message to majord...@vger.kernel.org 
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

In my cluster:
 351 OSDs with same size and 8192 pgs per pool. And 60% RAW space used.

Thanks
Swami


On Tue, Jun 7, 2016 at 7:22 PM, Corentin Bonneton  wrote:
> Hello,
> You how much your PG pools since he first saw you have left too big.
>
> --
> Cordialement,
> Corentin BONNETON
>
>
> Le 7 juin 2016 à 15:21, Sage Weil  a écrit :
>
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>
> OK, understood...
> To fix the nearfull warn, I am reducing the weight of a specific OSD,
> which filled >85%..
> Is this work-around advisable?
>
>
> Sure.  This is what reweight-by-utilization does for you, but
> automatically.
>
> sage
>
>
> Thanks
> Swami
>
> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
>
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>
> Hi Sage,
>
> Jewel and the latest hammer point release have an improved
> reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.
>
>
> Thank youBut not planning to upgrade the cluster soon.
> So, in this case - are there any tunable options will help? like
> "crush tunable optimal" or so?
> OR any other configuration options change will help?
>
>
> Firefly also has reweight-by-utilization... it's just a bit less friendly
> than the newer versions.  CRUSH tunables don't generally help here unless
> you have lots of OSDs that are down+out.
>
> Note that firefly is no longer supported.
>
> sage
>
>
>
>
> Thanks
> Swami
>
>
> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
>
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>
> Hello,
> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
> with >85% of data and few OSDs filled with ~60%-70% of data.
>
> Any reason why the unevenly OSDs filling happned? do I need to any
> tweaks on configuration to fix the above? Please advise.
>
> PS: Ceph version is - 0.80.7
>
>
> Jewel and the latest hammer point release have an improved
> reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> run) to correct this.
>
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New user questions with radosgw with Jewel 10.2.1

[Sylvain, Eric]

Hi JC,

Thanks for the reply.

Yes, my system is set to run as “ceph”:
   
/etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.p6-os1-mon7.service
   ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i 
-conf  --setuser ceph --setgroup ceph

Yet changing these to “root” has no effect.
   ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i -conf  
--setuser root --setgroup root

Also doing: chown root.root /usr/bin/radosgw; chmod 4755 /usr/bin/radosgw
Had no effect.

I feel the issue is in reading /etc/ceph/ceph.conf, because even if I change
   rgw frontends = “bogus bogus bogus”
Expecting some failure, it still started up fine (on port 7480).
The config still says:
# ceph --admin-daemon /var/run/ceph/ceph-client.rgw.p6-os1-mon7.asok config 
show
…
"rgw_frontends": "fastcgi, civetweb port=7480",
…

Again suspecting keyring I created “client.radosgw.gateway” and changed 
ceph.conf for it, to see if that would help, no luck…

Could this be tied to having admin, mon and radosgw on same host?

Does keyring restrict what parts of ceph.conf are available?

Thanks in advance for anything you can provide

-Eric




From: LOPEZ Jean-Charles [mailto:jelo...@redhat.com]
Sent: Tuesday, May 24, 2016 1:52 PM
To: Sylvain, Eric 
Cc: ceph-us...@ceph.com
Subject: Re: [ceph-users] New user questions with radosgw with Jewel 10.2.1

Hi Sylvain,

this is probably related to the fact that the systemd unit file for the RGW is 
configured to run as user ceph. As ceph is not a privileged user, it can not 
bind to lower port numbers.

Modify the ceph-radosgw unit file and make sure the set user is set for root.

To verify this is the root cause, manually start the ceph-radosgw process from 
the command line being connected as root. If it works and you can query your 
RGW on port 80, then it is this problem.

Cheers
JC

On May 23, 2016, at 10:03, Sylvain, Eric 
mailto:eric.sylv...@arris.com>> wrote:


Hello,

I’m a fairly new user and I am trying to bring up radosgw.

I am following this page: 
http://docs.ceph.com/docs/master/install/install-ceph-gateway/

I have Jewel 10.2.1 installed with a co-located admin/mon host and a separate 
osd host

First a question: Can I run radosgw on a co-located admin/monitor host?

And then my issue: I cannot seem to get the frontent (civetweb) to run on port 
80.
(I do the following)
su – ceph
cd my-cluster
sudo ceph-deploy install –rgw p6-os1-mon2
ceph-deploy --overwrite-conf rgw create p6-os1-mon
   (Note it says: The Ceph Object Gateway (RGW) is now running on host 
p6-os1-mon2 and default port 7480)
   (Testing to http://p6-os1-mon2:7480 show the 
correct info)
Edit ~ceph/my-cluster/ceph.conf and add:
   [client.radosgw.p6-os1-mon2]
   rgw_frontends = "civetweb port=80”
Reboot p6-os1/mon2
After everything is back up, try http://p6-os1-mon2:80
   No luck, it appears it is still running at 7480…
What am I missing that would make radosgw get the correct port?
(Running by hand, I see “listening_ports”
2016-05-23 12:13:43.479136 7fc0793f7a40 20 civetweb config: decode_url: no
2016-05-23 12:13:43.479138 7fc0793f7a40 20 civetweb config: enable_keep_alive: 
yes
2016-05-23 12:13:43.479138 7fc0793f7a40 20 civetweb config: listening_ports: 
7480
2016-05-23 12:13:43.479139 7fc0793f7a40 20 civetweb config: num_threads: 100
2016-05-23 12:13:43.479140 7fc0793f7a40 20 civetweb config: run_as_user: ceph

I suspected maybe the keyring? So I did:
  $ sudo rm -rf /etc/ceph/ceph.client.radosgw.keyring
  $ sudo ceph-authtool --create-keyring 
/etc/ceph/ceph.client.radosgw.keyring
  $ sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
  $ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.p6-os1-mon2
  $ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
client.radosgw.p6-os1-mon2 --gen-key
  $ sudo ceph-authtool -n client.radosgw.p6-os1-mon2 --cap osd 'allow rwx' 
--cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
  $ sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
client.radosgw.p6-os1-mon2 -i /etc/ceph/ceph.client.radosgw.keyring
  (Also copied to to my osd node…)
  Reboot everything, still no luck civetweb remains on 7480

Ideas? Comments?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New user questions with radosgw with Jewel 10.2.1

[Karol Mroz]

Hi Eric,

Please see inline...

On Tue, Jun 07, 2016 at 05:14:25PM +, Sylvain, Eric wrote:
> 
> Yes, my system is set to run as “ceph”:
>
> /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.p6-os1-mon7.service
>ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i 
> -conf  --setuser ceph --setgroup ceph

The first thing to check is the heading line of the RGW section in your 
ceph.conf file.

Systemd passes --name as "client.%i" where "%i" expands to (in your case): 
rgw.p6-os1-mon7.
Thus, your ceph.conf RGW heading should be: [client.rgw.p6-os1-mon7]
If these entires do not match, RGW will not parse the necessary configuration
section and will simply use defaults (ie. port 7480, etc).

> 
> Yet changing these to “root” has no effect.
>ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i -conf  
> --setuser root --setgroup root

Using user root is not needed for binding to privileged ports. When configured
to use civetweb, RGW delegates the permission drop to civetweb. Civetweb
does this _after_ binding to a port, reading certificates, etc. So, using
the default "--user ceph" and "--group ceph" is best.

> 
> Also doing: chown root.root /usr/bin/radosgw; chmod 4755 /usr/bin/radosgw
> Had no effect.

This is not needed.

> 
> I feel the issue is in reading /etc/ceph/ceph.conf, because even if I change
>rgw frontends = “bogus bogus bogus”
> Expecting some failure, it still started up fine (on port 7480).
> The config still says:
> # ceph --admin-daemon /var/run/ceph/ceph-client.rgw.p6-os1-mon7.asok 
> config show
> …
> "rgw_frontends": "fastcgi, civetweb port=7480",
> …

Which points to RGW not parsing it's config section.

> 
> Again suspecting keyring I created “client.radosgw.gateway” and changed 
> ceph.conf for it, to see if that would help, no luck…
> 
> Could this be tied to having admin, mon and radosgw on same host?

RGW should reside happily alongside any of the ceph daemons.

> 
> Does keyring restrict what parts of ceph.conf are available?
> 
> Thanks in advance for anything you can provide
> 

Hope this helps.

-- 
Regards,
Karol


signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Disk failures

[Gandalf Corvotempesta]

Hi,
How ceph detect and manage disk failures?  What happens if some data are
wrote on a bad sector?

Are there any change to get the bad sector "distributed" across the cluster
due to the replication?

Is ceph able to remove the OSD bound to the failed disk automatically?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD rollback error mesage

[Brendan Moloney]

Hi,

I am trying out a Ceph 10.2.1 cluster and noticed this message almost every 
time I do a snap rollback:

12:18:08.203349 7f11ee46c700 -1 librbd::object_map::LockRequest: failed to lock 
object map: (17) File exists

The rollback still seems to work fine. Nothing else should be accessing the RBD 
image at the same time (the
VM is shutdown).

I have the following image features enabled:

  - layering
  - exclusive-lock
  - object-map
  - fast-diff
  - deep-flatten

Is this something I need to worry about?  Any info is appreciated!

Thanks,
Brendan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Filestore update script?

[WRIGHT, JON R (JON R)]

I'm trying to recover an OSD after running xfs_repair on the disk. It 
seems to be ok now.  There is a log message that includes the following: 
"Please run the FileStore update script before starting the OSD, or set 
filestore_update_to to 4"


What is the FileStore update script?  Google search doesn't produce 
useful information on what or where it is.   Also the 
filestore_update_to option is set in what config file?


Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Blair Bethwaite]

Swami,

Try 
https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py,
that'll work with Firefly and allow you to only tune down weight of a
specific number of overfull OSDs.

Cheers,

On 7 June 2016 at 23:11, M Ranga Swami Reddy  wrote:
> OK, understood...
> To fix the nearfull warn, I am reducing the weight of a specific OSD,
> which filled >85%..
> Is this work-around advisable?
>
> Thanks
> Swami
>
> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
>> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>>> Hi Sage,
>>> >Jewel and the latest hammer point release have an improved
>>> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
>>> > run) to correct this.
>>>
>>> Thank youBut not planning to upgrade the cluster soon.
>>> So, in this case - are there any tunable options will help? like
>>> "crush tunable optimal" or so?
>>> OR any other configuration options change will help?
>>
>> Firefly also has reweight-by-utilization... it's just a bit less friendly
>> than the newer versions.  CRUSH tunables don't generally help here unless
>> you have lots of OSDs that are down+out.
>>
>> Note that firefly is no longer supported.
>>
>> sage
>>
>>
>>>
>>>
>>> Thanks
>>> Swami
>>>
>>>
>>> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
>>> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>>> >> Hello,
>>> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
>>> >> with >85% of data and few OSDs filled with ~60%-70% of data.
>>> >>
>>> >> Any reason why the unevenly OSDs filling happned? do I need to any
>>> >> tweaks on configuration to fix the above? Please advise.
>>> >>
>>> >> PS: Ceph version is - 0.80.7
>>> >
>>> > Jewel and the latest hammer point release have an improved
>>> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
>>> > run) to correct this.
>>> >
>>> > sage
>>> >
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majord...@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD rollback error mesage

[Jason Dillaman]

Can you run "rbd info" against that image?  I suspect it is a harmless
but alarming error message.  I actually just opened a tracker ticket
this morning for a similar issue for rbd-mirror [1] when it bootstraps
an image to a peer cluster.  In that case, it was a harmless error
message that we will correct.

[1] http://tracker.ceph.com/issues/16179

On Tue, Jun 7, 2016 at 4:30 PM, Brendan Moloney  wrote:
> Hi,
>
> I am trying out a Ceph 10.2.1 cluster and noticed this message almost every
> time I do a snap rollback:
>
> 12:18:08.203349 7f11ee46c700 -1 librbd::object_map::LockRequest: failed to
> lock object map: (17) File exists
>
> The rollback still seems to work fine. Nothing else should be accessing the
> RBD image at the same time (the
> VM is shutdown).
>
> I have the following image features enabled:
>
>   - layering
>   - exclusive-lock
>   - object-map
>   - fast-diff
>   - deep-flatten
>
> Is this something I need to worry about?  Any info is appreciated!
>
> Thanks,
> Brendan
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD rollback error mesage

[Brendan Moloney]

Here you go:

rbd image 'aircds':
size 8192 MB in 2048 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.39202eb141f2
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags: 
parent: rbd/xenial-base@gold-copy
overlap: 8192 MB


Brendan


From: Jason Dillaman [jdill...@redhat.com]
Sent: Tuesday, June 07, 2016 6:56 PM
To: Brendan Moloney
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] RBD rollback error mesage

Can you run "rbd info" against that image?  I suspect it is a harmless
but alarming error message.  I actually just opened a tracker ticket
this morning for a similar issue for rbd-mirror [1] when it bootstraps
an image to a peer cluster.  In that case, it was a harmless error
message that we will correct.

[1] http://tracker.ceph.com/issues/16179

On Tue, Jun 7, 2016 at 4:30 PM, Brendan Moloney  wrote:
> Hi,
>
> I am trying out a Ceph 10.2.1 cluster and noticed this message almost every
> time I do a snap rollback:
>
> 12:18:08.203349 7f11ee46c700 -1 librbd::object_map::LockRequest: failed to
> lock object map: (17) File exists
>
> The rollback still seems to work fine. Nothing else should be accessing the
> RBD image at the same time (the
> VM is shutdown).
>
> I have the following image features enabled:
>
>   - layering
>   - exclusive-lock
>   - object-map
>   - fast-diff
>   - deep-flatten
>
> Is this something I need to worry about?  Any info is appreciated!
>
> Thanks,
> Brendan
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



--
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD rollback error mesage

[Jason Dillaman]

OK -- looks like it's an innocent error message.  I just wanted to
ensure that "flags" didn't include "object map invalid" since that
would indicate a real issue.

I'll update that ticket to include the other use-case where it appears.

On Tue, Jun 7, 2016 at 10:03 PM, Brendan Moloney  wrote:
> Here you go:
>
> rbd image 'aircds':
> size 8192 MB in 2048 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.39202eb141f2
> format: 2
> features: layering, exclusive-lock, object-map, fast-diff, 
> deep-flatten
> flags:
> parent: rbd/xenial-base@gold-copy
> overlap: 8192 MB
>
>
> Brendan
>
> 
> From: Jason Dillaman [jdill...@redhat.com]
> Sent: Tuesday, June 07, 2016 6:56 PM
> To: Brendan Moloney
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RBD rollback error mesage
>
> Can you run "rbd info" against that image?  I suspect it is a harmless
> but alarming error message.  I actually just opened a tracker ticket
> this morning for a similar issue for rbd-mirror [1] when it bootstraps
> an image to a peer cluster.  In that case, it was a harmless error
> message that we will correct.
>
> [1] http://tracker.ceph.com/issues/16179
>
> On Tue, Jun 7, 2016 at 4:30 PM, Brendan Moloney  wrote:
>> Hi,
>>
>> I am trying out a Ceph 10.2.1 cluster and noticed this message almost every
>> time I do a snap rollback:
>>
>> 12:18:08.203349 7f11ee46c700 -1 librbd::object_map::LockRequest: failed to
>> lock object map: (17) File exists
>>
>> The rollback still seems to work fine. Nothing else should be accessing the
>> RBD image at the same time (the
>> VM is shutdown).
>>
>> I have the following image features enabled:
>>
>>   - layering
>>   - exclusive-lock
>>   - object-map
>>   - fast-diff
>>   - deep-flatten
>>
>> Is this something I need to worry about?  Any info is appreciated!
>>
>> Thanks,
>> Brendan
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
> Jason



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Must host bucket name be the same with hostname ?

[????]

Hi all,

There are SASes & SSDs in my nodes at the same time.
Now i want divide them into 2 groups, one composed of SASes and one only 
contained SSDs.
When i configure CRUSH rulesets, segment below:


# buckets
host robert-a {
id -2   # do not change unnecessarily
# weight 1.640
alg straw
hash 0  # rjenkins1
item osd.0 weight 0.250#SAS
item osd.1 weight 0.250#SAS
item osd.2 weight 0.250#SSD
item osd.3 weight 0.250#SSD

}


So, i am not sure must host bucket name be the same with hostname.


Or host bucket name does no matter?



Best regards,

Xiucai___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Must host bucket name be the same with hostname ?

[Christian Balzer]

Hello,

you will want to read:
https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/

especially section III and IV.

Another approach w/o editing the CRUSH map is here:
https://elkano.org/blog/ceph-sata-ssd-pools-server-editing-crushmap/

Christian

On Wed, 8 Jun 2016 10:54:36 +0800 秀才 wrote:

> Hi all,
> 
> There are SASes & SSDs in my nodes at the same time.
> Now i want divide them into 2 groups, one composed of SASes and one
> only contained SSDs. When i configure CRUSH rulesets, segment below:
> 
> 
> # buckets
> host robert-a {
>   id -2   # do not change unnecessarily
>   # weight 1.640
>   alg straw
>   hash 0  # rjenkins1
>   item osd.0 weight 0.250#SAS
>   item osd.1 weight 0.250#SAS
>   item osd.2 weight 0.250#SSD
>   item osd.3 weight 0.250#SSD
> 
> }
> 
> 
> So, i am not sure must host bucket name be the same with hostname.
> 
> 
> Or host bucket name does no matter?
> 
> 
> 
> Best regards,
> 
> Xiucai

-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] monitor clock skew warning when date/time is the same

[pixelfairy]

test cluster running on vmware fusion. all 3 nodes are both monitor and
osd, and are running opentpd

$ ansible ceph1 -a "ceph -s"
ceph1 | SUCCESS | rc=0 >>
cluster d7d2a02c-915f-4725-8d8d-8d42fcd87242
 health HEALTH_WARN
clock skew detected on mon.ceph2, mon.ceph3
Monitor clock skew detected
 monmap e1: 3 mons at {ceph1=
192.168.113.31:6789/0,ceph2=192.168.113.32:6789/0,ceph3=192.168.113.33:6789/0
}
election epoch 4, quorum 0,1,2 ceph1,ceph2,ceph3
 osdmap e7: 3 osds: 3 up, 3 in
flags sortbitwise
  pgmap v16: 64 pgs, 1 pools, 0 bytes data, 0 objects
102328 kB used, 289 GB / 289 GB avail
  64 active+clean

$ ansible ceph -a date
ceph1 | SUCCESS | rc=0 >>
Tue Jun  7 21:36:37 PDT 2016

ceph2 | SUCCESS | rc=0 >>
Tue Jun  7 21:36:37 PDT 2016

ceph3 | SUCCESS | rc=0 >>
Tue Jun  7 21:36:37 PDT 2016
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

Blair - Thanks for the script...Btw, is this script has option for dry run?

Thanks
Swami

On Wed, Jun 8, 2016 at 6:35 AM, Blair Bethwaite
 wrote:
> Swami,
>
> Try 
> https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py,
> that'll work with Firefly and allow you to only tune down weight of a
> specific number of overfull OSDs.
>
> Cheers,
>
> On 7 June 2016 at 23:11, M Ranga Swami Reddy  wrote:
>> OK, understood...
>> To fix the nearfull warn, I am reducing the weight of a specific OSD,
>> which filled >85%..
>> Is this work-around advisable?
>>
>> Thanks
>> Swami
>>
>> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
>>> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
 Hi Sage,
 >Jewel and the latest hammer point release have an improved
 >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
 > run) to correct this.

 Thank youBut not planning to upgrade the cluster soon.
 So, in this case - are there any tunable options will help? like
 "crush tunable optimal" or so?
 OR any other configuration options change will help?
>>>
>>> Firefly also has reweight-by-utilization... it's just a bit less friendly
>>> than the newer versions.  CRUSH tunables don't generally help here unless
>>> you have lots of OSDs that are down+out.
>>>
>>> Note that firefly is no longer supported.
>>>
>>> sage
>>>
>>>


 Thanks
 Swami


 On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
 > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
 >> Hello,
 >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
 >> with >85% of data and few OSDs filled with ~60%-70% of data.
 >>
 >> Any reason why the unevenly OSDs filling happned? do I need to any
 >> tweaks on configuration to fix the above? Please advise.
 >>
 >> PS: Ceph version is - 0.80.7
 >
 > Jewel and the latest hammer point release have an improved
 > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
 > run) to correct this.
 >
 > sage
 >
 --
 To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html


>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Cheers,
> ~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[Blair Bethwaite]

It runs by default in dry-run mode, which IMHO opinion should be the
default for operations like this. IIRC you add "-d -r" to make it
actually apply the re-weighting.

Cheers,

On 8 June 2016 at 15:04, M Ranga Swami Reddy  wrote:
> Blair - Thanks for the script...Btw, is this script has option for dry run?
>
> Thanks
> Swami
>
> On Wed, Jun 8, 2016 at 6:35 AM, Blair Bethwaite
>  wrote:
>> Swami,
>>
>> Try 
>> https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py,
>> that'll work with Firefly and allow you to only tune down weight of a
>> specific number of overfull OSDs.
>>
>> Cheers,
>>
>> On 7 June 2016 at 23:11, M Ranga Swami Reddy  wrote:
>>> OK, understood...
>>> To fix the nearfull warn, I am reducing the weight of a specific OSD,
>>> which filled >85%..
>>> Is this work-around advisable?
>>>
>>> Thanks
>>> Swami
>>>
>>> On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
 On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> Hi Sage,
> >Jewel and the latest hammer point release have an improved
> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to dry
> > run) to correct this.
>
> Thank youBut not planning to upgrade the cluster soon.
> So, in this case - are there any tunable options will help? like
> "crush tunable optimal" or so?
> OR any other configuration options change will help?

 Firefly also has reweight-by-utilization... it's just a bit less friendly
 than the newer versions.  CRUSH tunables don't generally help here unless
 you have lots of OSDs that are down+out.

 Note that firefly is no longer supported.

 sage


>
>
> Thanks
> Swami
>
>
> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
> >> Hello,
> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
> >> with >85% of data and few OSDs filled with ~60%-70% of data.
> >>
> >> Any reason why the unevenly OSDs filling happned? do I need to any
> >> tweaks on configuration to fix the above? Please advise.
> >>
> >> PS: Ceph version is - 0.80.7
> >
> > Jewel and the latest hammer point release have an improved
> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to 
> > dry
> > run) to correct this.
> >
> > sage
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majord...@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Cheers,
>> ~Blairo



-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] un-even data filled on OSDs

[M Ranga Swami Reddy]

Thats great.. Will try this..

Thanks
Swami

On Wed, Jun 8, 2016 at 10:38 AM, Blair Bethwaite
 wrote:
> It runs by default in dry-run mode, which IMHO opinion should be the
> default for operations like this. IIRC you add "-d -r" to make it
> actually apply the re-weighting.
>
> Cheers,
>
> On 8 June 2016 at 15:04, M Ranga Swami Reddy  wrote:
>> Blair - Thanks for the script...Btw, is this script has option for dry run?
>>
>> Thanks
>> Swami
>>
>> On Wed, Jun 8, 2016 at 6:35 AM, Blair Bethwaite
>>  wrote:
>>> Swami,
>>>
>>> Try 
>>> https://github.com/cernceph/ceph-scripts/blob/master/tools/crush-reweight-by-utilization.py,
>>> that'll work with Firefly and allow you to only tune down weight of a
>>> specific number of overfull OSDs.
>>>
>>> Cheers,
>>>
>>> On 7 June 2016 at 23:11, M Ranga Swami Reddy  wrote:
 OK, understood...
 To fix the nearfull warn, I am reducing the weight of a specific OSD,
 which filled >85%..
 Is this work-around advisable?

 Thanks
 Swami

 On Tue, Jun 7, 2016 at 6:37 PM, Sage Weil  wrote:
> On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> Hi Sage,
>> >Jewel and the latest hammer point release have an improved
>> >reweight-by-utilization (ceph osd test-reweight-by-utilization ... to 
>> >dry
>> > run) to correct this.
>>
>> Thank youBut not planning to upgrade the cluster soon.
>> So, in this case - are there any tunable options will help? like
>> "crush tunable optimal" or so?
>> OR any other configuration options change will help?
>
> Firefly also has reweight-by-utilization... it's just a bit less friendly
> than the newer versions.  CRUSH tunables don't generally help here unless
> you have lots of OSDs that are down+out.
>
> Note that firefly is no longer supported.
>
> sage
>
>
>>
>>
>> Thanks
>> Swami
>>
>>
>> On Tue, Jun 7, 2016 at 6:00 PM, Sage Weil  wrote:
>> > On Tue, 7 Jun 2016, M Ranga Swami Reddy wrote:
>> >> Hello,
>> >> I have aorund 100 OSDs in my ceph cluster. In this a few OSDs filled
>> >> with >85% of data and few OSDs filled with ~60%-70% of data.
>> >>
>> >> Any reason why the unevenly OSDs filling happned? do I need to any
>> >> tweaks on configuration to fix the above? Please advise.
>> >>
>> >> PS: Ceph version is - 0.80.7
>> >
>> > Jewel and the latest hammer point release have an improved
>> > reweight-by-utilization (ceph osd test-reweight-by-utilization ... to 
>> > dry
>> > run) to correct this.
>> >
>> > sage
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
 --
 To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> ~Blairo
>
>
>
> --
> Cheers,
> ~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com