Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Miguel Afonso Oliveira]

If your issue is caused by the bug I presume, you need to use the
newest client (0.72 ceph-fuse or 3.12 kernel)

Regards
Yan, Zheng


Hi,

We are running 0.72-1 throughout the cluster but on kernel 
2.6.32-358.6.2.el6.x86_64... This is a big deployment (~1 cores), 
not so easy to update kernel has it has too many implications...


Is there any way I can help iron out this bug?

Cheers,

MAO





Cheers,

MAO





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw Segmentation fault on obj copy

[Dominik Mostowiec]

Thanks.

--
Regards
Dominik

2013/12/3 Yehuda Sadeh :
> For bobtail at this point yes. You can try the unofficial version with
> that fix off the gitbuilder. Another option is to upgrade everything
> to dumpling.
>
> Yehuda
>
> On Mon, Dec 2, 2013 at 10:24 PM, Dominik Mostowiec
>  wrote:
>> Thanks
>> Workaround, don't use multipart when obj size == 0 ?
>>
>> On Dec 3, 2013 6:43 AM, "Yehuda Sadeh"  wrote:
>>>
>>> I created earlier an issue (6919) and updated it with the relevant
>>> issue. This has been fixed in dumpling, although I don't remember
>>> hitting the scenario that you did. Was probably hitting it as part of
>>> the development work that was done then.
>>> In any case I created a branch with the relevant fixes in it (wip-6919).
>>>
>>> Thanks,
>>> Yehuda
>>>
>>> On Mon, Dec 2, 2013 at 8:39 PM, Dominik Mostowiec
>>>  wrote:
>>> > for another object.
>>> > http://pastebin.com/VkVAYgwn
>>> >
>>> >
>>> > 2013/12/3 Yehuda Sadeh :
>>> >> I see. Do you have backtrace for the crash?
>>> >>
>>> >> On Mon, Dec 2, 2013 at 6:19 PM, Dominik Mostowiec
>>> >>  wrote:
>>> >>> 0.56.7
>>> >>>
>>> >>> W dniu poniedziałek, 2 grudnia 2013 użytkownik Yehuda Sadeh napisał:
>>> >>>
>>>  I'm having trouble reproducing the issue. What version are you using?
>>> 
>>>  Thanks,
>>>  Yehuda
>>> 
>>>  On Mon, Dec 2, 2013 at 2:16 PM, Yehuda Sadeh 
>>>  wrote:
>>>  > Actually, I read that differently. It only says that if there's
>>>  > more
>>>  > than 1 part, all parts except for the last one need to be > 5M.
>>>  > Which
>>>  > means that for uploads that are smaller than 5M there should be
>>>  > zero
>>>  > or one parts.
>>>  >
>>>  > On Mon, Dec 2, 2013 at 12:54 PM, Dominik Mostowiec
>>>  >  wrote:
>>>  >> You're right.
>>>  >>
>>>  >> S3 api doc:
>>>  >>
>>>  >> http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
>>>  >> "Err:EntityTooSmall
>>>  >> Your proposed upload is smaller than the minimum allowed object
>>>  >> size.
>>>  >> Each part must be at least 5 MB in size, except the last part."
>>>  >>
>>>  >> Thanks.
>>>  >>
>>>  >> This error should be triggered from radosgw also.
>>>  >>
>>>  >> --
>>>  >> Regards
>>>  >> Dominik
>>>  >>
>>>  >> 2013/12/2 Yehuda Sadeh :
>>>  >>> Looks like it. There should be a guard against it (mulitpart
>>>  >>> upload
>>>  >>> minimum is 5M).
>>>  >>>
>>>  >>> On Mon, Dec 2, 2013 at 12:32 PM, Dominik Mostowiec
>>>  >>>  wrote:
>>>   Yes, this is probably upload empty file.
>>>   This is the problem?
>>>  
>>>   --
>>>   Regards
>>>   Dominik
>>>  
>>>  
>>>   2013/12/2 Yehuda Sadeh :
>>>  > By any chance are you uploading empty objects through the
>>>  > multipart
>>>  > upload api?
>>>  >
>>>  > On Mon, Dec 2, 2013 at 12:08 PM, Dominik Mostowiec
>>>  >  wrote:
>>>  >> Hi,
>>>  >> Another file with the same problems:
>>>  >>
>>>  >> 2013-12-01 11:37:15.556687 7f7891fd3700  1 == starting new
>>>  >> request
>>>  >> req=0x25406d0 =
>>>  >> 2013-12-01 11:37:15.556739 7f7891fd3700  2 req
>>>  >> 1314:0.52initializing
>>>  >> 2013-12-01 11:37:15.556789 7f7891fd3700 10
>>>  >> s->object=files/192.txt
>>>  >> s->bucket=testbucket
>>>  >> 2013-12-01 11:37:15.556799 7f7891fd3700  2 req
>>>  >> 1314:0.000112:s3:POST
>>>  >> /testbucket/files/192.txt::getting op
>>>  >> 2013-12-01 11:37:15.556804 7f7891fd3700  2 req
>>>  >> 1314:0.000118:s3:POST
>>>  >> /testbucket/files/192.txt:complete_multipart:authorizing
>>>  >> 2013-12-01 11:37:15.560013 7f7891fd3700 10
>>>  >> get_canon_resource():
>>>  >>
>>>  >>
>>>  >> dest=/testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>>  >> 2013-12-01 11:37:15.560027 7f7891fd3700 10 auth_hdr:
>>>  >> POST
>>>  >>
>>>  >> application/xml
>>>  >> Sun, 01 Dec 2013 10:37:10 GMT
>>>  >>
>>>  >> /testbucket/files/192.txt?uploadId=i92xi2olzDtFAeLXlfU2PFP9CDU87BC
>>>  >> 2013-12-01 11:37:15.560085 7f7891fd3700  2 req
>>>  >> 1314:0.003399:s3:POST
>>>  >> /testbucket/files/192.txt:complete_multipart:reading
>>>  >> permissions
>>>  >> 2013-12-01 11:37:15.562356 7f7891fd3700  2 req
>>>  >> 1314:0.005670:s3:POST
>>>  >> /testbucket/files/192.txt:complete_multipart:verifying op
>>>  >> permissions
>>>  >> 2013-12-01 11:37:15.562373 7f7891fd3700  5 Searching
>>>  >> permissions
>>>  >> for
>>>  >> uid=0 mask=2
>>>  >> 2013-12-01 11:37:15.562377 7f7891fd3700  5 Found permission:
>>>  >> 15
>>>  

[ceph-users] Can't stop osd (state D)

[Emmanuel Lacour]

Dear ceph users,


I have a ceph cluster running 0.67.4. Two osd are down in "ceph -s".
They are stil there in ps and I can't stop them (service ceph stop osd.x
or kill or even kill -9)!

Any idea?


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't stop osd (state D)

[James Harper]

> Dear ceph users,
> 
> I have a ceph cluster running 0.67.4. Two osd are down in "ceph -s".
> They are stil there in ps and I can't stop them (service ceph stop osd.x
> or kill or even kill -9)!
> 
> Any idea?
> 

Do you know why the OSDs went down? If their state is D in 'ps -ax' then check 
dmesg to see if anything is logged there, in particular filesystem or disk 
errors.

What filesystem are you using? If btrfs, what kernel?

James
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't stop osd (state D)

[Loic Dachary]

Hi Emmanuel,

Funny that you are 20 meters away :-) From the dmesg output you just showed me 
it looks like you've been hit by

http://tracker.ceph.com/issues/6301 ceph-osd hung by XFS using linux 3.10

Cheers

On 03/12/2013 10:12, Emmanuel Lacour wrote:
> 
> Dear ceph users,
> 
> 
> I have a ceph cluster running 0.67.4. Two osd are down in "ceph -s".
> They are stil there in ps and I can't stop them (service ceph stop osd.x
> or kill or even kill -9)!
> 
> Any idea?
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't stop osd (state D)

[Emmanuel Lacour]

On Tue, Dec 03, 2013 at 10:12:03AM +0100, Emmanuel Lacour wrote:
> 
> Dear ceph users,
> 
> 
> I have a ceph cluster running 0.67.4. Two osd are down in "ceph -s".
> They are stil there in ps and I can't stop them (service ceph stop osd.x
> or kill or even kill -9)!
> 


this seems related to some memory corruption on XFS side.

I see a lot of those lines here:

kernel: [4583064.070081] XFS: possible memory allocation deadlock in kmem_alloc 
(mode:0x250)

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Indra Pramana]

Hi Brian and Robert,

Thanks for your replies! Appreciate.

Can I safely say that there will be no downtime to the cluster when I
increase the pg_num and pgp_num values?

Looking forward to your reply, thank you.

Cheers.



On Tue, Dec 3, 2013 at 2:31 PM, Robert van Leeuwen <
robert.vanleeu...@spilgames.com> wrote:

>
> > On 2 dec. 2013, at 18:26, "Brian Andrus" 
> wrote:
>
> >  Setting your pg_num and pgp_num to say... 1024 would A) increase data
> granularity, B) likely lend no noticeable increase to resource consumption,
> and C) allow some room for future OSDs two be added while still within
> range of acceptable pg numbers. You could probably safely double even that
> number if you plan on expanding at a rapid rate and want to avoid splitting
> PGs every time a node is added.
> >
> > In general, you can conservatively err on the larger side when it comes
> to pg/p_num. Any excess resource utilization will be negligible (up to a
> certain point). If you have a comfortable amount of available RAM, you
> could experiment with increasing the multiplier in the equation you are
> using and see how it affects your final number.
> >
> > The pg_num and pgp_num parameters can safely be changed before or after
> your new nodes are integrated.
>
> I would be a bit conservative with the PGs / PGPs.
> I've experimented with the PG number a bit and noticed the following
> random IO performance drop.
> ( this could be something to our specific setup but since the PG is easily
> increased and impossible to decrease I would be conservative)
>
>  The setup:
> 3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
> Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.
>
> We use a replica count of 3 so optimum according to formula is about 1000
> With 1000 PGs I got about 2000-2500 random 4k IOPS.
>
> Because the nodes are fast enough and I expect the cluster to be expanded
> with 3 more nodes I set the PGs to 2000.
> Performance dropped to about 1200-1400 IOPS.
>
> I noticed that the spinning disks where no longer maxing out on 100% usage.
> Memory and CPU did not seem to be a problem.
> Since had the option to recreate the pool and I was not using the
> recommended settings I did not really dive into the issue.
> I will not stray to far from the recommended settings in the future though
> :)
>
> Cheers,
> Robert van Leeuwen
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Robert van Leeuwen]

> On 3 dec. 2013, at 10:49, "Indra Pramana"  wrote:
> 
> Hi Brian and Robert,
> 
> Thanks for your replies! Appreciate.
> 
> Can I safely say that there will be no downtime to the cluster when I 
> increase the pg_num and pgp_num values?

There is no actual downtime.
What I did see is that IOs will crawl to a halt during pg creation ( 1000 took 
a few minutes).
Also expect reduced performance during the rebalance of the data. 
The OSDs will be quite busy during that time.

I would certainly pick a time with low traffic to do this.


Cheers,
Robert van Leeuwen



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to fix active+remapped pg

[Ugis]

Hi,
Upgaded to emperor, restarted all nodes.

Still have "31 actige+remapped" pgs.

Compared remapped and healthy pg query output - some remapped pgs do
not have data, some do, some have been scrubbed some don't. Now
running read for whole rbd - may be that would trigger those stuck
pgs.

state on remapped pgs like:
{ "state": "active+remapped",
  "epoch": 9420,
  "up": [
9],
  "acting": [
9,
5],

Any help/hints how to trigger those stuck pgs to up state on 2 osds?

Ugis


2013/11/22 Ugis :
> Update: I noticed that I hadn't increased pgp_num for default data
> pool for which I increased pg_num time ago. So I did now and some
> backfilling happened.
> Now I still have "31 actige+remapped" pgs.
> Remapped pgs belong to all pools, even those where is no data.
> To me suspicious is that host ceph8 has weight 10.88(I had some osds
> there temporarily, but due to low ram I remover those)
> If that is of importance ceph7 is also low on ram(4GB) and is slower
> to respond at times than ceph5(Sage mentioned "lagging pg peering
> workqueue" in Bug#3747).
>
> Results follow:
> # ceph osd tree
> # idweight  type name   up/down reweight
> -5  0   root slow
> -4  0   host ceph5-slow
> -1  32.46   root default
> -2  10.5host ceph5
> 0   0.2 osd.0   up  0
> 2   2.8 osd.2   up  1
> 3   2.8 osd.3   up  1
> 4   1.9 osd.4   up  1
> 5   2.8 osd.5   up  1
> -3  0.2 host ceph6
> 1   0.2 osd.1   up  0
> -6  10.88   host ceph7
> 6   2.73osd.6   up  1
> 7   2.73osd.7   up  1
> 8   2.71osd.8   up  1
> 9   2.71osd.9   up  1
> -7  10.88   host ceph8
>
> # ceph osd crush dump
> { "devices": [
> { "id": 0,
>   "name": "osd.0"},
> { "id": 1,
>   "name": "osd.1"},
> { "id": 2,
>   "name": "osd.2"},
> { "id": 3,
>   "name": "osd.3"},
> { "id": 4,
>   "name": "osd.4"},
> { "id": 5,
>   "name": "osd.5"},
> { "id": 6,
>   "name": "osd.6"},
> { "id": 7,
>   "name": "osd.7"},
> { "id": 8,
>   "name": "osd.8"},
> { "id": 9,
>   "name": "osd.9"}],
>   "types": [
> { "type_id": 0,
>   "name": "osd"},
> { "type_id": 1,
>   "name": "host"},
> { "type_id": 2,
>   "name": "rack"},
> { "type_id": 3,
>   "name": "row"},
> { "type_id": 4,
>   "name": "room"},
> { "type_id": 5,
>   "name": "datacenter"},
> { "type_id": 6,
>   "name": "root"}],
>   "buckets": [
> { "id": -1,
>   "name": "default",
>   "type_id": 6,
>   "type_name": "root",
>   "weight": 2127297,
>   "alg": "straw",
>   "hash": "rjenkins1",
>   "items": [
> { "id": -2,
>   "weight": 688128,
>   "pos": 0},
> { "id": -3,
>   "weight": 13107,
>   "pos": 1},
> { "id": -6,
>   "weight": 713031,
>   "pos": 2},
> { "id": -7,
>   "weight": 713031,
>   "pos": 3}]},
> { "id": -2,
>   "name": "ceph5",
>   "type_id": 1,
>   "type_name": "host",
>   "weight": 688125,
>   "alg": "straw",
>   "hash": "rjenkins1",
>   "items": [
> { "id": 0,
>   "weight": 13107,
>   "pos": 0},
> { "id": 2,
>   "weight": 183500,
>   "pos": 1},
> { "id": 3,
>   "weight": 183500,
>   "pos": 2},
> { "id": 4,
>   "weight": 124518,
>   "pos": 3},
> { "id": 5,
>   "weight": 183500,
>   "pos": 4}]},
> { "id": -3,
>   "name": "ceph6",
>   "type_id": 1,
>   "type_name": "host",
>   "weight": 13107,
>   "alg": "straw",
>   "hash": "rjenkins1",
>   "items": [
> { "id": 1,
>   "weight": 13107,
>   "pos": 0}]},
> { "id": -4,
>   "name": "ceph5-slow",
>   "type_id": 1,
>   "type_name": "host",
>   "weight": 0,
>   "alg": "straw",
>   "hash": "rjenkins1",
>   "items": []},
> { "id": -5,
>   "name": "slow",
>   "type_id": 6,
>   "type_name": "root",
>   "weight": 0,
>   "alg": "straw",
>   "hash": "rjenkins1

[ceph-users] SSD journal partition and trim

[Loic Dachary]

Hi Ceph,

When an SSD partition is used to store a journal

https://github.com/ceph/ceph/blob/master/src/os/FileJournal.cc#L90

how is it trimmed ?

http://en.wikipedia.org/wiki/Trim_%28computing%29

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[James Pearce]

Since the journal partitions are generally small, it shouldn't need to 
be.


For example implement with substantial under-provisioning, either via 
HPA or simple partitions.


On 2013-12-03 12:18, Loic Dachary wrote:

Hi Ceph,

When an SSD partition is used to store a journal

https://github.com/ceph/ceph/blob/master/src/os/FileJournal.cc#L90

how is it trimmed ?

http://en.wikipedia.org/wiki/Trim_%28computing%29

Cheers

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[Emmanuel Lacour]

On Tue, Dec 03, 2013 at 12:38:54PM +, James Pearce wrote:
> Since the journal partitions are generally small, it shouldn't need
> to be.
> 

here with 2 journals (2 osds) on two ssd (samsung 850 pro, soft raid1 +
lvm + xfs) trim is just obligatory. We forget to set it at cluster setup
and one week later, after migrating a lot of VM, the performances
dropped to something like 10MB/s. Running fstrim on it gave back normal
performances.

-- 
Easter-eggs  Spécialiste GNU/Linux
44-46 rue de l'Ouest  -  75014 Paris  -  France -  Métro Gaité
Phone: +33 (0) 1 43 35 00 37-   Fax: +33 (0) 1 43 35 00 76
mailto:elac...@easter-eggs.com  -   http://www.easter-eggs.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[James Pearce]

How much (%) is left unprovisioned on those (840s?) ?  And were they 
trim'd/secure erased before deployment?


On 2013-12-03 12:45, Emmanuel Lacour wrote:

On Tue, Dec 03, 2013 at 12:38:54PM +, James Pearce wrote:

Since the journal partitions are generally small, it shouldn't need
to be.



here with 2 journals (2 osds) on two ssd (samsung 850 pro, soft raid1 
+
lvm + xfs) trim is just obligatory. We forget to set it at cluster 
setup

and one week later, after migrating a lot of VM, the performances
dropped to something like 10MB/s. Running fstrim on it gave back 
normal

performances.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[Emmanuel Lacour]

On Tue, Dec 03, 2013 at 12:48:21PM +, James Pearce wrote:
> How much (%) is left unprovisioned on those (840s?) ?  And were they
> trim'd/secure erased before deployment?
> 

unfortunatly, everything was provisioned (thought, there is free spaces
in the VG) due to lack of knowledge.

Nothing special was made before running the cluster, just a somewhat
standard Debian install on top of those SSDs.

That may explain we needed to run fstrim (I added a daily cron for
this).


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[James Pearce]

Most likely.  When fully provisioned the device has a much smaller pool 
of cells to manage (i.e. charge) in the background, hence once that pool 
is exhausted the device has no option but to stall whilst it clears 
(re-charges) a cell, which takes something like 2-5ms.


Daily cron task is though still a good idea - enabling discard mount 
option is generally counter-productive since trim is issue way too 
often, destroying performance (in my testing).


On 2013-12-03 13:12, Emmanuel Lacour wrote:

On Tue, Dec 03, 2013 at 12:48:21PM +, James Pearce wrote:

How much (%) is left unprovisioned on those (840s?) ?  And were they
trim'd/secure erased before deployment?



unfortunatly, everything was provisioned (thought, there is free 
spaces

in the VG) due to lack of knowledge.

Nothing special was made before running the cluster, just a somewhat
standard Debian install on top of those SSDs.

That may explain we needed to run fstrim (I added a daily cron for
this).


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Impact of fancy striping

[nicolasc]

Hi Kyle,

All OSDs are SATA drives in JBOD. The journals are all on a pair of SAS 
in RAID0. All of those are on a shared backplane with a single RAID 
controller (8 ports -> 12 disks).


I also have a pair of SAS in RAID1 holding the OS, which may be on a 
different port/data-path. I am going to try and put the journals there, 
and see how that goes. Thank you for this answer, Kyle.


Once again, every solution to my fancy striping problems is about the 
journal settings. I agree, my journal setup is not optimal, but I would 
really appreciate it if someone could:
- explain why the journal setup is way more important than striping 
settings;
- give me hints about fancy striping itself, and about how to make it 
work with XFS on RBD.


Best regards,

Nicolas Canceill
Scalable Storage Systems
SURFsara (Amsterdam, NL)


On 12/01/2013 12:47 AM, Kyle Bader wrote:


> This journal problem is a bit of wizardry to me, I even had weird 
intermittent issues with OSDs not starting because the journal was not 
found, so please do not hesitate to suggest a better journal setup.


You mentioned using SAS for journal, if your OSDs are SATA and a 
expander is in the data path it might be slow from MUX/STP/etc 
overhead. If the setup is all SAS you might try collocating the 
journal with it's matching data partition on a single disk. Two 
spindles must be contended with 9 OSDs. How are your drives attached?




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph on Raspberry Pi

[Shlomo Dubrowin]

I'm trying to deploy Ceph on a group of Raspberry Pis using the procedure
documented in: http://ceph.com/docs/master/start/quick-ceph-deploy/

There used to be a site: http://ceph.com/docs/master/start/quick-start/ but
that page is no longer valid.

The first thing I noticed is that the command lsb_release -sc specified in
the Ceph Deploy setup provides n/a even though the answer should be wheezy.
 I manually changed the /etc/apt/sources.list.d/ceph.list to specify wheezy.

I've installed ceph-deploy and setup the keys to communicate between the
deploy Raspberry Pi and all the Raspberry Pis including the system reunning
ceph-deploy.

When I tried to run ceph-deploy install  the installation failed.
I went to each node and installed ceph manually and I see the versions are:

$ ceph-deploy --version
1.3.3

$ ceph --version
ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)


All the nodes have the same ceph version.

When I try to run a command, I get errors:

$ ceph-deploy mon create baxter
[ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon create
baxter
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts baxter
[ceph_deploy.mon][DEBUG ] detecting platform for host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote host
[baxter][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: debian 7.0 wheezy
[baxter][DEBUG ] determining if provided host has same hostname in remote
[baxter][DEBUG ] get remote short hostname
[baxter][DEBUG ] deploying mon to baxter
[baxter][DEBUG ] get remote short hostname
[baxter][DEBUG ] remote hostname: baxter
[baxter][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[baxter][DEBUG ] create the mon path if it does not exist
[baxter][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-baxter/done
[baxter][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-baxter/done
[baxter][INFO  ] creating keyring file:
/var/lib/ceph/tmp/ceph-baxter.mon.keyring
[baxter][DEBUG ] create the monitor keyring file
[baxter][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
[baxter][WARNIN] too many arguments: [--cluster,ceph]
[baxter][WARNIN] usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
[baxter][WARNIN]   --debug_mon n
[baxter][WARNIN] debug monitor level (e.g. 10)
[baxter][WARNIN]   --mkfs
[baxter][WARNIN] build fresh monitor fs
[baxter][DEBUG ] --conf/-cRead configuration from the given
configuration file
[baxter][DEBUG ] -d   Run in foreground, log to stderr.
[baxter][DEBUG ] -f   Run in foreground, log to usual location.
[baxter][DEBUG ] --id/-i  set ID portion of my name
[baxter][DEBUG ] --name/-nset name (TYPE.ID)
[baxter][DEBUG ] --versionshow version and quit
[baxter][DEBUG ]
[baxter][DEBUG ]--debug_ms N
[baxter][DEBUG ] set message debug level (e.g. 1)
[baxter][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.mon][ERROR ] Failed to execute command: ceph-mon --cluster
ceph --mkfs -i baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors


If I try to run the same command as the user ceph on the local machine, I
get the same error:

$ ceph-mon --cluster ceph --mkfs -i baxter --keyring
/var/lib/ceph/tmp/ceph-baxter.mon.keyring
failed to open log file '/var/log/ceph/mon.baxter.log': (13) Permission
denied
too many arguments: [--cluster,ceph]
usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
  --debug_mon n
debug monitor level (e.g. 10)
  --mkfs
build fresh monitor fs
--conf/-cRead configuration from the given configuration file
-d   Run in foreground, log to stderr.
-f   Run in foreground, log to usual location.
--id/-i  set ID portion of my name
--name/-nset name (TYPE.ID)
--versionshow version and quit

   --debug_ms N
set message debug level (e.g. 1)



This makes me think that the ceph-deploy installed doesn't work with the
ceph installed on each node.

So I guess I have 2 questions:

   1. How can I determine which ceph-deploy goes with which ceph and how
   can I correct this apparent mismatch?
   2. Should I be looking at just using local ceph commands to configure
   this by hand? Are there instructions on how to do this by hand?

Thank you.

  Shlomo

-
Shlomo Dubrowin

The Solution to the water crisis in Israel:

# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
  cat /sea/med /sea/red |\
  grep -vi "salt" |\
  tee /sea/dead /lake/kinneret
  LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done
___
ceph-users mailing list
ceph-u

Re: [ceph-users] Impact of fancy striping

[James Pearce]

I would really appreciate it if someone could:
 - explain why the journal setup is way more important than striping 
settings;


I'm not sure if it's what you're asking, but any write must be 
physically written to the journal before the operation is acknowledged.  
So the overall cluster performance (or rather write latency) is always 
governed by the speed of those journals.  Data is then gathered up into 
(hopefully) larger blocks and committed to OSDs later.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[Emmanuel Lacour]

On Tue, Dec 03, 2013 at 01:22:50PM +, James Pearce wrote:
> 
> Daily cron task is though still a good idea - enabling discard mount
> option is generally counter-productive since trim is issue way too
> often, destroying performance (in my testing).
> 

yes that's why we are using cron here.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph on Raspberry Pi

[Alfredo Deza]

On Tue, Dec 3, 2013 at 8:55 AM, Shlomo Dubrowin  wrote:
> I'm trying to deploy Ceph on a group of Raspberry Pis using the procedure
> documented in: http://ceph.com/docs/master/start/quick-ceph-deploy/
>
> There used to be a site: http://ceph.com/docs/master/start/quick-start/ but
> that page is no longer valid.
>
> The first thing I noticed is that the command lsb_release -sc specified in
> the Ceph Deploy setup provides n/a even though the answer should be wheezy.
> I manually changed the /etc/apt/sources.list.d/ceph.list to specify wheezy.
>
> I've installed ceph-deploy and setup the keys to communicate between the
> deploy Raspberry Pi and all the Raspberry Pis including the system reunning
> ceph-deploy.
>
> When I tried to run ceph-deploy install  the installation failed.
> I went to each node and installed ceph manually and I see the versions are:
>
> $ ceph-deploy --version
> 1.3.3
>
> $ ceph --version
> ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)
>
>
> All the nodes have the same ceph version.

That looks like a *very* old ceph version. Is there any reason you are
using 0.43 as opposed to the latest one?

How did you installed Ceph on those nodes?
>
> When I try to run a command, I get errors:
>
> $ ceph-deploy mon create baxter
> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon create
> baxter
> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts baxter
> [ceph_deploy.mon][DEBUG ] detecting platform for host baxter ...
> [baxter][DEBUG ] connected to host: baxter
> [baxter][DEBUG ] detect platform information from remote host
> [baxter][DEBUG ] detect machine type
> [ceph_deploy.mon][INFO  ] distro info: debian 7.0 wheezy
> [baxter][DEBUG ] determining if provided host has same hostname in remote
> [baxter][DEBUG ] get remote short hostname
> [baxter][DEBUG ] deploying mon to baxter
> [baxter][DEBUG ] get remote short hostname
> [baxter][DEBUG ] remote hostname: baxter
> [baxter][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [baxter][DEBUG ] create the mon path if it does not exist
> [baxter][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-baxter/done
> [baxter][DEBUG ] done path does not exist:
> /var/lib/ceph/mon/ceph-baxter/done
> [baxter][INFO  ] creating keyring file:
> /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> [baxter][DEBUG ] create the monitor keyring file
> [baxter][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
> baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> [baxter][WARNIN] too many arguments: [--cluster,ceph]
> [baxter][WARNIN] usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
> [baxter][WARNIN]   --debug_mon n
> [baxter][WARNIN] debug monitor level (e.g. 10)
> [baxter][WARNIN]   --mkfs
> [baxter][WARNIN] build fresh monitor fs
> [baxter][DEBUG ] --conf/-cRead configuration from the given
> configuration file
> [baxter][DEBUG ] -d   Run in foreground, log to stderr.
> [baxter][DEBUG ] -f   Run in foreground, log to usual location.
> [baxter][DEBUG ] --id/-i  set ID portion of my name
> [baxter][DEBUG ] --name/-nset name (TYPE.ID)
> [baxter][DEBUG ] --versionshow version and quit
> [baxter][DEBUG ]
> [baxter][DEBUG ]--debug_ms N
> [baxter][DEBUG ] set message debug level (e.g. 1)
> [baxter][ERROR ] RuntimeError: command returned non-zero exit status: 1
> [ceph_deploy.mon][ERROR ] Failed to execute command: ceph-mon --cluster ceph
> --mkfs -i baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> [ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
>
>
> If I try to run the same command as the user ceph on the local machine, I
> get the same error:
>
> $ ceph-mon --cluster ceph --mkfs -i baxter --keyring
> /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> failed to open log file '/var/log/ceph/mon.baxter.log': (13) Permission
> denied
> too many arguments: [--cluster,ceph]
> usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
>   --debug_mon n
> debug monitor level (e.g. 10)
>   --mkfs
> build fresh monitor fs
> --conf/-cRead configuration from the given configuration file
> -d   Run in foreground, log to stderr.
> -f   Run in foreground, log to usual location.
> --id/-i  set ID portion of my name
> --name/-nset name (TYPE.ID)
> --versionshow version and quit
>
>--debug_ms N
> set message debug level (e.g. 1)
>
>
>
> This makes me think that the ceph-deploy installed doesn't work with the
> ceph installed on each node.
>
> So I guess I have 2 questions:
>
> How can I determine which ceph-deploy goes with which ceph and how can I
> correct this apparent mismatch?

I don't think there was a ceph-deploy per-se that would've worked for
that Ceph version.

I believe that ceph-deploy is very backwards compatible for a few Ceph
versions. At the very least we should
fully support the latest 3 majo

Re: [ceph-users] Ceph on Raspberry Pi

[Shlomo Dubrowin]

Alfredo,

Thank you for your response.  I simply did apt-get install ceph on the
nodes.

My /etc/apt/sources.list.d/ceph.list contains:

deb http://ceph.com/debian-emperor/ wheezy main


and the versions I received are what I got.

  Shlomo

-
Shlomo Dubrowin

The Solution to the water crisis in Israel:

# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
  cat /sea/med /sea/red |\
  grep -vi "salt" |\
  tee /sea/dead /lake/kinneret
  LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done


On Tue, Dec 3, 2013 at 4:15 PM, Alfredo Deza wrote:

> On Tue, Dec 3, 2013 at 8:55 AM, Shlomo Dubrowin 
> wrote:
> > I'm trying to deploy Ceph on a group of Raspberry Pis using the procedure
> > documented in: http://ceph.com/docs/master/start/quick-ceph-deploy/
> >
> > There used to be a site: http://ceph.com/docs/master/start/quick-start/but
> > that page is no longer valid.
> >
> > The first thing I noticed is that the command lsb_release -sc specified
> in
> > the Ceph Deploy setup provides n/a even though the answer should be
> wheezy.
> > I manually changed the /etc/apt/sources.list.d/ceph.list to specify
> wheezy.
> >
> > I've installed ceph-deploy and setup the keys to communicate between the
> > deploy Raspberry Pi and all the Raspberry Pis including the system
> reunning
> > ceph-deploy.
> >
> > When I tried to run ceph-deploy install  the installation failed.
> > I went to each node and installed ceph manually and I see the versions
> are:
> >
> > $ ceph-deploy --version
> > 1.3.3
> >
> > $ ceph --version
> > ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)
> >
> >
> > All the nodes have the same ceph version.
>
> That looks like a *very* old ceph version. Is there any reason you are
> using 0.43 as opposed to the latest one?
>
> How did you installed Ceph on those nodes?
> >
> > When I try to run a command, I get errors:
> >
> > $ ceph-deploy mon create baxter
> > [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
> create
> > baxter
> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts baxter
> > [ceph_deploy.mon][DEBUG ] detecting platform for host baxter ...
> > [baxter][DEBUG ] connected to host: baxter
> > [baxter][DEBUG ] detect platform information from remote host
> > [baxter][DEBUG ] detect machine type
> > [ceph_deploy.mon][INFO  ] distro info: debian 7.0 wheezy
> > [baxter][DEBUG ] determining if provided host has same hostname in remote
> > [baxter][DEBUG ] get remote short hostname
> > [baxter][DEBUG ] deploying mon to baxter
> > [baxter][DEBUG ] get remote short hostname
> > [baxter][DEBUG ] remote hostname: baxter
> > [baxter][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> > [baxter][DEBUG ] create the mon path if it does not exist
> > [baxter][DEBUG ] checking for done path:
> /var/lib/ceph/mon/ceph-baxter/done
> > [baxter][DEBUG ] done path does not exist:
> > /var/lib/ceph/mon/ceph-baxter/done
> > [baxter][INFO  ] creating keyring file:
> > /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> > [baxter][DEBUG ] create the monitor keyring file
> > [baxter][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
> > baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> > [baxter][WARNIN] too many arguments: [--cluster,ceph]
> > [baxter][WARNIN] usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
> > [baxter][WARNIN]   --debug_mon n
> > [baxter][WARNIN] debug monitor level (e.g. 10)
> > [baxter][WARNIN]   --mkfs
> > [baxter][WARNIN] build fresh monitor fs
> > [baxter][DEBUG ] --conf/-cRead configuration from the given
> > configuration file
> > [baxter][DEBUG ] -d   Run in foreground, log to stderr.
> > [baxter][DEBUG ] -f   Run in foreground, log to usual
> location.
> > [baxter][DEBUG ] --id/-i  set ID portion of my name
> > [baxter][DEBUG ] --name/-nset name (TYPE.ID)
> > [baxter][DEBUG ] --versionshow version and quit
> > [baxter][DEBUG ]
> > [baxter][DEBUG ]--debug_ms N
> > [baxter][DEBUG ] set message debug level (e.g. 1)
> > [baxter][ERROR ] RuntimeError: command returned non-zero exit status: 1
> > [ceph_deploy.mon][ERROR ] Failed to execute command: ceph-mon --cluster
> ceph
> > --mkfs -i baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> > [ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
> >
> >
> > If I try to run the same command as the user ceph on the local machine, I
> > get the same error:
> >
> > $ ceph-mon --cluster ceph --mkfs -i baxter --keyring
> > /var/lib/ceph/tmp/ceph-baxter.mon.keyring
> > failed to open log file '/var/log/ceph/mon.baxter.log': (13) Permission
> > denied
> > too many arguments: [--cluster,ceph]
> > usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
> >   --debug_mon n
> > debug monitor level (e.g. 10)
> >   --mkfs
> >

Re: [ceph-users] Ceph on Raspberry Pi

[Alfredo Deza]

On Tue, Dec 3, 2013 at 9:21 AM, Shlomo Dubrowin  wrote:
> Alfredo,
>
> Thank you for your response.  I simply did apt-get install ceph on the
> nodes.
>
> My /etc/apt/sources.list.d/ceph.list contains:
>
> deb http://ceph.com/debian-emperor/ wheezy main
>

Was that added manually? ceph-deploy can take care of handling the
sources list for you.

It is possible that you have something else in that machine that is
saying that 0.43 is the version you should
be getting.

Have you tried to install with ceph-deploy?:

ceph-deploy install baxter



>
> and the versions I received are what I got.
>
>   Shlomo
>
> -
> Shlomo Dubrowin
>
> The Solution to the water crisis in Israel:
>
> # According to WikiPedia, the Kinneret can hold
> # 4 km^3, so FULL here is in cubit meters
> FULL="4000"
> while [ "$LEVEL" -lt "$FULL" ]; do
>   cat /sea/med /sea/red |\
>   grep -vi "salt" |\
>   tee /sea/dead /lake/kinneret
>   LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
> done
>
>
> On Tue, Dec 3, 2013 at 4:15 PM, Alfredo Deza 
> wrote:
>>
>> On Tue, Dec 3, 2013 at 8:55 AM, Shlomo Dubrowin 
>> wrote:
>> > I'm trying to deploy Ceph on a group of Raspberry Pis using the
>> > procedure
>> > documented in: http://ceph.com/docs/master/start/quick-ceph-deploy/
>> >
>> > There used to be a site: http://ceph.com/docs/master/start/quick-start/
>> > but
>> > that page is no longer valid.
>> >
>> > The first thing I noticed is that the command lsb_release -sc specified
>> > in
>> > the Ceph Deploy setup provides n/a even though the answer should be
>> > wheezy.
>> > I manually changed the /etc/apt/sources.list.d/ceph.list to specify
>> > wheezy.
>> >
>> > I've installed ceph-deploy and setup the keys to communicate between the
>> > deploy Raspberry Pi and all the Raspberry Pis including the system
>> > reunning
>> > ceph-deploy.
>> >
>> > When I tried to run ceph-deploy install  the installation failed.
>> > I went to each node and installed ceph manually and I see the versions
>> > are:
>> >
>> > $ ceph-deploy --version
>> > 1.3.3
>> >
>> > $ ceph --version
>> > ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)
>> >
>> >
>> > All the nodes have the same ceph version.
>>
>> That looks like a *very* old ceph version. Is there any reason you are
>> using 0.43 as opposed to the latest one?
>>
>> How did you installed Ceph on those nodes?
>> >
>> > When I try to run a command, I get errors:
>> >
>> > $ ceph-deploy mon create baxter
>> > [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
>> > create
>> > baxter
>> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts baxter
>> > [ceph_deploy.mon][DEBUG ] detecting platform for host baxter ...
>> > [baxter][DEBUG ] connected to host: baxter
>> > [baxter][DEBUG ] detect platform information from remote host
>> > [baxter][DEBUG ] detect machine type
>> > [ceph_deploy.mon][INFO  ] distro info: debian 7.0 wheezy
>> > [baxter][DEBUG ] determining if provided host has same hostname in
>> > remote
>> > [baxter][DEBUG ] get remote short hostname
>> > [baxter][DEBUG ] deploying mon to baxter
>> > [baxter][DEBUG ] get remote short hostname
>> > [baxter][DEBUG ] remote hostname: baxter
>> > [baxter][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
>> > [baxter][DEBUG ] create the mon path if it does not exist
>> > [baxter][DEBUG ] checking for done path:
>> > /var/lib/ceph/mon/ceph-baxter/done
>> > [baxter][DEBUG ] done path does not exist:
>> > /var/lib/ceph/mon/ceph-baxter/done
>> > [baxter][INFO  ] creating keyring file:
>> > /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>> > [baxter][DEBUG ] create the monitor keyring file
>> > [baxter][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i
>> > baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>> > [baxter][WARNIN] too many arguments: [--cluster,ceph]
>> > [baxter][WARNIN] usage: ceph-mon -i monid [--mon-data=pathtodata]
>> > [flags]
>> > [baxter][WARNIN]   --debug_mon n
>> > [baxter][WARNIN] debug monitor level (e.g. 10)
>> > [baxter][WARNIN]   --mkfs
>> > [baxter][WARNIN] build fresh monitor fs
>> > [baxter][DEBUG ] --conf/-cRead configuration from the given
>> > configuration file
>> > [baxter][DEBUG ] -d   Run in foreground, log to stderr.
>> > [baxter][DEBUG ] -f   Run in foreground, log to usual
>> > location.
>> > [baxter][DEBUG ] --id/-i  set ID portion of my name
>> > [baxter][DEBUG ] --name/-nset name (TYPE.ID)
>> > [baxter][DEBUG ] --versionshow version and quit
>> > [baxter][DEBUG ]
>> > [baxter][DEBUG ]--debug_ms N
>> > [baxter][DEBUG ] set message debug level (e.g. 1)
>> > [baxter][ERROR ] RuntimeError: command returned non-zero exit status: 1
>> > [ceph_deploy.mon][ERROR ] Failed to execute command: ceph-mon --cluster
>> > ceph
>> > --mkfs -i baxter --keyring /var/lib/ceph/tmp/ceph-baxter.mon.keyring
>> > [ceph_deploy][ERROR ] 

Re: [ceph-users] Ceph on Raspberry Pi

[Shlomo Dubrowin]

Alfredo,

I started that way, but I run into an error:

$ ceph-deploy install baxter
[ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
baxter
[ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster
ceph hosts baxter
[ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote host
[baxter][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
[baxter][INFO  ] installing ceph on baxter
[baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
apt-get -q install --assume-yes ca-certificates
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][DEBUG ] ca-certificates is already the newest version.
[baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
upgraded.
[baxter][INFO  ] Running command: sudo wget -O release.asc
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] --2013-12-03 16:32:45--
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
[baxter][WARNIN] Connecting to ceph.com (ceph.com)|208.113.241.137|:443...
connected.
[baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
[baxter][WARNIN] Length: unspecified [text/plain]
[baxter][WARNIN] Saving to: `release.asc'
[baxter][WARNIN]
[baxter][WARNIN]  0K .
 1.06M=0.002s
[baxter][WARNIN]
[baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
[1752]
[baxter][WARNIN]
[baxter][INFO  ] Running command: sudo apt-key add release.asc
[baxter][DEBUG ] OK
[baxter][DEBUG ] add ceph deb repo to sources.list
[baxter][INFO  ] Running command: sudo apt-get -q update
[baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy
Release.gpg [490 B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
[baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
[490 B]
[baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
[14.4 kB]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
[baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
[baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release [7224
B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
Packages
[baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main armhf
Packages [7414 kB]
[baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
Packages [12.1 kB]
[baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
[baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
Translation-en
[baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages [1010 B]
[baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib
armhf Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
armhf Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
Packages
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
Translation-en
[baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
[baxter][DEBUG ] Reading package lists...
[baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
--no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common
ceph-fs-common gdisk
[baxter][WARNIN] E: Unable to locate package ceph-mds
[baxter][WARNIN] E: Unable to locate package ceph-fs-common
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][ERROR ] RuntimeError: command returned non-zero exit status: 100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o
Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes
install -- ceph ceph-mds ceph-common ceph-fs-common gdisk


When running manually:

$ env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o
Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes
install -- ceph ceph-mds ceph-common ceph-fs-common gdisk
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission
denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you
root?


Maybe this is supposed to be a sudo command?

$ sudo env DEBIAN

Re: [ceph-users] Ceph on Raspberry Pi

[Alfredo Deza]

On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin  wrote:
> Alfredo,
>
> I started that way, but I run into an error:
>
> $ ceph-deploy install baxter
> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
> baxter
> [ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster
> ceph hosts baxter
> [ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
> [baxter][DEBUG ] connected to host: baxter
> [baxter][DEBUG ] detect platform information from remote host
> [baxter][DEBUG ] detect machine type
> [ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
> [baxter][INFO  ] installing ceph on baxter
> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
> apt-get -q install --assume-yes ca-certificates
> [baxter][DEBUG ] Reading package lists...
> [baxter][DEBUG ] Building dependency tree...
> [baxter][DEBUG ] Reading state information...
> [baxter][DEBUG ] ca-certificates is already the newest version.
> [baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
> upgraded.
> [baxter][INFO  ] Running command: sudo wget -O release.asc
> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
> [baxter][WARNIN] --2013-12-03 16:32:45--
> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
> [baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
> [baxter][WARNIN] Connecting to ceph.com (ceph.com)|208.113.241.137|:443...
> connected.
> [baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
> [baxter][WARNIN] Length: unspecified [text/plain]
> [baxter][WARNIN] Saving to: `release.asc'
> [baxter][WARNIN]
> [baxter][WARNIN]  0K .
> 1.06M=0.002s
> [baxter][WARNIN]
> [baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
> [1752]
> [baxter][WARNIN]
> [baxter][INFO  ] Running command: sudo apt-key add release.asc
> [baxter][DEBUG ] OK
> [baxter][DEBUG ] add ceph deb repo to sources.list
> [baxter][INFO  ] Running command: sudo apt-get -q update
> [baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy Release.gpg
> [490 B]
> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
> [baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
> [490 B]
> [baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
> [14.4 kB]
> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
> [baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
> [baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release [7224
> B]
> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
> Packages
> [baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main armhf
> Packages [7414 kB]
> [baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
> Packages [12.1 kB]
> [baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
> [baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
> Translation-en
> [baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages [1010 B]
> [baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
> Translation-en
> [baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib armhf
> Packages
> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
> armhf Packages
> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
> Packages
> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
> Translation-en
> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
> Translation-en
> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
> Translation-en
> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
> Translation-en
> [baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
> [baxter][DEBUG ] Reading package lists...
> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
> DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
> --no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common
> ceph-fs-common gdisk
> [baxter][WARNIN] E: Unable to locate package ceph-mds
> [baxter][WARNIN] E: Unable to locate package ceph-fs-common
> [baxter][DEBUG ] Reading package lists...
> [baxter][DEBUG ] Building dependency tree...
> [baxter][DEBUG ] Reading state information...
> [baxter][ERROR ] RuntimeError: command returned non-zero exit status: 100
> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
> DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o
> Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install
> -- ceph ceph-mds ceph-common ceph-fs-common gdisk

That output looks unexpected. I wonder if it is just a network hiccup
to fail to get to those two packages.
>
>
> When running manually:
>
> $ env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical ap

[ceph-users] Journal, SSD and OS

[Gandalf Corvotempesta]

Hi,
what do you think to use the same SSD as journal and as root partition?

Forexample:
 1x 128GB SSD
 6 OSD
 15GB for each journal, for each OSD
 5GB as root partition for OS.

This give me 105GB of used space and 23GB of unused space (i've read
somewhere that is better to not use the whole SSD for durability)

The root partition will be mounted with "noatime, nodiratime"

/tmp
/var/tmp
/var/run
/var/lock

will be mounted on a ramfs

All logs are stored remotely via rsyslogd
Is this good ? AFAIK, in this configuration, ceph will be executed
totally in ram.

Can I put the root partition in a RAID1 between 2 SSD (both will be
used as journal, 6 OSDs on SSD1, 6 OSDs on disk2, total 12 OSDs)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[Loic Dachary]

On 03/12/2013 13:38, James Pearce wrote:
> Since the journal partitions are generally small, it shouldn't need to be.
> 
> For example implement with substantial under-provisioning, either via HPA or 
> simple partitions.
> 

Does that mean the problem will happen much later or that it will never happen 
? As far as I understand it is just postponing it but I just discover this and 
may be completely mistaken :-)

Cheers

> On 2013-12-03 12:18, Loic Dachary wrote:
>> Hi Ceph,
>>
>> When an SSD partition is used to store a journal
>>
>> https://github.com/ceph/ceph/blob/master/src/os/FileJournal.cc#L90
>>
>> how is it trimmed ?
>>
>> http://en.wikipedia.org/wiki/Trim_%28computing%29
>>
>> Cheers
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Mike Dawson]

Robert,

Interesting results on the effect of # of PG/PGPs. My cluster struggles 
a bit under the strain of heavy random small-sized writes.


The IOPS you mention seem high to me given 30 drives and 3x replication 
unless they were pure reads or on high-rpm drives. Instead of assuming, 
I want to pose a few questions:


- How are you testing? rados bench, rbd bench, rbd bench with writeback 
cache, etc?


- Were the 2000-2500 random 4k IOPS more reads than writes? If you test 
100% 4k random reads, what do you get? If you test 100% 4k random 
writes, what do you get?


- What drives do you have? Any RAID involved under your OSDs?

Thanks,
Mike Dawson


On 12/3/2013 1:31 AM, Robert van Leeuwen wrote:



On 2 dec. 2013, at 18:26, "Brian Andrus"  wrote:



  Setting your pg_num and pgp_num to say... 1024 would A) increase data 
granularity, B) likely lend no noticeable increase to resource consumption, and 
C) allow some room for future OSDs two be added while still within range of 
acceptable pg numbers. You could probably safely double even that number if you 
plan on expanding at a rapid rate and want to avoid splitting PGs every time a 
node is added.

In general, you can conservatively err on the larger side when it comes to 
pg/p_num. Any excess resource utilization will be negligible (up to a certain 
point). If you have a comfortable amount of available RAM, you could experiment 
with increasing the multiplier in the equation you are using and see how it 
affects your final number.

The pg_num and pgp_num parameters can safely be changed before or after your 
new nodes are integrated.


I would be a bit conservative with the PGs / PGPs.
I've experimented with the PG number a bit and noticed the following random IO 
performance drop.
( this could be something to our specific setup but since the PG is easily 
increased and impossible to decrease I would be conservative)

  The setup:
3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.

We use a replica count of 3 so optimum according to formula is about 1000
With 1000 PGs I got about 2000-2500 random 4k IOPS.

Because the nodes are fast enough and I expect the cluster to be expanded with 
3 more nodes I set the PGs to 2000.
Performance dropped to about 1200-1400 IOPS.

I noticed that the spinning disks where no longer maxing out on 100% usage.
Memory and CPU did not seem to be a problem.
Since had the option to recreate the pool and I was not using the recommended 
settings I did not really dive into the issue.
I will not stray to far from the recommended settings in the future though :)

Cheers,
Robert van Leeuwen
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Radosgw on Ubuntu vs CentOS

[Andy McCrae]

Hi ceph-users,

I've been playing around with radosgw and I notice there is an
inconsistency between the Ubuntu and CentOS startup scripts.

On Ubuntu, if I run a start ceph-all (which will start radosgw), or I run
the init script /etc/init.d/radosgw start - the radosgw process starts up
fine, but running as root.

On CentOS the init script starts radosgw as the "apache" user by default.

I can see the Ubuntu init script is specifying "www-data" which would be in
keeping with the CentOS init script, but the process runs as root.

+ start-stop-daemon --start -u www-data -x /usr/bin/radosgw -- -n
client.radosgw.ubunTest
2013-12-03 15:13:26.449087 7fee1d33b780 -1 WARNING: libcurl doesn't support
curl_multi_wait()
2013-12-03 15:13:26.449093 7fee1d33b780 -1 WARNING: cross zone / region
transfer performance may be affected
root@ubunTest:~# ps -ef | grep radosgw
root 28528 1  0 15:13 ?00:00:00 /usr/bin/radosgw -n
client.radosgw.ubunTest


The question is, do we consider this a bug in that radosgw shouldn't run as
root by default, or should the CentOS/RHEL (rpm) init scripts start radosgw
as root - I'd assume the former.

Thanks!
Andy
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph on Raspberry Pi

[Mark Nelson]

Guys, I don't think we have pre-released packages of anything new that 
is going to work on the pi regardless if you use ceph-deploy.  Look at 
our armhf packages file:


http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages

Unless I'm mistaken, you're going to have to compile it yourself.  I 
think Joao was going to try that, not sure if he ever got around to it 
though.


Mark

On 12/03/2013 09:03 AM, Alfredo Deza wrote:

On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin  wrote:

Alfredo,

I started that way, but I run into an error:

$ ceph-deploy install baxter
[ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
baxter
[ceph_deploy.install][DEBUG ] Installing stable version emperor on cluster
ceph hosts baxter
[ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote host
[baxter][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
[baxter][INFO  ] installing ceph on baxter
[baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
apt-get -q install --assume-yes ca-certificates
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][DEBUG ] ca-certificates is already the newest version.
[baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
upgraded.
[baxter][INFO  ] Running command: sudo wget -O release.asc
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] --2013-12-03 16:32:45--
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
[baxter][WARNIN] Connecting to ceph.com (ceph.com)|208.113.241.137|:443...
connected.
[baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
[baxter][WARNIN] Length: unspecified [text/plain]
[baxter][WARNIN] Saving to: `release.asc'
[baxter][WARNIN]
[baxter][WARNIN]  0K .
1.06M=0.002s
[baxter][WARNIN]
[baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
[1752]
[baxter][WARNIN]
[baxter][INFO  ] Running command: sudo apt-key add release.asc
[baxter][DEBUG ] OK
[baxter][DEBUG ] add ceph deb repo to sources.list
[baxter][INFO  ] Running command: sudo apt-get -q update
[baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy Release.gpg
[490 B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
[baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
[490 B]
[baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
[14.4 kB]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
[baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
[baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release [7224
B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
Packages
[baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main armhf
Packages [7414 kB]
[baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
Packages [12.1 kB]
[baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
[baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
Translation-en
[baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages [1010 B]
[baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib armhf
Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
armhf Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
Packages
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
Translation-en
[baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
[baxter][DEBUG ] Reading package lists...
[baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
--no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common
ceph-fs-common gdisk
[baxter][WARNIN] E: Unable to locate package ceph-mds
[baxter][WARNIN] E: Unable to locate package ceph-fs-common
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][ERROR ] RuntimeError: command returned non-zero exit status: 100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q -o
Dpkg::Options::=--force-confnew --no-install-recommends --assume-yes install
-- 

[ceph-users] Journaling logs on ssds

[Kenneth Waegeman]

Hi all,

I have some questions about the journaling logs.
My setup: I have 3 hosts, each having 12 * 4TB sata disks and 2 *  
200GB ssd disks, 12 cores (2 hexacores) and 32GB RAM per host.


- I read that it is not a good idea to put the OS on the same SSD than  
the journals
- I also read that it's not a good idea to put all the journals on the  
same SSD, because if that SSD fail, all osds of that hosts will fail  
(because the journals are lost). Also 12 partitions on one ssd is  
probably too much?
- An RAID1 SSD array would fix the total failure issue, but then the  
OS is on the same disk again and this could be a performance bottleneck


Is my assumption correct that none of these configurations are  
recommended, and if so: is one of the configuration acceptable? Are  
there other configurations I could use with this setup?


A last question: Is it a good idea to put the journal of XFS itself  
(or other underlying fs) on the ssd?


Thanks for your help!

Kind regards,
Kenneth Waegeman





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph on Raspberry Pi

[Alfredo Deza]

On Tue, Dec 3, 2013 at 10:21 AM, Mark Nelson  wrote:
> Guys, I don't think we have pre-released packages of anything new that is
> going to work on the pi regardless if you use ceph-deploy.  Look at our
> armhf packages file:
>
> http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages
>
> Unless I'm mistaken, you're going to have to compile it yourself.  I think
> Joao was going to try that, not sure if he ever got around to it though.

Oh, good point. I was just assuming that because we support the Distro
the packages would exist.

>
> Mark
>
>
> On 12/03/2013 09:03 AM, Alfredo Deza wrote:
>>
>> On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin 
>> wrote:
>>>
>>> Alfredo,
>>>
>>> I started that way, but I run into an error:
>>>
>>> $ ceph-deploy install baxter
>>> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
>>> baxter
>>> [ceph_deploy.install][DEBUG ] Installing stable version emperor on
>>> cluster
>>> ceph hosts baxter
>>> [ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
>>> [baxter][DEBUG ] connected to host: baxter
>>> [baxter][DEBUG ] detect platform information from remote host
>>> [baxter][DEBUG ] detect machine type
>>> [ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
>>> [baxter][INFO  ] installing ceph on baxter
>>> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
>>> apt-get -q install --assume-yes ca-certificates
>>> [baxter][DEBUG ] Reading package lists...
>>> [baxter][DEBUG ] Building dependency tree...
>>> [baxter][DEBUG ] Reading state information...
>>> [baxter][DEBUG ] ca-certificates is already the newest version.
>>> [baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
>>> upgraded.
>>> [baxter][INFO  ] Running command: sudo wget -O release.asc
>>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
>>> [baxter][WARNIN] --2013-12-03 16:32:45--
>>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
>>> [baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
>>> [baxter][WARNIN] Connecting to ceph.com
>>> (ceph.com)|208.113.241.137|:443...
>>> connected.
>>> [baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
>>> [baxter][WARNIN] Length: unspecified [text/plain]
>>> [baxter][WARNIN] Saving to: `release.asc'
>>> [baxter][WARNIN]
>>> [baxter][WARNIN]  0K .
>>> 1.06M=0.002s
>>> [baxter][WARNIN]
>>> [baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
>>> [1752]
>>> [baxter][WARNIN]
>>> [baxter][INFO  ] Running command: sudo apt-key add release.asc
>>> [baxter][DEBUG ] OK
>>> [baxter][DEBUG ] add ceph deb repo to sources.list
>>> [baxter][INFO  ] Running command: sudo apt-get -q update
>>> [baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy
>>> Release.gpg
>>> [490 B]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
>>> [baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
>>> [490 B]
>>> [baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
>>> [14.4 kB]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
>>> [baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
>>> [baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release
>>> [7224
>>> B]
>>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
>>> Packages
>>> [baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main
>>> armhf
>>> Packages [7414 kB]
>>> [baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
>>> Packages [12.1 kB]
>>> [baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
>>> [baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
>>> Translation-en
>>> [baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages [1010
>>> B]
>>> [baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib
>>> armhf
>>> Packages
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
>>> armhf Packages
>>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
>>> Packages
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
>>> Translation-en
>>> [baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
>>> Translation-en
>>> [baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
>>> [baxter][DEBUG ] Reading package lists...
>>> [baxter][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive
>>> DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
>>> --no-install-recommends --assume-yes install -- ceph ceph-mds ceph-common
>>> ce

Re: [ceph-users] Ceph on Raspberry Pi

[Shlomo Dubrowin]

OK,

So I'll need to do the installation manually, but the rest of the commands
I should run via ceph-deploy?  What version should I be trying to grab for
the manual compilation?  Should I be grabbing from git or is there a better
place?

  Shlomo

-
Shlomo Dubrowin

The Solution to the water crisis in Israel:

# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
  cat /sea/med /sea/red |\
  grep -vi "salt" |\
  tee /sea/dead /lake/kinneret
  LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done


On Tue, Dec 3, 2013 at 5:25 PM, Alfredo Deza wrote:

> On Tue, Dec 3, 2013 at 10:21 AM, Mark Nelson 
> wrote:
> > Guys, I don't think we have pre-released packages of anything new that is
> > going to work on the pi regardless if you use ceph-deploy.  Look at our
> > armhf packages file:
> >
> > http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages
> >
> > Unless I'm mistaken, you're going to have to compile it yourself.  I
> think
> > Joao was going to try that, not sure if he ever got around to it though.
>
> Oh, good point. I was just assuming that because we support the Distro
> the packages would exist.
>
> >
> > Mark
> >
> >
> > On 12/03/2013 09:03 AM, Alfredo Deza wrote:
> >>
> >> On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin 
> >> wrote:
> >>>
> >>> Alfredo,
> >>>
> >>> I started that way, but I run into an error:
> >>>
> >>> $ ceph-deploy install baxter
> >>> [ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
> >>> baxter
> >>> [ceph_deploy.install][DEBUG ] Installing stable version emperor on
> >>> cluster
> >>> ceph hosts baxter
> >>> [ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
> >>> [baxter][DEBUG ] connected to host: baxter
> >>> [baxter][DEBUG ] detect platform information from remote host
> >>> [baxter][DEBUG ] detect machine type
> >>> [ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
> >>> [baxter][INFO  ] installing ceph on baxter
> >>> [baxter][INFO  ] Running command: sudo env
> DEBIAN_FRONTEND=noninteractive
> >>> apt-get -q install --assume-yes ca-certificates
> >>> [baxter][DEBUG ] Reading package lists...
> >>> [baxter][DEBUG ] Building dependency tree...
> >>> [baxter][DEBUG ] Reading state information...
> >>> [baxter][DEBUG ] ca-certificates is already the newest version.
> >>> [baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
> >>> upgraded.
> >>> [baxter][INFO  ] Running command: sudo wget -O release.asc
> >>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
> >>> [baxter][WARNIN] --2013-12-03 16:32:45--
> >>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
> >>> [baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
> >>> [baxter][WARNIN] Connecting to ceph.com
> >>> (ceph.com)|208.113.241.137|:443...
> >>> connected.
> >>> [baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
> >>> [baxter][WARNIN] Length: unspecified [text/plain]
> >>> [baxter][WARNIN] Saving to: `release.asc'
> >>> [baxter][WARNIN]
> >>> [baxter][WARNIN]  0K .
> >>> 1.06M=0.002s
> >>> [baxter][WARNIN]
> >>> [baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
> >>> [1752]
> >>> [baxter][WARNIN]
> >>> [baxter][INFO  ] Running command: sudo apt-key add release.asc
> >>> [baxter][DEBUG ] OK
> >>> [baxter][DEBUG ] add ceph deb repo to sources.list
> >>> [baxter][INFO  ] Running command: sudo apt-get -q update
> >>> [baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy
> >>> Release.gpg
> >>> [490 B]
> >>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy
> Release.gpg
> >>> [baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy
> Release.gpg
> >>> [490 B]
> >>> [baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy
> Release
> >>> [14.4 kB]
> >>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
> >>> [baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
> >>> [baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release
> >>> [7224
> >>> B]
> >>> [baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
> >>> Packages
> >>> [baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main
> >>> armhf
> >>> Packages [7414 kB]
> >>> [baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main
> armhf
> >>> Packages [12.1 kB]
> >>> [baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
> >>> [baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
> >>> Translation-en
> >>> [baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages
> [1010
> >>> B]
> >>> [baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
> >>> Translation-en
> >>> [baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
> >>> [baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/contrib
> >>> armhf
> >>> 

Re: [ceph-users] Ceph on Raspberry Pi

[Joao Eduardo Luis]

On 12/03/2013 03:21 PM, Mark Nelson wrote:

Guys, I don't think we have pre-released packages of anything new that
is going to work on the pi regardless if you use ceph-deploy.  Look at
our armhf packages file:

http://ceph.com/debian-emperor/dists/wheezy/main/binary-armhf/Packages

Unless I'm mistaken, you're going to have to compile it yourself.  I
think Joao was going to try that, not sure if he ever got around to it
though.


I got as far as hitting some missing leveldb dependency.

Then decided to distcc and cross-compile Ceph using my desktop, but 
messed the toolchain somehow and have been attending to figuring out 
that every now and then but to no avail yet.


My best guess atm is that regardless of being able to compile it on the 
pi, we're going to hit some missing dependencies anyway rendering the 
whole task mighty annoying.  I'm hoping to be able to cross-compile a 
static version of the binaries and have a couple of monitors on the pi 
and an OSD on a cubietruck by Christmas :)


  -Joao



Mark

On 12/03/2013 09:03 AM, Alfredo Deza wrote:

On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin 
wrote:

Alfredo,

I started that way, but I run into an error:

$ ceph-deploy install baxter
[ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy install
baxter
[ceph_deploy.install][DEBUG ] Installing stable version emperor on
cluster
ceph hosts baxter
[ceph_deploy.install][DEBUG ] Detecting platform for host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote host
[baxter][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
[baxter][INFO  ] installing ceph on baxter
[baxter][INFO  ] Running command: sudo env
DEBIAN_FRONTEND=noninteractive
apt-get -q install --assume-yes ca-certificates
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][DEBUG ] ca-certificates is already the newest version.
[baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 85 not
upgraded.
[baxter][INFO  ] Running command: sudo wget -O release.asc
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] --2013-12-03 16:32:45--
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
[baxter][WARNIN] Resolving ceph.com (ceph.com)... 208.113.241.137
[baxter][WARNIN] Connecting to ceph.com
(ceph.com)|208.113.241.137|:443...
connected.
[baxter][WARNIN] HTTP request sent, awaiting response... 200 OK
[baxter][WARNIN] Length: unspecified [text/plain]
[baxter][WARNIN] Saving to: `release.asc'
[baxter][WARNIN]
[baxter][WARNIN]  0K .
1.06M=0.002s
[baxter][WARNIN]
[baxter][WARNIN] 2013-12-03 16:32:53 (1.06 MB/s) - `release.asc' saved
[1752]
[baxter][WARNIN]
[baxter][INFO  ] Running command: sudo apt-key add release.asc
[baxter][DEBUG ] OK
[baxter][DEBUG ] add ceph deb repo to sources.list
[baxter][INFO  ] Running command: sudo apt-get -q update
[baxter][DEBUG ] Get:1 http://mirrordirector.raspbian.org wheezy
Release.gpg
[490 B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release.gpg
[baxter][DEBUG ] Get:2 http://archive.raspberrypi.org wheezy Release.gpg
[490 B]
[baxter][DEBUG ] Get:3 http://mirrordirector.raspbian.org wheezy Release
[14.4 kB]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy Release
[baxter][DEBUG ] Get:4 http://ceph.com wheezy Release.gpg [836 B]
[baxter][DEBUG ] Get:5 http://archive.raspberrypi.org wheezy Release
[7224
B]
[baxter][DEBUG ] Hit http://raspberrypi.collabora.com wheezy/rpi armhf
Packages
[baxter][DEBUG ] Get:6 http://mirrordirector.raspbian.org wheezy/main
armhf
Packages [7414 kB]
[baxter][DEBUG ] Get:7 http://archive.raspberrypi.org wheezy/main armhf
Packages [12.1 kB]
[baxter][DEBUG ] Get:8 http://ceph.com wheezy Release [5984 B]
[baxter][DEBUG ] Ign http://raspberrypi.collabora.com wheezy/rpi
Translation-en
[baxter][DEBUG ] Get:9 http://ceph.com wheezy/main armhf Packages
[1010 B]
[baxter][DEBUG ] Ign http://archive.raspberrypi.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://ceph.com wheezy/main Translation-en
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org
wheezy/contrib armhf
Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/non-free
armhf Packages
[baxter][DEBUG ] Hit http://mirrordirector.raspbian.org wheezy/rpi armhf
Packages
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/contrib
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/main
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/non-free
Translation-en
[baxter][DEBUG ] Ign http://mirrordirector.raspbian.org wheezy/rpi
Translation-en
[baxter][DEBUG ] Fetched 7456 kB in 47s (157 kB/s)
[baxter][DEBUG ] Reading package lists...
[baxter][INFO  ] Running command: sudo env
DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical apt-get -q -o Dpkg::Options::=--force-confnew
--no-i

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Robert van Leeuwen]

Hi Mike,

I am using filebench within a kvm virtual. (Like an actual workload we will 
have)
Using 100% synchronous 4k writes with a 50GB file on a 100GB volume with 32 
writer threads.
Also tried from multiple KVM machines from multiple hosts. 
Aggregated performance keeps at 2k+ IOPS

The disks are 7200RPM 2.5 inch drives, no RAID whatsoever.
I agree the amount of IOPS seem high. 
Maybe the journal on SSD (2 x Intel 3500) helps a bit in this regard but the 
SSD's where not maxed out yet.
The writes seem to be limited by the spinning disks:
As soon as the benchmark starts the are used for 100% percent.
Also the usage dropped to 0% pretty much immediately after the benchmark so it 
looks like it's not lagging behind the journal.

Did not really test reads yet since we have so much read cache (128 GB per 
node) I assume we will mostly be write limited.

Cheers,
Robert van Leeuwen



Sent from my iPad

> On 3 dec. 2013, at 16:15, "Mike Dawson"  wrote:
> 
> Robert,
> 
> Interesting results on the effect of # of PG/PGPs. My cluster struggles a bit 
> under the strain of heavy random small-sized writes.
> 
> The IOPS you mention seem high to me given 30 drives and 3x replication 
> unless they were pure reads or on high-rpm drives. Instead of assuming, I 
> want to pose a few questions:
> 
> - How are you testing? rados bench, rbd bench, rbd bench with writeback 
> cache, etc?
> 
> - Were the 2000-2500 random 4k IOPS more reads than writes? If you test 100% 
> 4k random reads, what do you get? If you test 100% 4k random writes, what do 
> you get?
> 
> - What drives do you have? Any RAID involved under your OSDs?
> 
> Thanks,
> Mike Dawson
> 
> 
>> On 12/3/2013 1:31 AM, Robert van Leeuwen wrote:
>> 
 On 2 dec. 2013, at 18:26, "Brian Andrus"  wrote:
>>> 
>>>  Setting your pg_num and pgp_num to say... 1024 would A) increase data 
>>> granularity, B) likely lend no noticeable increase to resource consumption, 
>>> and C) allow some room for future OSDs two be added while still within 
>>> range of acceptable pg numbers. You could probably safely double even that 
>>> number if you plan on expanding at a rapid rate and want to avoid splitting 
>>> PGs every time a node is added.
>>> 
>>> In general, you can conservatively err on the larger side when it comes to 
>>> pg/p_num. Any excess resource utilization will be negligible (up to a 
>>> certain point). If you have a comfortable amount of available RAM, you 
>>> could experiment with increasing the multiplier in the equation you are 
>>> using and see how it affects your final number.
>>> 
>>> The pg_num and pgp_num parameters can safely be changed before or after 
>>> your new nodes are integrated.
>> 
>> I would be a bit conservative with the PGs / PGPs.
>> I've experimented with the PG number a bit and noticed the following random 
>> IO performance drop.
>> ( this could be something to our specific setup but since the PG is easily 
>> increased and impossible to decrease I would be conservative)
>> 
>>  The setup:
>> 3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
>> Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.
>> 
>> We use a replica count of 3 so optimum according to formula is about 1000
>> With 1000 PGs I got about 2000-2500 random 4k IOPS.
>> 
>> Because the nodes are fast enough and I expect the cluster to be expanded 
>> with 3 more nodes I set the PGs to 2000.
>> Performance dropped to about 1200-1400 IOPS.
>> 
>> I noticed that the spinning disks where no longer maxing out on 100% usage.
>> Memory and CPU did not seem to be a problem.
>> Since had the option to recreate the pool and I was not using the 
>> recommended settings I did not really dive into the issue.
>> I will not stray to far from the recommended settings in the future though :)
>> 
>> Cheers,
>> Robert van Leeuwen
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal partition and trim

[James Pearce]

It shouldn't happen, provided the sustained write rate doesn't exceed 
the sustained erase capabilities of the device I guess.  Daily fstrim 
will not hurt though.


Essentially the mapping between LBAs and physical cells isn't 
persistent in SSDs (unlike LBA and physical sectors on an HDD).  
Provided the SSD has a free cell, a write is committed to an available 
(pre-charged) cell and the mapping table updated.  In the process, the 
cell that contained the now 'overwritten' (from the OS perspective) data 
is marked as part of the free pool, so cleared in the background ready 
to accept some future write.


Under-provisioning maintain this mode of operation since LBAs that have 
never had a write (at least since a TRIM operation) will have no 
physical backing, i.e. cells will be free for the controller to use in 
the background.


Some reasonable background info here: 
http://en.wikipedia.org/wiki/Write_amplification


Hope that helps.

On 2013-12-03 15:10, Loic Dachary wrote:

On 03/12/2013 13:38, James Pearce wrote:
Since the journal partitions are generally small, it shouldn't need 
to be.


For example implement with substantial under-provisioning, either 
via HPA or simple partitions.




Does that mean the problem will happen much later or that it will
never happen ? As far as I understand it is just postponing it but I
just discover this and may be completely mistaken :-)

Cheers


On 2013-12-03 12:18, Loic Dachary wrote:

Hi Ceph,

When an SSD partition is used to store a journal

https://github.com/ceph/ceph/blob/master/src/os/FileJournal.cc#L90

how is it trimmed ?

http://en.wikipedia.org/wiki/Trim_%28computing%29

Cheers

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Mike Dawson]

Robert,

Do you have rbd writeback cache enabled on these volumes? That could 
certainly explain the higher than expected write performance. Any chance 
you could re-test with rbd writeback on vs. off?


Thanks,
Mike Dawson

On 12/3/2013 10:37 AM, Robert van Leeuwen wrote:

Hi Mike,

I am using filebench within a kvm virtual. (Like an actual workload we will 
have)
Using 100% synchronous 4k writes with a 50GB file on a 100GB volume with 32 
writer threads.
Also tried from multiple KVM machines from multiple hosts.
Aggregated performance keeps at 2k+ IOPS

The disks are 7200RPM 2.5 inch drives, no RAID whatsoever.
I agree the amount of IOPS seem high.
Maybe the journal on SSD (2 x Intel 3500) helps a bit in this regard but the 
SSD's where not maxed out yet.
The writes seem to be limited by the spinning disks:
As soon as the benchmark starts the are used for 100% percent.
Also the usage dropped to 0% pretty much immediately after the benchmark so it 
looks like it's not lagging behind the journal.

Did not really test reads yet since we have so much read cache (128 GB per 
node) I assume we will mostly be write limited.

Cheers,
Robert van Leeuwen



Sent from my iPad


On 3 dec. 2013, at 16:15, "Mike Dawson"  wrote:

Robert,

Interesting results on the effect of # of PG/PGPs. My cluster struggles a bit 
under the strain of heavy random small-sized writes.

The IOPS you mention seem high to me given 30 drives and 3x replication unless 
they were pure reads or on high-rpm drives. Instead of assuming, I want to pose 
a few questions:

- How are you testing? rados bench, rbd bench, rbd bench with writeback cache, 
etc?

- Were the 2000-2500 random 4k IOPS more reads than writes? If you test 100% 4k 
random reads, what do you get? If you test 100% 4k random writes, what do you 
get?

- What drives do you have? Any RAID involved under your OSDs?

Thanks,
Mike Dawson



On 12/3/2013 1:31 AM, Robert van Leeuwen wrote:


On 2 dec. 2013, at 18:26, "Brian Andrus"  wrote:


  Setting your pg_num and pgp_num to say... 1024 would A) increase data 
granularity, B) likely lend no noticeable increase to resource consumption, and 
C) allow some room for future OSDs two be added while still within range of 
acceptable pg numbers. You could probably safely double even that number if you 
plan on expanding at a rapid rate and want to avoid splitting PGs every time a 
node is added.

In general, you can conservatively err on the larger side when it comes to 
pg/p_num. Any excess resource utilization will be negligible (up to a certain 
point). If you have a comfortable amount of available RAM, you could experiment 
with increasing the multiplier in the equation you are using and see how it 
affects your final number.

The pg_num and pgp_num parameters can safely be changed before or after your 
new nodes are integrated.


I would be a bit conservative with the PGs / PGPs.
I've experimented with the PG number a bit and noticed the following random IO 
performance drop.
( this could be something to our specific setup but since the PG is easily 
increased and impossible to decrease I would be conservative)

  The setup:
3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.

We use a replica count of 3 so optimum according to formula is about 1000
With 1000 PGs I got about 2000-2500 random 4k IOPS.

Because the nodes are fast enough and I expect the cluster to be expanded with 
3 more nodes I set the PGs to 2000.
Performance dropped to about 1200-1400 IOPS.

I noticed that the spinning disks where no longer maxing out on 100% usage.
Memory and CPU did not seem to be a problem.
Since had the option to recreate the pool and I was not using the recommended 
settings I did not really dive into the issue.
I will not stray to far from the recommended settings in the future though :)

Cheers,
Robert van Leeuwen
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw on Ubuntu vs CentOS

[Sage Weil]

On Tue, 3 Dec 2013, Andy McCrae wrote:
> Hi ceph-users,
> I've been playing around with radosgw and I notice there is an inconsistency
> between the Ubuntu and CentOS startup scripts.
> 
> On Ubuntu, if I run a start ceph-all (which will start radosgw), or I run
> the init script /etc/init.d/radosgw start - the radosgw process starts up
> fine, but running as root.
> 
> On CentOS the init script starts radosgw as the "apache" user by default.
> 
> I can see the Ubuntu init script is specifying "www-data" which would be in
> keeping with the CentOS init script, but the process runs as root.
> 
> + start-stop-daemon --start -u www-data -x /usr/bin/radosgw -- -n
> client.radosgw.ubunTest
> 2013-12-03 15:13:26.449087 7fee1d33b780 -1 WARNING: libcurl doesn't support
> curl_multi_wait()
> 2013-12-03 15:13:26.449093 7fee1d33b780 -1 WARNING: cross zone / region
> transfer performance may be affected
> root@ubunTest:~# ps -ef | grep radosgw
> root     28528     1  0 15:13 ?        00:00:00 /usr/bin/radosgw -n
> client.radosgw.ubunTest
> 
> 
> The question is, do we consider this a bug in that radosgw shouldn't run as
> root by default, or should the CentOS/RHEL (rpm) init scripts start radosgw
> as root - I'd assume the former.

I think it's a bug.  There is no real need for radosgw to run as root, 
except that it needs to log to /var/log/radosgw/*.  We should update the 
packaging (rpm and deb) to create a radosgw user (and/or a ceph group?) 
and then make the two environments behave consistently.

Anyone with strong opinions in this area interested?

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw on Ubuntu vs CentOS

[Andy McCrae]

Tracked it down - its the start-stop-daemon command flag, -u isn't for user
change, that should be -c, so I'll submit a fix soon.


On Tue, Dec 3, 2013 at 4:06 PM, Sage Weil  wrote:

> On Tue, 3 Dec 2013, Andy McCrae wrote:
> > Hi ceph-users,
> > I've been playing around with radosgw and I notice there is an
> inconsistency
> > between the Ubuntu and CentOS startup scripts.
> >
> > On Ubuntu, if I run a start ceph-all (which will start radosgw), or I run
> > the init script /etc/init.d/radosgw start - the radosgw process starts up
> > fine, but running as root.
> >
> > On CentOS the init script starts radosgw as the "apache" user by default.
> >
> > I can see the Ubuntu init script is specifying "www-data" which would be
> in
> > keeping with the CentOS init script, but the process runs as root.
> >
> > + start-stop-daemon --start -u www-data -x /usr/bin/radosgw -- -n
> > client.radosgw.ubunTest
> > 2013-12-03 15:13:26.449087 7fee1d33b780 -1 WARNING: libcurl doesn't
> support
> > curl_multi_wait()
> > 2013-12-03 15:13:26.449093 7fee1d33b780 -1 WARNING: cross zone / region
> > transfer performance may be affected
> > root@ubunTest:~# ps -ef | grep radosgw
> > root 28528 1  0 15:13 ?00:00:00 /usr/bin/radosgw -n
> > client.radosgw.ubunTest
> >
> >
> > The question is, do we consider this a bug in that radosgw shouldn't run
> as
> > root by default, or should the CentOS/RHEL (rpm) init scripts start
> radosgw
> > as root - I'd assume the former.
>
> I think it's a bug.  There is no real need for radosgw to run as root,
> except that it needs to log to /var/log/radosgw/*.  We should update the
> packaging (rpm and deb) to create a radosgw user (and/or a ceph group?)
> and then make the two environments behave consistently.
>
> Anyone with strong opinions in this area interested?
>
> sage
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding new OSDs, need to increase PGs?

[Robert van Leeuwen]

Not specifically set, according to the docs the default is off...

I am using the async qemu cuttlefish rpm. 
Maybe it does cache something but I think not:
Specifically setting writeback on in the client config yielded different 
results:
In our DEV environment we had issues with the virtual becoming unreachable 
under heavy IO load so I have not enabled it on the live machines..

I am currently at home but I could run a few tests tomorrow at work.
Is there an way to do small random IO write tests with the rados tools?
I just gave it a quick glance and it looked like it just did large sequential 
writes.

Cheers,
Robert van Leeuwen




Sent from my iPad

> On 3 dec. 2013, at 17:02, "Mike Dawson"  wrote:
> 
> Robert,
> 
> Do you have rbd writeback cache enabled on these volumes? That could 
> certainly explain the higher than expected write performance. Any chance you 
> could re-test with rbd writeback on vs. off?
> 
> Thanks,
> Mike Dawson
> 
>> On 12/3/2013 10:37 AM, Robert van Leeuwen wrote:
>> Hi Mike,
>> 
>> I am using filebench within a kvm virtual. (Like an actual workload we will 
>> have)
>> Using 100% synchronous 4k writes with a 50GB file on a 100GB volume with 32 
>> writer threads.
>> Also tried from multiple KVM machines from multiple hosts.
>> Aggregated performance keeps at 2k+ IOPS
>> 
>> The disks are 7200RPM 2.5 inch drives, no RAID whatsoever.
>> I agree the amount of IOPS seem high.
>> Maybe the journal on SSD (2 x Intel 3500) helps a bit in this regard but the 
>> SSD's where not maxed out yet.
>> The writes seem to be limited by the spinning disks:
>> As soon as the benchmark starts the are used for 100% percent.
>> Also the usage dropped to 0% pretty much immediately after the benchmark so 
>> it looks like it's not lagging behind the journal.
>> 
>> Did not really test reads yet since we have so much read cache (128 GB per 
>> node) I assume we will mostly be write limited.
>> 
>> Cheers,
>> Robert van Leeuwen
>> 
>> 
>> 
>> Sent from my iPad
>> 
>>> On 3 dec. 2013, at 16:15, "Mike Dawson"  wrote:
>>> 
>>> Robert,
>>> 
>>> Interesting results on the effect of # of PG/PGPs. My cluster struggles a 
>>> bit under the strain of heavy random small-sized writes.
>>> 
>>> The IOPS you mention seem high to me given 30 drives and 3x replication 
>>> unless they were pure reads or on high-rpm drives. Instead of assuming, I 
>>> want to pose a few questions:
>>> 
>>> - How are you testing? rados bench, rbd bench, rbd bench with writeback 
>>> cache, etc?
>>> 
>>> - Were the 2000-2500 random 4k IOPS more reads than writes? If you test 
>>> 100% 4k random reads, what do you get? If you test 100% 4k random writes, 
>>> what do you get?
>>> 
>>> - What drives do you have? Any RAID involved under your OSDs?
>>> 
>>> Thanks,
>>> Mike Dawson
>>> 
>>> 
 On 12/3/2013 1:31 AM, Robert van Leeuwen wrote:
 
>> On 2 dec. 2013, at 18:26, "Brian Andrus"  
>> wrote:
> 
>  Setting your pg_num and pgp_num to say... 1024 would A) increase data 
> granularity, B) likely lend no noticeable increase to resource 
> consumption, and C) allow some room for future OSDs two be added while 
> still within range of acceptable pg numbers. You could probably safely 
> double even that number if you plan on expanding at a rapid rate and want 
> to avoid splitting PGs every time a node is added.
> 
> In general, you can conservatively err on the larger side when it comes 
> to pg/p_num. Any excess resource utilization will be negligible (up to a 
> certain point). If you have a comfortable amount of available RAM, you 
> could experiment with increasing the multiplier in the equation you are 
> using and see how it affects your final number.
> 
> The pg_num and pgp_num parameters can safely be changed before or after 
> your new nodes are integrated.
 
 I would be a bit conservative with the PGs / PGPs.
 I've experimented with the PG number a bit and noticed the following 
 random IO performance drop.
 ( this could be something to our specific setup but since the PG is easily 
 increased and impossible to decrease I would be conservative)
 
  The setup:
 3 OSD nodes with 128GB ram, 2 * 6 core CPU (12 with ht).
 Nodes have 10 OSDs running on 1 tb disks and 2 SSDs for Journals.
 
 We use a replica count of 3 so optimum according to formula is about 1000
 With 1000 PGs I got about 2000-2500 random 4k IOPS.
 
 Because the nodes are fast enough and I expect the cluster to be expanded 
 with 3 more nodes I set the PGs to 2000.
 Performance dropped to about 1200-1400 IOPS.
 
 I noticed that the spinning disks where no longer maxing out on 100% usage.
 Memory and CPU did not seem to be a problem.
 Since had the option to recreate the pool and I was not using the 
 recommended settings I did not really dive into the issue.
 I will not stray to far from the recommen

[ceph-users] Openstack+ceph volume mounting to vm

[Karan Singh]

Hello Cephers 

Need your guidance 

In my setup ceph cluster and openstack are working good , i am able to create 
volumes using cinder as well. 

What i want is to mount ceph volume to VM instance. But getting deadly errors 
like this . Expecting your help in this 




[root@rdo /(keystone_admin)]# virsh attach-device instance-0018 disk.xml 
error: Failed to attach device from disk.xml 
error: internal error unable to execute QEMU command '__com.redhat_drive_add': 
Device 'drive-virtio-disk5' could not be initialized 

[root@rdo /(keystone_admin)]# 







My Setup details :- 




[root@rdo /(keystone_admin)]# rpm -qa | grep -i qemu 
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
gpxe-roms-qemu-0.9.7-6.10.el6.noarch 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# uname -a 
Linux rdo 3.10.18-1.el6.elrepo.x86_64 #1 SMP Mon Nov 4 19:12:54 EST 2013 x86_64 
x86_64 x86_64 GNU/Linux 
[root@rdo /(keystone_admin)]# 
[root@rdo /(keystone_admin)]# cat /etc/redhat-release 
CentOS release 6.5 (Final) 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# cinder list 
+--+---+--+--+--+--+-+
 
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | 
+--+---+--+--+--+--+-+
 
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | 
ceph-storage | false | | 
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | 
None | false | | 
+--+---+--+--+--+--+-+
 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# ceph status 
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a 
health HEALTH_OK 
monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 30, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 
osdmap e157: 11 osds: 11 up, 11 in 
pgmap v12102: 448 pgs: 448 active+clean; 135 GB data, 272 GB used, 5935 GB / 
6207 GB avail 
mdsmap e27: 1/1/1 up {0=ceph-mon1=up:active} 

[root@rdo /(keystone_admin)]# 





[root@rdo /(keystone_admin)]# cat disk.xml 
 
 
 
 
 
 
 
 
 
 
 
 
[root@rdo /(keystone_admin)]# 




[root@rdo /(keystone_admin)]# service libvirtd status 
libvirtd (pid 17947) is running... 
[root@rdo /(keystone_admin)]# 







virsh # list 
Id Name State 
 
2 instance-0018 running 

virsh # 





Karan Singh 
Systems Specialist, Computing Environments Group 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph on Raspberry Pi

[Joao Eduardo Luis]

On 12/03/2013 03:38 PM, Shlomo Dubrowin wrote:

Joao,

Is there a reason you aren't putting OSDs on the Pis? Do you expect that
the OSD won't run on the Pi?


The question really isn't about why I'm not putting the OSDs on the pi's 
but why I'd prefer to put them on the Cubietruck: unlike the pi, 
cubietruck has a SATA connection.


Besides, yeah, the Pi doesn't have much RAM anyway, and although I'm 
doing it just for the kicks of it and don't aim at a performing cluster, 
I don't see the 512MB of RAM to be enough for a mon, an OSD and the 
overall OS.


  -Joao



   Shlomo

-
Shlomo Dubrowin

The Solution to the water crisis in Israel:

# According to WikiPedia, the Kinneret can hold
# 4 km^3, so FULL here is in cubit meters
FULL="4000"
while [ "$LEVEL" -lt "$FULL" ]; do
   cat /sea/med /sea/red |\
   grep -vi "salt" |\
   tee /sea/dead /lake/kinneret
   LEVEL=`du -c /sea/dead /lake/kinneret | grep total | awk '{print $1}'`
done


On Tue, Dec 3, 2013 at 5:33 PM, Joao Eduardo Luis mailto:joao.l...@inktank.com>> wrote:

On 12/03/2013 03:21 PM, Mark Nelson wrote:

Guys, I don't think we have pre-released packages of anything
new that
is going to work on the pi regardless if you use ceph-deploy.
  Look at
our armhf packages file:


http://ceph.com/debian-__emperor/dists/wheezy/main/__binary-armhf/Packages


Unless I'm mistaken, you're going to have to compile it yourself.  I
think Joao was going to try that, not sure if he ever got around
to it
though.


I got as far as hitting some missing leveldb dependency.

Then decided to distcc and cross-compile Ceph using my desktop, but
messed the toolchain somehow and have been attending to figuring out
that every now and then but to no avail yet.

My best guess atm is that regardless of being able to compile it on
the pi, we're going to hit some missing dependencies anyway
rendering the whole task mighty annoying.  I'm hoping to be able to
cross-compile a static version of the binaries and have a couple of
monitors on the pi and an OSD on a cubietruck by Christmas :)

   -Joao



Mark

On 12/03/2013 09:03 AM, Alfredo Deza wrote:

On Tue, Dec 3, 2013 at 9:56 AM, Shlomo Dubrowin
mailto:shl...@dubrowin.org>>
wrote:

Alfredo,

I started that way, but I run into an error:

$ ceph-deploy install baxter
[ceph_deploy.cli][INFO  ] Invoked (1.3.3):
/usr/bin/ceph-deploy install
baxter
[ceph_deploy.install][DEBUG ] Installing stable version
emperor on
cluster
ceph hosts baxter
[ceph_deploy.install][DEBUG ] Detecting platform for
host baxter ...
[baxter][DEBUG ] connected to host: baxter
[baxter][DEBUG ] detect platform information from remote
host
[baxter][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: debian 7.0 wheezy
[baxter][INFO  ] installing ceph on baxter
[baxter][INFO  ] Running command: sudo env
DEBIAN_FRONTEND=noninteractive
apt-get -q install --assume-yes ca-certificates
[baxter][DEBUG ] Reading package lists...
[baxter][DEBUG ] Building dependency tree...
[baxter][DEBUG ] Reading state information...
[baxter][DEBUG ] ca-certificates is already the newest
version.
[baxter][DEBUG ] 0 upgraded, 0 newly installed, 0 to
remove and 85 not
upgraded.
[baxter][INFO  ] Running command: sudo wget -O release.asc

https://ceph.com/git/?p=ceph.__git;a=blob_plain;f=keys/__release.asc


[baxter][WARNIN] --2013-12-03 16:32:45--

https://ceph.com/git/?p=ceph.__git;a=blob_plain;f=keys/__release.asc


[baxter][WARNIN] Resolving ceph.com 
(ceph.com )... 208.113.241.137
[baxter][WARNIN] Connecting to ceph.com 
(ceph.com )|208.113.241.137|:__443...
connected.
[baxter][WARNIN] HTTP request sent, awaiting response...
200 OK
[baxter][WARNIN] Length: unspecified [text/plain]
[baxter][WARNIN] Saving to: `release.asc'
[baxter][WARNIN]
[baxter][WARNIN]  0K .
  

Re: [ceph-users] optimal setup with 4 x ethernet ports

[Scottix]

I found network to be the most limiting factor in Ceph.
Any chance to move to 10G+ would be beneficial.
I did have success with Bonding and just doing a simple RR increased the
throughput.


On Mon, Dec 2, 2013 at 10:17 PM, Kyle Bader  wrote:

> > Is having two cluster networks like this a supported configuration?
> Every osd and mon can reach every other so I think it should be.
>
> Maybe. If your back end network is a supernet and each cluster network is
> a subnet of that supernet. For example:
>
> Ceph.conf cluster network (supernet): 10.0.0.0/8
>
> Cluster network #1:  10.1.1.0/24
> Cluster network #2: 10.1.2.0/24
>
> With that configuration OSD address autodection *should* just work.
>
> > 1. move osd traffic to eth1. This obviously limits maximum throughput to
> ~100Mbytes/second, but I'm getting nowhere near that right now anyway.
>
> Given three links I would probably do this if your replication factor is
> >= 3. Keep in mind 100Mbps links could very well end up being a limiting
> factor.
>
> What are you backing each OSD with storage wise and how many OSDs do you
> expect to participate in this cluster?
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Follow Me: @Scottix 
http://about.me/scottix
scot...@gmail.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to fix active+remapped pg

[Gregory Farnum]

CRUSH is failing to map all the PGs to the right number of OSDs.
You've got a completely empty host which has ~1/3 of the cluster's
total weight, and that is probably why — remove it!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Tue, Dec 3, 2013 at 3:13 AM, Ugis  wrote:
> Hi,
> Upgaded to emperor, restarted all nodes.
>
> Still have "31 actige+remapped" pgs.
>
> Compared remapped and healthy pg query output - some remapped pgs do
> not have data, some do, some have been scrubbed some don't. Now
> running read for whole rbd - may be that would trigger those stuck
> pgs.
>
> state on remapped pgs like:
> { "state": "active+remapped",
>   "epoch": 9420,
>   "up": [
> 9],
>   "acting": [
> 9,
> 5],
>
> Any help/hints how to trigger those stuck pgs to up state on 2 osds?
>
> Ugis
>
>
> 2013/11/22 Ugis :
>> Update: I noticed that I hadn't increased pgp_num for default data
>> pool for which I increased pg_num time ago. So I did now and some
>> backfilling happened.
>> Now I still have "31 actige+remapped" pgs.
>> Remapped pgs belong to all pools, even those where is no data.
>> To me suspicious is that host ceph8 has weight 10.88(I had some osds
>> there temporarily, but due to low ram I remover those)
>> If that is of importance ceph7 is also low on ram(4GB) and is slower
>> to respond at times than ceph5(Sage mentioned "lagging pg peering
>> workqueue" in Bug#3747).
>>
>> Results follow:
>> # ceph osd tree
>> # idweight  type name   up/down reweight
>> -5  0   root slow
>> -4  0   host ceph5-slow
>> -1  32.46   root default
>> -2  10.5host ceph5
>> 0   0.2 osd.0   up  0
>> 2   2.8 osd.2   up  1
>> 3   2.8 osd.3   up  1
>> 4   1.9 osd.4   up  1
>> 5   2.8 osd.5   up  1
>> -3  0.2 host ceph6
>> 1   0.2 osd.1   up  0
>> -6  10.88   host ceph7
>> 6   2.73osd.6   up  1
>> 7   2.73osd.7   up  1
>> 8   2.71osd.8   up  1
>> 9   2.71osd.9   up  1
>> -7  10.88   host ceph8
>>
>> # ceph osd crush dump
>> { "devices": [
>> { "id": 0,
>>   "name": "osd.0"},
>> { "id": 1,
>>   "name": "osd.1"},
>> { "id": 2,
>>   "name": "osd.2"},
>> { "id": 3,
>>   "name": "osd.3"},
>> { "id": 4,
>>   "name": "osd.4"},
>> { "id": 5,
>>   "name": "osd.5"},
>> { "id": 6,
>>   "name": "osd.6"},
>> { "id": 7,
>>   "name": "osd.7"},
>> { "id": 8,
>>   "name": "osd.8"},
>> { "id": 9,
>>   "name": "osd.9"}],
>>   "types": [
>> { "type_id": 0,
>>   "name": "osd"},
>> { "type_id": 1,
>>   "name": "host"},
>> { "type_id": 2,
>>   "name": "rack"},
>> { "type_id": 3,
>>   "name": "row"},
>> { "type_id": 4,
>>   "name": "room"},
>> { "type_id": 5,
>>   "name": "datacenter"},
>> { "type_id": 6,
>>   "name": "root"}],
>>   "buckets": [
>> { "id": -1,
>>   "name": "default",
>>   "type_id": 6,
>>   "type_name": "root",
>>   "weight": 2127297,
>>   "alg": "straw",
>>   "hash": "rjenkins1",
>>   "items": [
>> { "id": -2,
>>   "weight": 688128,
>>   "pos": 0},
>> { "id": -3,
>>   "weight": 13107,
>>   "pos": 1},
>> { "id": -6,
>>   "weight": 713031,
>>   "pos": 2},
>> { "id": -7,
>>   "weight": 713031,
>>   "pos": 3}]},
>> { "id": -2,
>>   "name": "ceph5",
>>   "type_id": 1,
>>   "type_name": "host",
>>   "weight": 688125,
>>   "alg": "straw",
>>   "hash": "rjenkins1",
>>   "items": [
>> { "id": 0,
>>   "weight": 13107,
>>   "pos": 0},
>> { "id": 2,
>>   "weight": 183500,
>>   "pos": 1},
>> { "id": 3,
>>   "weight": 183500,
>>   "pos": 2},
>> { "id": 4,
>>   "weight": 124518,
>>   "pos": 3},
>> { "id": 5,
>>   "weight": 183500,
>>   "pos": 4}]},
>> { "id": -3,
>>   "name": "ceph6",
>>   "type_id": 1,
>>   "type_name": "host",
>>   "weight": 13107,
>>   "alg": "straw",
>>   "hash": "rjenkins1",
>>   "items": [
>> { "id": 1,
>>   "weig

Re: [ceph-users] SSD journal partition and trim

[Loic Dachary]

Crystal clear, thanks for the tutorial :-)

On 03/12/2013 16:52, James Pearce wrote:
> It shouldn't happen, provided the sustained write rate doesn't exceed the 
> sustained erase capabilities of the device I guess.  Daily fstrim will not 
> hurt though.
> 
> Essentially the mapping between LBAs and physical cells isn't persistent in 
> SSDs (unlike LBA and physical sectors on an HDD).  Provided the SSD has a 
> free cell, a write is committed to an available (pre-charged) cell and the 
> mapping table updated.  In the process, the cell that contained the now 
> 'overwritten' (from the OS perspective) data is marked as part of the free 
> pool, so cleared in the background ready to accept some future write.
> 
> Under-provisioning maintain this mode of operation since LBAs that have never 
> had a write (at least since a TRIM operation) will have no physical backing, 
> i.e. cells will be free for the controller to use in the background.
> 
> Some reasonable background info here: 
> http://en.wikipedia.org/wiki/Write_amplification
> 
> Hope that helps.
> 
> On 2013-12-03 15:10, Loic Dachary wrote:
>> On 03/12/2013 13:38, James Pearce wrote:
>>> Since the journal partitions are generally small, it shouldn't need to be.
>>>
>>> For example implement with substantial under-provisioning, either via HPA 
>>> or simple partitions.
>>>
>>
>> Does that mean the problem will happen much later or that it will
>> never happen ? As far as I understand it is just postponing it but I
>> just discover this and may be completely mistaken :-)
>>
>> Cheers
>>
>>> On 2013-12-03 12:18, Loic Dachary wrote:
 Hi Ceph,

 When an SSD partition is used to store a journal

 https://github.com/ceph/ceph/blob/master/src/os/FileJournal.cc#L90

 how is it trimmed ?

 http://en.wikipedia.org/wiki/Trim_%28computing%29

 Cheers

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-kvm packages for centos

[Alvin Starr]

I got them and they appear to work.
Thanks.


On 12/02/2013 04:21 PM, Dan Van Der Ster wrote:


You need the rhev package.

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg05962.html

On Dec 2, 2013 10:18 PM, Alvin Starr  wrote:
I have been looking at the src rpm qemu-img-0.12.1.2-2.415.el6.x86_64
It looks like it there are a number of comments implying that it supprts
rdb but

qemu -h returns.
Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2
qed vhdx parallels nbd blkdebug host_cdrom host_floppy host_device file
gluster gluster gluster gluster

I most of all like the gluster gluster gluster gluster Is that a 
hint??


I am trying to track down if rdb is actually enabled.
when/if I figure it out I will post it to the list.


  On 12/02/2013 10:46 AM, Dan Van Der Ster wrote:
> Hi,
> See this one also: http://tracker.ceph.com/issues/6365
> But I’m not sure the Inktank patched qemu-kvm is relevant any longer 
since RedHat just released qemu-kvm-rhev with RBD support.

> Cheers, Dan
>
> On 02 Dec 2013, at 15:36, Darren Birkett  
wrote:

>
>> Hi List,
>>
>> Any chance the following will be updated with the latest packages 
for dumpling/emperor:

>>
>> http://ceph.com/packages/qemu-kvm/centos/x86_64/
>>
>> Using CentOS 6.4 and dumpling with OpenStack Havana, I am unable to 
boot from rbd volumes until I install an rbd-ified qemu-kvm.  I have 
grabbed the latest (cuttlefish) and it seems to work ok, but I can't 
be 100% sure.

>>
>> I noted the following ticket where the original packaging work was 
done:

>>
>> http://tracker.ceph.com/issues/4834
>>
>> ...but nothing seems to have been done since.
>>
>> Also, is this the official place for that package to live?  I'm in 
the process of working with the ceph chef cookbooks, as well as our 
own openstack cookbooks, and want to make sure the package is pulled 
from the right place.

>>
>> Thanks
>> Darren
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||




--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph website for rpm pacakges is down?

[Gautam Saxena]

In trying to download the RPM packages for CEPH, the yum commands timed
out. I then tried just downloading them via Chrome browser (
http://ceph.com/rpm-emperor/el6/x86_64/ceph-0.72.1-0.el6.x86_64.rpm) and it
only downloaded 64KB. (The website www.ceph.com is slow too)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph website for rpm pacakges is down?

[Gautam Saxena]

The website is repsonding quickly now, but I'm getting very strange error
of:

===
Downloaded more than max size for
http://ceph.com/rpm-emperor/el6/x86_64/ceph-0.72.1-0.el6.x86_64.rpm:
15221737 > 13835720.


 I didn't get this before. Is something outdated on the website?

Here's the logs of what I did:

[ceph@ia1 ~]$ sudo rpm --import '
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'
[ceph@ia1 ~]$ sudo yum -y install ceph
Loaded plugins: dellsysid, fastestmirror, priorities, refresh-packagekit,
  : security
Loading mirror speeds from cached hostfile
 * base: mirror.symnds.com
 * epel: mirror.east.ig2ad.com
 * extras: mirror.umd.edu
 * updates: centos.mirror.constant.com
15 packages excluded due to repository priority protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ceph.x86_64 0:0.72.1-0.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved


 PackageArch Version   Repository
 Size

Installing:
 ceph   x86_64   0.72.1-0.el6  ceph
 13 M

Transaction Summary

Install   1 Package(s)

Total download size: 13 M
Installed size: 55 M
Downloading Packages:
http://ceph.com/rpm-emperor/el6/x86_64/ceph-0.72.1-0.el6.x86_64.rpm: [Errno
14] Downloaded more than max size for
http://ceph.com/rpm-emperor/el6/x86_64/ceph-0.72.1-0.el6.x86_64.rpm:
15221737 > 13835720
Trying other mirror.


Error Downloading Packages:
  ceph-0.72.1-0.el6.x86_64: failure: ceph-0.72.1-0.el6.x86_64.rpm from
ceph: [Errno 256] No more mirrors to try.


On Tue, Dec 3, 2013 at 4:07 PM, Gautam Saxena  wrote:

> In trying to download the RPM packages for CEPH, the yum commands timed
> out. I then tried just downloading them via Chrome browser (
> http://ceph.com/rpm-emperor/el6/x86_64/ceph-0.72.1-0.el6.x86_64.rpm) and
> it only downloaded 64KB. (The website www.ceph.com is slow too)
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Openstack+ceph volume mounting to vm

[Karan Singh]

Hello Every 

Still waiting .. Any help in this would be highly appreciated. 

Many Thanks 
Karan Singh 


- Original Message -

From: "Karan Singh"  
To: ceph-users@lists.ceph.com 
Sent: Tuesday, 3 December, 2013 6:27:29 PM 
Subject: [ceph-users] Openstack+ceph volume mounting to vm 

Hello Cephers 

Need your guidance 

In my setup ceph cluster and openstack are working good , i am able to create 
volumes using cinder as well. 

What i want is to mount ceph volume to VM instance. But getting deadly errors 
like this . Expecting your help in this 




[root@rdo /(keystone_admin)]# virsh attach-device instance-0018 disk.xml 
error: Failed to attach device from disk.xml 
error: internal error unable to execute QEMU command '__com.redhat_drive_add': 
Device 'drive-virtio-disk5' could not be initialized 

[root@rdo /(keystone_admin)]# 







My Setup details :- 




[root@rdo /(keystone_admin)]# rpm -qa | grep -i qemu 
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
gpxe-roms-qemu-0.9.7-6.10.el6.noarch 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# uname -a 
Linux rdo 3.10.18-1.el6.elrepo.x86_64 #1 SMP Mon Nov 4 19:12:54 EST 2013 x86_64 
x86_64 x86_64 GNU/Linux 
[root@rdo /(keystone_admin)]# 
[root@rdo /(keystone_admin)]# cat /etc/redhat-release 
CentOS release 6.5 (Final) 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# cinder list 
+--+---+--+--+--+--+-+
 
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | 
+--+---+--+--+--+--+-+
 
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | 
ceph-storage | false | | 
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | 
None | false | | 
+--+---+--+--+--+--+-+
 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# ceph status 
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a 
health HEALTH_OK 
monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 30, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 
osdmap e157: 11 osds: 11 up, 11 in 
pgmap v12102: 448 pgs: 448 active+clean; 135 GB data, 272 GB used, 5935 GB / 
6207 GB avail 
mdsmap e27: 1/1/1 up {0=ceph-mon1=up:active} 

[root@rdo /(keystone_admin)]# 





[root@rdo /(keystone_admin)]# cat disk.xml 
 
 
 
 
 
 
 
 
 
 
 
 
[root@rdo /(keystone_admin)]# 




[root@rdo /(keystone_admin)]# service libvirtd status 
libvirtd (pid 17947) is running... 
[root@rdo /(keystone_admin)]# 







virsh # list 
Id Name State 
 
2 instance-0018 running 

virsh # 





Karan Singh 
Systems Specialist, Computing Environments Group 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 


___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Current answers for ceph as backend to parallel filesystem

[JR]

Greetings all,

Does anyone have any recommendations for using ceph as a reliable,
distributed backend for any existing parallel filesystems?  My final
goal would be to have data reliability and availability handled by ceph
and the serving of a filesystem handled by .. well, a distributed,
parallel filesystem ;-)

To make this question clear I'll spell out a scenario that I've used and
ask about how ceph can fit it.

GPFS:
Servers running GPFS get their blocks from raid arrays over fiber
channel. The raid arrays do RAID 6. GPFS, as well, replicates the
metadata to guard against loss).

The GPFS servers also run samba/ctdb to serve files to clients, i.e.,

  \\files\path

refers to a variety of physical servers (via round robin dns); if a
server goes down the client is seemlessly directed to another server.

GPFS+ceph:
Servers run the ceph client software get blocks from ceph servers (e.g.,
boxes with lots of disk running the osd, mon, mds, processes ...). Ceph
replicates the data on the backend.  GPFS doesn't replicate either data
or metadata.

I haven't yet tried to use this approach since the intended use to which
I wish to put this storage cluster (probably) doesn't allow GPFS.  I
also have questions about the final performance given:
  ceph io server -> xfs filesystem -> osd processes -> network -> gpfs
server processes exported ceph blocks, etc

However, there are other parallel filesystems which might do, e.g., GFS,
gluster, others?

Am I on the right track in thinking about ceph being usable in this
scenario, or is ceph really better suited to being an object store and a
provider of blocks for virtual machines?

Also, how will the ceph filesystem help with the above problem when it
becomes available (if it will)?

Thanks much for your time,
JR


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Remove PGs that are stuck-unclean-stale

[Sean Patronis]

Background

New ceph setup with 3 nodes and a mon running on each node.  OSDs are 
split up across the nodes.  This is a brand new cluster and no data has 
been added.


I zapped osd.0 and re-added it and now I am stuck with:

health HEALTH_WARN 12 pgs degraded; 12 pgs stale; 12 pgs stuck stale; 12 
pgs stuck unclean



What is the best way to clean this up so everything reads hunky-dory?  I 
do not care about data loss since there is no data in the cluster yet.


Thanks.

--Sean

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Granularity/efficiency of copy-on-write?

[Josh Durgin]

On 12/02/2013 03:26 PM, Bill Eldridge wrote:

Hi all,

We're looking at using Ceph's copy-on-write for a ton of users'
replicated cloud image environments,
and are wondering how efficient Ceph is for adding user data to base
images -
is data added in normal 4kB or 64kB sizes, or can you specify block size
for volumes
(so you can have video partitions with large content and email/chat/web
cache partitions with small files)


Copy-on-write is currently implemented at object granularity. Object 
size is determined when you create an rbd image, and defaults to 4MB.



Is Ceph's behavior & configuration for copy-on-write documented well
somewhere?


For a detailed description of the current copy-on-write implementation 
see [1]. You may also be interested in rbd's striping configuration, 
which uses the same parameters as cephfs [2].


Josh

[1] http://ceph.com/docs/master/dev/rbd-layering/
[2] http://ceph.com/docs/master/dev/file-striping/

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Current answers for ceph as backend to parallel filesystem

[Noah Watkins]

I don't think there is any inherent limitation to using RADOS or RBD
as a backend for an a non-CephFS file system, as CephFS is inherently
built on top of RADOS (though I suppose it doesn't directly use
librados). However, the challenge would be in configuring and tuning
the two independent systems to perform well together, each of which
may make certain assumptions that are incompatible with the other.

For instance, instead of configuring GPFS to assume RAID6 devices, one
might configure an RBD pool with no replication, and rely instead on
GPFS Native RAID for fault-tolerance and availbiilty. Likewise, if an
RBD device were to be treated as a RAID6 device from the GPFS point of
view, it would be a waste of effort for GPFS to do anything related to
the failure of the device which is already handled by Ceph.

On Tue, Dec 3, 2013 at 2:14 PM, JR  wrote:
> Greetings all,
>
> Does anyone have any recommendations for using ceph as a reliable,
> distributed backend for any existing parallel filesystems?  My final
> goal would be to have data reliability and availability handled by ceph
> and the serving of a filesystem handled by .. well, a distributed,
> parallel filesystem ;-)
>
> To make this question clear I'll spell out a scenario that I've used and
> ask about how ceph can fit it.
>
> GPFS:
> Servers running GPFS get their blocks from raid arrays over fiber
> channel. The raid arrays do RAID 6. GPFS, as well, replicates the
> metadata to guard against loss).
>
> The GPFS servers also run samba/ctdb to serve files to clients, i.e.,
>
>   \\files\path
>
> refers to a variety of physical servers (via round robin dns); if a
> server goes down the client is seemlessly directed to another server.
>
> GPFS+ceph:
> Servers run the ceph client software get blocks from ceph servers (e.g.,
> boxes with lots of disk running the osd, mon, mds, processes ...). Ceph
> replicates the data on the backend.  GPFS doesn't replicate either data
> or metadata.
>
> I haven't yet tried to use this approach since the intended use to which
> I wish to put this storage cluster (probably) doesn't allow GPFS.  I
> also have questions about the final performance given:
>   ceph io server -> xfs filesystem -> osd processes -> network -> gpfs
> server processes exported ceph blocks, etc
>
> However, there are other parallel filesystems which might do, e.g., GFS,
> gluster, others?
>
> Am I on the right track in thinking about ceph being usable in this
> scenario, or is ceph really better suited to being an object store and a
> provider of blocks for virtual machines?
>
> Also, how will the ceph filesystem help with the above problem when it
> becomes available (if it will)?
>
> Thanks much for your time,
> JR
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Yan, Zheng]

On Tue, Dec 3, 2013 at 4:00 PM, Miguel Afonso Oliveira
 wrote:
>
>> If your issue is caused by the bug I presume, you need to use the
>> newest client (0.72 ceph-fuse or 3.12 kernel)
>>
>> Regards
>> Yan, Zheng
>
>
> Hi,
>
> We are running 0.72-1 throughout the cluster but on kernel
> 2.6.32-358.6.2.el6.x86_64... This is a big deployment (~1 cores), not so
> easy to update kernel has it has too many implications...
>
> Is there any way I can help iron out this bug?

2.6.32 kernel is too old, it's not easy to back port fixes to it. how
about using 0.72 ceph-fuse?

Regards
Yan, Zheng

>
> Cheers,
>
> MAO
>
>>
>>
>>> Cheers,
>>>
>>> MAO
>>>
>>>
>>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] btrfs constant background write activity even at idle

[James Harper]

I have been testing osd on btrfs, and the first thing I notice is that there is 
constant write activity when idle.

The write activity hovers between 5Mbytes/second and 30Mbytes/second, and 
averages around 9Mbytes/second (as determined by iostat -x 30). On average, 
iostat is showing around 90 w/s, and 25% util.

There were two OSD's on this server running btrfs and both were behaving like 
this. I changed one back to xfs last night and it's not showing any write 
activity at idle now.

There is no matching write activity on the journal device, but the high writes 
stop when I kill the OSD process.

kworker, ceph-osd, and btrfs-cleaner processes feature highly in iotop

Is this normal? And is it a problem, or just btrfs doing background writes for 
some reason?

I'm running dumpling on Debian, fwiw.

Thanks

James

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] HEALTH_WARN pool .rgw.buckets has too few pgs

[Alexis GÜNST HORN]

Hello,

I can't understand an error I have since now :

HEALTH_WARN pool .rgw.buckets has too few pgs.
Do you have any ideas ?

Some info :

[root@admin ~]# ceph --version
ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de)

[root@admin ~]# ceph osd pool get .rgw.buckets pgp_num
pgp_num: 10050

[root@admin ~]# ceph osd pool get .rgw.buckets pg_num
pg_num: 10050

[root@admin ~]# ceph -s
 (...)
 osdmap e30632: 201 osds: 201 up, 201 in
 pgmap v4984359: 90666 pgs, 13 pools, 1276 GB data, 340 kobjects
 3916 GB used, 727 TB / 731 TB avail
 90666 active+clean


Thanks a lot,
Alexis
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com