[ceph-users] Sunday's Ceph based business model

[Loic Dachary]

Hi Ceph,

Disclaimer: I'm no entrepreneur, the "business model" idea that came to me this 
Sunday should not be taken seriously ;-)

Let say individuals can buy hardware that are "Ceph ready" (i.e. contain some 
variation of https://wiki.ceph.com/Clustering_a_few_NAS_into_a_Ceph_cluster) 
and build a cluster with a few friends and family members. The cluster we all 
dream off, where a broken disk at home does not mean we can't view our photos 
anymore, and replacing it is a matter of plugging in a new one and watch it 
download objects from its peer OSDs. We would have distributed and self healing 
storage at no extra cost but more importantly without any company involved.

I'm hopeful such private storage clusters will become common place. But that 
leaves us with a minority of cases where a company could help. If I had no 
friend, I would be happy to pay a company to run a few OSDs for me and keep my 
data safe. And once I get enough friends the company would stop running OSDs 
for me and I would not need to pay it anymore. I would also subscribe to a 
company offer to run OSDs for me in case our cluster does not have enough. The 
company would not charge me anything nor add OSDs to the cluster unless it goes 
below a threshold (available free space / degraded objects...). I would be able 
to verify that and get an alert in case it intervenes when it should not, i.e. 
verifying if the company abuses the permissions to run OSDs to get a peek at my 
dataa. In other words, a company business model could be to bootstrap clusters 
and assist when they shrink too much. The majority of the users would not need 
the company's services but could seamlessly subscribe becaus
e the technology is the same. I believe that's the idea behind most freemium 
business models.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] HEALTH_WARN too few pgs per osd (0 < min 20)

[Jesus Chavez (jeschave)]

Hi all, does anybody know why still get WARN  message status?

I don’t even have pools yet son I am not sure why is warning me…


[root@capricornio ceph-cluster]# ceph status
cluster d39f6247-1543-432d-9247-6c56f65bb6cd
 health HEALTH_WARN too few pgs per osd (0 < min 20)
 monmap e1: 3 mons at 
{capricornio=192.168.4.44:6789/0,geminis=192.168.4.37:6789/0,tauro=192.168.4.36:6789/0},
 election epoch 46, quorum 0,1,2 tauro,geminis,capricornio
 osdmap e956: 120 osds: 120 up, 120 in
  pgmap v6252: 0 pgs, 0 pools, 0 bytes data, 0 objects
10177 MB used, 435 TB / 435 TB avail
[root@capricornio ceph-cluster]#


[root@capricornio ceph-cluster]# ceph osd lspools
[root@capricornio ceph-cluster]#


Thanks
[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN too few pgs per osd (0 < min 20)

[Thomas Foster]

I believe you are getting this message because you dont have any pools, and
placement groups.  Thats what the message is telling you that you have less
(0) than the minimum amount(20)..once you create the pools and placement
groups verify if you are still receiving this message.
On Mar 15, 2015 11:23 AM, "Jesus Chavez (jeschave)" 
wrote:

>  Hi all, does anybody know why still get WARN  message status?
>
>  I don’t even have pools yet son I am not sure why is warning me…
>
>
>  [root@capricornio ceph-cluster]# ceph status
> cluster d39f6247-1543-432d-9247-6c56f65bb6cd
>  health HEALTH_WARN too few pgs per osd (0 < min 20)
>  monmap e1: 3 mons at {capricornio=
> 192.168.4.44:6789/0,geminis=192.168.4.37:6789/0,tauro=192.168.4.36:6789/0},
> election epoch 46, quorum 0,1,2 tauro,geminis,capricornio
>  osdmap e956: 120 osds: 120 up, 120 in
>   pgmap v6252: 0 pgs, 0 pools, 0 bytes data, 0 objects
> 10177 MB used, 435 TB / 435 TB avail
> [root@capricornio ceph-cluster]#
>
>
>  [root@capricornio ceph-cluster]# ceph osd lspools
> [root@capricornio ceph-cluster]#
>
>
>  Thanks
>
>
> * Jesus Chavez*
> SYSTEMS ENGINEER-C.SALES
>
> jesch...@cisco.com
> Phone: *+52 55 5267 3146 <%2B52%2055%205267%203146>*
> Mobile: *+51 1 5538883255*
>
> CCIE - 44433
>
>
> Cisco.com 
>
>
>
>
>
>   Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
>  for
> Company Registration Information.
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN too few pgs per osd (0 < min 20)

[Loic Dachary]

Hi,

On 15/03/2015 16:23, Jesus Chavez (jeschave) wrote:
> Hi all, does anybody know why still get WARN  message status?
> 
> I don’t even have pools yet son I am not sure why is warning me…
> 
> 
> [root@capricornio ceph-cluster]# ceph status
> cluster d39f6247-1543-432d-9247-6c56f65bb6cd
>  health HEALTH_WARN too few pgs per osd (0 < min 20)
>  monmap e1: 3 mons at 
> {capricornio=192.168.4.44:6789/0,geminis=192.168.4.37:6789/0,tauro=192.168.4.36:6789/0},
>  election epoch 46, quorum 0,1,2 tauro,geminis,capricornio
>  osdmap e956: 120 osds: 120 up, 120 in
>   pgmap v6252: 0 pgs, 0 pools, 0 bytes data, 0 objects
> 10177 MB used, 435 TB / 435 TB avail
> [root@capricornio ceph-cluster]#

It's an interesting border case ;-) It should not complain that there are not 
enough PG per OSD if there are no pools.

Cheers

> 
> 
> [root@capricornio ceph-cluster]# ceph osd lspools
> [root@capricornio ceph-cluster]#
> 
> 
> Thanks
> 
> *
> *Jesus Chavez**
> SYSTEMS ENGINEER-C.SALES
> 
> jesch...@cisco.com 
> Phone: *+52 55 5267 3146*
> Mobile: *+51 1 5538883255*
> 
> CCIE - 44433
> 
>   
> 
> 
> Cisco.com 
> 
> 
> 
> 
> 
> 
>   Think before you print.
> 
> This email may contain confidential and privileged material for the sole use 
> of the intended recipient. Any review, use, distribution or disclosure by 
> others is strictly prohibited. If you are not the intended recipient (or 
> authorized to receive for the recipient), please contact the sender by reply 
> email and delete all copies of this message.
> 
> Please click here 
>  for 
> Company Registration Information.
> 
> 
> 
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN too few pgs per osd (0 < min 20)

[Jesus Chavez (jeschave)]

yeah thats what I thought, I had the same WARN message with RHEL6.6 but I had a 
pool and when I changed the value of pgs the message just gone...
[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





On Mar 15, 2015, at 9:23 AM, Jesus Chavez (jeschave) 
mailto:jesch...@cisco.com>> wrote:

Hi all, does anybody know why still get WARN  message status?

I don’t even have pools yet son I am not sure why is warning me…


[root@capricornio ceph-cluster]# ceph status
cluster d39f6247-1543-432d-9247-6c56f65bb6cd
 health HEALTH_WARN too few pgs per osd (0 < min 20)
 monmap e1: 3 mons at 
{capricornio=192.168.4.44:6789/0,geminis=192.168.4.37:6789/0,tauro=192.168.4.36:6789/0},
 election epoch 46, quorum 0,1,2 tauro,geminis,capricornio
 osdmap e956: 120 osds: 120 up, 120 in
  pgmap v6252: 0 pgs, 0 pools, 0 bytes data, 0 objects
10177 MB used, 435 TB / 435 TB avail
[root@capricornio ceph-cluster]#


[root@capricornio ceph-cluster]# ceph osd lspools
[root@capricornio ceph-cluster]#


Thanks



Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com









  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FW: More than 50% osds down, CPUs still busy; will the cluster recover without help?

[Chris Murray]

Apologies if anyone receives this twice. I didn't see this e-mail come back 
through to the list ...

-Original Message-
From: Chris Murray 
Sent: 14 March 2015 08:56
To: 'Gregory Farnum'
Cc: 'ceph-users'
Subject: RE: [ceph-users] More than 50% osds down, CPUs still busy; will the 
cluster recover without help?

Good evening all,

Just had another quick look at this with some further logging on and thought 
I'd post the results in case anyone can keep me moving in the right direction. 

Long story short, some OSDs just don't appear to come up after one failing 
after another. Dealing with one in isolation, after a load of IO, it never 
starts. The last in the log looks like this:

...
2015-03-13 18:43:11.875392 7f29d1e98780 10 filestore  > header.spos 0.0.0
2015-03-13 18:43:11.876568 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
_omap_rmkeyrange meta/39e3fb/pglog_4.57c/0//-1 
[00.,4294967295.18446744073709551615]
2015-03-13 18:43:11.876598 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
get_omap_iterator meta/39e3fb/pglog_4.57c/0//-1
2015-03-13 18:43:11.952511 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
_omap_rmkeys meta/39e3fb/pglog_4.57c/0//-1
2015-03-13 18:43:11.952878 7f29d1e98780 10 filestore oid: 
39e3fb/pglog_4.57c/0//-1 not skipping op, *spos 13288339.0.3
2015-03-13 18:43:11.952892 7f29d1e98780 10 filestore  > header.spos 0.0.0
2015-03-13 18:43:11.961127 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
_omap_rmkeys meta/39e3fb/pglog_4.57c/0//-1
2015-03-13 18:43:11.961516 7f29d1e98780 10 filestore oid: 
39e3fb/pglog_4.57c/0//-1 not skipping op, *spos 13288339.0.4
2015-03-13 18:43:11.961529 7f29d1e98780 10 filestore  > header.spos 0.0.0
2015-03-13 18:43:11.965687 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
_omap_setkeys meta/39e3fb/pglog_4.57c/0//-1
2015-03-13 18:43:11.966082 7f29d1e98780 10 filestore oid: 
39e3fb/pglog_4.57c/0//-1 not skipping op, *spos 13288339.0.5
2015-03-13 18:43:11.966095 7f29d1e98780 10 filestore  > header.spos 0.0.0
2015-03-13 18:43:11.989820 7f29d1e98780 10 journal op_apply_finish 13288339 
open_ops 1 -> 0, max_applied_seq 13288338 -> 13288339
2015-03-13 18:43:11.989861 7f29d1e98780  3 journal journal_replay: r = 0, 
op_seq now 13288339
2015-03-13 18:43:11.989896 7f29d1e98780  2 journal read_entry 3951706112 : seq 
13288340 1755 bytes
2015-03-13 18:43:11.989900 7f29d1e98780  3 journal journal_replay: applying op 
seq 13288340
2015-03-13 18:43:11.989903 7f29d1e98780 10 journal op_apply_start 13288340 
open_ops 0 -> 1
2015-03-13 18:43:11.989906 7f29d1e98780 10 filestore(/var/lib/ceph/osd/ceph-1) 
_do_transaction on 0x2750480
2015-03-13 18:43:11.989919 7f29d1e98780 15 filestore(/var/lib/ceph/osd/ceph-1) 
_omap_setkeys meta/16ef7597/infos/head//-1
2015-03-13 18:43:11.990251 7f29d1e98780 10 filestore oid: 
16ef7597/infos/head//-1 not skipping op, *spos 13288340.0.1
2015-03-13 18:43:11.990263 7f29d1e98780 10 filestore  > header.spos 0.0.0
2015-03-13 18:43:15.404558 7f29c4439700 20 filestore(/var/lib/ceph/osd/ceph-1) 
sync_entry woke after 5.000217
2015-03-13 18:43:15.404600 7f29c4439700 10 journal commit_start max_applied_seq 
13288339, open_ops 1
2015-03-13 18:43:15.404603 7f29c4439700 10 journal commit_start waiting for 1 
open ops to drain

What might this 'open op' mean when it never seems to finish 'draining'? Could 
my suspicions be true that it's somehow a BTRFS funny?

Thanks again,
Chris

-Original Message-
From: Chris Murray
Sent: 03 March 2015 09:45
To: Gregory Farnum
Cc: ceph-users
Subject: RE: [ceph-users] More than 50% osds down, CPUs still busy; will the 
cluster recover without help?

Ah yes, that's a good point :-)

Thank you for your assistance Greg, I'm understanding a little more about how 
Ceph operates under the hood now.

We're probably at a reasonable point for me to say I'll just switch the 
machines off and forget about them for a while. It's no great loss; I just 
wanted to see if the cluster would come back to life despite any mis-treatment, 
and how far it can be pushed with the limited resources on the Microservers.

Getting to the admin socket fails:

root@ceph26:~# ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok help
admin_socket: exception getting command descriptions: [Errno 111] Connection 
refused

And after activity ceased on /dev/sdb ...

(60 second intervals again, snipped many hours of these sorts of figures)
sdb   5.52 0.00   801.27  0  48076
sdb   4.68 0.00   731.80  0  43908
sdb   5.25 0.00   792.80  0  47568
sdb  18.83   483.07   569.53  28984  34172
sdb  28.28   894.6035.40  53676   2124
sdb   0.00 0.00 0.00  0  0
sdb   0.00 0.00 0.00  0  0
sdb   0.00 0.00 0.00  0 

[ceph-users] Ceph release timeline

[Loic Dachary]

Hi Ceph,

In an attempt to clarify what Ceph release is stable, LTS or development. a new 
page was added to the documentation: http://ceph.com/docs/master/releases/ It 
is a matrix where each cell is a release number linked to the release notes 
from http://ceph.com/docs/master/release-notes/. One line per month and one 
column per release.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd goes down

[Jesus Chavez (jeschave)]

Hi guys!

After get a cluster for 10 servers and build a image for a pool of 450TB, it 
got sucked at the mkfs moment and what I noticed was that my entire cluster was 
failing in a really weird way, some osds goes down and up from different nodes 
and repeat so now I have a lot of PGs degraded and stuck :S I don’t know why 
the cluster behave in that way (osds going up and down) do you think is a bug?

[root@capricornio ceph-cluster]# ceph status
cluster d39f6247-1543-432d-9247-6c56f65bb6cd
 health HEALTH_WARN 109 pgs degraded; 251 pgs down; 1647 pgs peering; 2598 
pgs stale; 1618 pgs stuck inactive; 1643 pgs stuck unclean; 109 pgs undersized; 
1 requests are blocked > 32 sec; recovery 13/1838 objects degraded (0.707%); 
1/107 in osds are down
 monmap e1: 3 mons at 
{capricornio=192.168.4.44:6789/0,geminis=192.168.4.37:6789/0,tauro=192.168.4.36:6789/0},
 election epoch 50, quorum 0,1,2 tauro,geminis,capricornio
 osdmap e5275: 119 osds: 106 up, 107 in
  pgmap v15826: 8192 pgs, 1 pools, 2016 MB data, 919 objects
48484 MB used, 388 TB / 388 TB avail
13/1838 objects degraded (0.707%)
2162 stale+active+clean
1031 peering
4274 active+clean
   3 stale+remapped+peering
  32 stale+active+undersized+degraded
  43 stale+down+peering
   4 remapped+peering
  77 active+undersized+degraded
 208 down+peering
 358 stale+peering

[cid:image005.png@01D00809.A6D502D0]


Jesus Chavez
SYSTEMS ENGINEER-C.SALES

jesch...@cisco.com
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255

CCIE - 44433


Cisco.com





[cid:image006.gif@01D00809.A6D502D0]



  Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release timeline

[Lindsay Mathieson]

Thanks, thats quite helpful.

On 16 March 2015 at 08:29, Loic Dachary  wrote:

> Hi Ceph,
>
> In an attempt to clarify what Ceph release is stable, LTS or development.
> a new page was added to the documentation:
> http://ceph.com/docs/master/releases/ It is a matrix where each cell is a
> release number linked to the release notes from
> http://ceph.com/docs/master/release-notes/. One line per month and one
> column per release.
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Lindsay
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release timeline

[Georgios Dimitrakakis]

Indeed it is!

Thanks!

George



Thanks, thats quite helpful.

On 16 March 2015 at 08:29, Loic Dachary  wrote:


Hi Ceph,

In an attempt to clarify what Ceph release is stable, LTS or
development. a new page was added to the documentation:
http://ceph.com/docs/master/releases/ [1] It is a matrix where each
cell is a release number linked to the release notes from
http://ceph.com/docs/master/release-notes/ [2]. One line per month
and one column per release.

Cheers

--
Loïc Dachary, Artisan Logiciel Libre

___
ceph-users mailing list
ceph-users@lists.ceph.com [3]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [4]


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Shadow files

[Ben]

It is either a problem with CEPH, Civetweb or something else in our 
configuration.
But deletes in user buckets is still leaving a high number of old shadow 
files. Since we have millions and millions of objects, it is hard to 
reconcile what should and shouldnt exist.


Looking at our cluster usage, there are no troughs, it is just a rising 
peak.
But when looking at users data usage, we can see peaks and troughs as 
you would expect as data is deleted and added.


Our ceph version 0.80.9

Please ideas?

On 2015-03-13 02:25, Yehuda Sadeh-Weinraub wrote:

- Original Message -

From: "Ben" 
To: ceph-us...@ceph.com
Sent: Wednesday, March 11, 2015 8:46:25 PM
Subject: Re: [ceph-users] Shadow files

Anyone got any info on this?

Is it safe to delete shadow files?


It depends. Shadow files are badly named objects that represent part
of the objects data. They are only safe to remove if you know that the
corresponding objects no longer exist.

Yehuda



On 2015-03-11 10:03, Ben wrote:
> We have a large number of shadow files in our cluster that aren't
> being deleted automatically as data is deleted.
>
> Is it safe to delete these files?
> Is there something we need to be aware of when deleting them?
> Is there a script that we can run that will delete these safely?
>
> Is there something wrong with our cluster that it isn't deleting these
> files when it should be?
>
> We are using civetweb with radosgw, with tengine ssl proxy infront of
> it
>
> Any advice please
> Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] tgt and krbd

[Mike Christie]

On 03/09/2015 11:15 AM, Nick Fisk wrote:
> Hi Mike,
> 
> I was using bs_aio with the krbd and still saw a small caching effect. I'm
> not sure if it was on the ESXi or tgt/krbd page cache side, but I was
> definitely seeing the IO's being coalesced into larger ones on the krbd

I am not sure what you mean here. By coalescing you mean merging right?
That is not the same as caching. Coalescing/merging is expected for both
aio and rdwr.


> device in iostat. Either way, it would make me potentially nervous to run it
> like that in a HA setup.
> 
> 
>> tgt itself does not do any type of caching, but depending on how you have
>> tgt access the underlying block device you might end up using the normal
> old
>> linux page cache like you would if you did
>>
>> dd if=/dev/rbd0 of=/dev/null bs=4K count=1 dd if=/dev/rbd0 of=/dev/null
>> bs=4K count=1
>>
>> This is what Ronnie meant in that thread when he was saying there might be
>> caching in the underlying device.
>>
>> If you use tgt bs_rdwr.c (--bstype=rdwr) with the default settings and
> with
>> krbd then you will end up doing caching, because the krbd's block device
> will
>> be accessed like in the dd example above (no direct bits set).
>>
>> You can tell tgt bs_rdwr devices to use O_DIRECT or O_SYNC. When you
>> create the lun pass in the "--bsoflags {direct | sync }". Here is an
> example
>> from the man page:
>>
>> tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1
> --bsoflags="sync" -
>> -backing-store=/data/100m_image.raw
>>
>>
>> If you use bs_aio.c then we always set O_DIRECT when opening the krbd
>> device, so no page caching is done. I think linux aio might require this
> or at
>> least it did at the time it was written.
>>
>> Also the cache settings exported to the other OS's initiator with that
>> modepage command might affect performance then too. It might change
>> how that OS does writes like send cache syncs down or do some sort of
>> barrier or FUA.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS: delayed objects deletion ?

[Yan, Zheng]

On Sat, Mar 14, 2015 at 5:22 PM, Florent B  wrote:
> Hi,
>
> What do you call "old MDS" ? I'm on Giant release, it is not very old...
>
> And I tried restarting both but it didn't solve my problem.
>
> Will it be OK in Hammer ?
>
> On 03/13/2015 04:27 AM, Yan, Zheng wrote:
>> On Fri, Mar 13, 2015 at 1:17 AM, Florent B  wrote:
>>> Hi all,
>>>
>>> I test CephFS again on Giant release.
>>>
>>> I use ceph-fuse.
>>>
>>> After deleting a large directory (few hours ago), I can see that my pool
>>> still contains 217 GB of objects.
>>>
>>> Even if my root directory on CephFS is empty.
>>>
>>> And metadata pool is 46 MB.
>>>
>>> Is it expected ? If not, how to debug this ?

I don't know what was wrong. could you use "rados -p data ls" to check
which objects still exist. Then restart the mds MDS with debug_mds=20
and search the log for name of the remaining objects.

>> Old mds does not work well in this area. Try umounting clients and
>> restarting MDS.
>>
>> Regards
>> Yan, Zheng
>>
>>
>>> Thank you.
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] tgt and krbd

[Mike Christie]

On 03/15/2015 07:54 PM, Mike Christie wrote:
> On 03/09/2015 11:15 AM, Nick Fisk wrote:
>> Hi Mike,
>>
>> I was using bs_aio with the krbd and still saw a small caching effect. I'm
>> not sure if it was on the ESXi or tgt/krbd page cache side, but I was
>> definitely seeing the IO's being coalesced into larger ones on the krbd
> 
> I am not sure what you mean here. By coalescing you mean merging right?
> That is not the same as caching. Coalescing/merging is expected for both
> aio and rdwr.

For being able to see caching with aio though, I think you are right and
there might be a case where can use buffered writes even when using
O_DIRECT to the rbd device. I am not too familiar with that code, so let
me ping a person that works here and get back to the list.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

[Yan, Zheng]

On Sat, Mar 14, 2015 at 7:03 AM, Scottix  wrote:
> ...
>
>
>> The time variation is caused cache coherence. when client has valid
>> information
>> in its cache, 'stat' operation will be fast. Otherwise the client need to
>> send
>> request to MDS and wait for reply, which will be slow.
>
>
> This sounds like the behavior I had with CephFS giving me question marks.
> When I had a directory with a large amount of files in it and the first ls
> -la took a while to populate and ended with some unknown stats. The second
> time I did an ls -la it ran quick with no question marks. My inquiry was if
> there is a timeout that could occur? since it has to go ask the mds on a
> different machine it seems plausible that the full response is not coming
> back in time or fails to get all stats at some point.

unknown stats shouldn't happen. which kernel are ypu using? can you reproduce
it with ceph-fuse?

Regards
Yan, Zheng

>
> I could test this more; is there a command or proccess I can perform to
> flush the ceph-fuse cache?
>
> Thanks,
> Scott
>
>
> On Fri, Mar 13, 2015 at 1:49 PM Francois Lafont  wrote:
>>
>> Hi,
>>
>> Yan, Zheng wrote :
>>
>> >> http://tracker.ceph.com/issues/11059
>> >>
>> >
>> > It's a bug in ACL code, I have updated
>> > http://tracker.ceph.com/issues/11059
>>
>> Ok, thanks. I have seen and I will answer quickly. ;)
>>
>> >> I'm still surprised by such times. For instance, It seems to me
>> >> that, with a mounted nfs share, commands like "ls -la" are very
>> >> fast in comparison (with a directory which contains the same number
>> >> of files). Can anyone explain to me why there is a such difference
>> >> between the nfs case and the cephfs case? This is absolutely not a
>> >> criticism but it's just to understand the concepts that come into
>> >> play. In the case of "ls -al" ie just reading (it is assumed that
>> >> there is no writing on the directory), the nfs and the cephfs cases
>> >> seem to me very similar: the client just requests a stat on each file
>> >> in the directory. Am I wrong?
>> >
>> > NFS has no cache coherence mechanism. It can't guarantee one client
>> > always
>> > see other client's change.
>>
>> Ah ok, I didn't know that. Indeed, now I understand that can generate
>> performance impact.
>>
>> > The time variation is caused cache coherence. when client has valid
>> > information
>> > in its cache, 'stat' operation will be fast. Otherwise the client need
>> > to send
>> > request to MDS and wait for reply, which will be slow.
>>
>> Ok, thanks a lot for your explanations.
>> Regards.
>>
>> --
>> François Lafont
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly, cephfs issues: different unix rights depending on the client and ls are slow

[Yan, Zheng]

On Sat, Mar 14, 2015 at 7:03 AM, Scottix  wrote:
> ...
>
>
>> The time variation is caused cache coherence. when client has valid
>> information
>> in its cache, 'stat' operation will be fast. Otherwise the client need to
>> send
>> request to MDS and wait for reply, which will be slow.
>
>
> This sounds like the behavior I had with CephFS giving me question marks.
> When I had a directory with a large amount of files in it and the first ls
> -la took a while to populate and ended with some unknown stats. The second
> time I did an ls -la it ran quick with no question marks. My inquiry was if
> there is a timeout that could occur? since it has to go ask the mds on a
> different machine it seems plausible that the full response is not coming
> back in time or fails to get all stats at some point.

unknown stats shouldn't happen. which kernel are you using? can you reproduce
this issue with ceph-fuse?

Regards
Yan, Zheng

>
> I could test this more; is there a command or proccess I can perform to
> flush the ceph-fuse cache?
>
> Thanks,
> Scott
>
>
> On Fri, Mar 13, 2015 at 1:49 PM Francois Lafont  wrote:
>>
>> Hi,
>>
>> Yan, Zheng wrote :
>>
>> >> http://tracker.ceph.com/issues/11059
>> >>
>> >
>> > It's a bug in ACL code, I have updated
>> > http://tracker.ceph.com/issues/11059
>>
>> Ok, thanks. I have seen and I will answer quickly. ;)
>>
>> >> I'm still surprised by such times. For instance, It seems to me
>> >> that, with a mounted nfs share, commands like "ls -la" are very
>> >> fast in comparison (with a directory which contains the same number
>> >> of files). Can anyone explain to me why there is a such difference
>> >> between the nfs case and the cephfs case? This is absolutely not a
>> >> criticism but it's just to understand the concepts that come into
>> >> play. In the case of "ls -al" ie just reading (it is assumed that
>> >> there is no writing on the directory), the nfs and the cephfs cases
>> >> seem to me very similar: the client just requests a stat on each file
>> >> in the directory. Am I wrong?
>> >
>> > NFS has no cache coherence mechanism. It can't guarantee one client
>> > always
>> > see other client's change.
>>
>> Ah ok, I didn't know that. Indeed, now I understand that can generate
>> performance impact.
>>
>> > The time variation is caused cache coherence. when client has valid
>> > information
>> > in its cache, 'stat' operation will be fast. Otherwise the client need
>> > to send
>> > request to MDS and wait for reply, which will be slow.
>>
>> Ok, thanks a lot for your explanations.
>> Regards.
>>
>> --
>> François Lafont
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Sunday's Ceph based business model

[Anthony D'Atri]

Interesting idea.  I'm not sure, though, Ceph is designed with this sort of 
latency in mind.  

Crashplan does let you do something very similar for free, as I understand it, 
though it's more of a nearline thing.  


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mapping users to different rgw pools

[Sreenath BH]

Thanks.

Is this possible outside of multi-zone setup. (With only one Zone)?

For example, I want to have pools with different replication
factors(or erasure codings) and map users to these pools.

-Sreenath


On 3/13/15, Craig Lewis  wrote:
> Yes, RadosGW has the concept of Placement Targets and Placement Pools.  You
> can create a target, and point it a set of RADOS pools.  Those pools can be
> configured to use different storage strategies by creating different
> crushmap rules, and assigning those rules to the pool.
>
> RGW users can be assigned a default placement target.  When they create a
> bucket, they can either specify the target, or use their default one.  All
> objects in a bucket are stored according to the bucket's placement target.
>
>
> I haven't seen a good guide for making use of these features.  The best
> guide I know of is the Federation guide (
> http://ceph.com/docs/giant/radosgw/federated-config/), but it only briefly
> mentions placement targets.
>
>
>
> On Thu, Mar 12, 2015 at 11:48 PM, Sreenath BH  wrote:
>
>> Hi all,
>>
>> Can one Radow gateway support more than one pool for storing objects?
>>
>> And as a follow-up question, is there a way to map different users to
>> separate rgw pools so that their obejcts get stored in different
>> pools?
>>
>> thanks,
>> Sreenath
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com