Re: [ceph-users] CTDB Cluster Samba on Cephfs

2013-03-29 Thread Jeremy Allison 
On Thu, Mar 28, 2013 at 07:41:55AM -0700, Sage Weil wrote:
> On Wed, 27 Mar 2013, Matthieu Patou wrote:
> > On 03/27/2013 10:41 AM, Marco Aroldi wrote:
> > > Hi list,
> > > I'm trying to create a active/active Samba cluster on top of Cephfs
> > > I would ask if Ceph fully supports CTDB at this time.
> >
> > If I'm not wrong Ceph (even CephFS) do not support exporting a block device 
> > or
> > mounting the same FS more than once whereas CTDB explicitly require that you
> > have a distributed filesystem where the same filesystem is mounted across 
> > all
> > the nodes.
> 
> Er, sort of.
> 
> RBD presents a block-based interface.  As many clients as you want can use 
> that at the same time, although the client caching should be disabled if 
> the users don't call flush() to make their writes visible to others.  
> Beyond that, RBD itself doesn't care how many people use it.  *However*, 
> if you put a "local file system" on top of RBD (like ext4, xfs, btrfs, 
> zfs, ...), only one client should use RBD at a time because those file 
> systems are designed for exclusive access to the disk.  If you use a 
> "clustered file system" like ocfs2 or gfs[2], multiple clients can share 
> the same RBD volume in a useful way, but they expect coherency to behave 
> like on a SAN (which means writes are visible immediately but not durable 
> until flush), which means RBD caching should be turned off.
> 
> CephFS is designed for concurrent, shared, coherent access from many many 
> clients.  Think NFS, but scalable, and coherent/consistent when clients 
> are accessing the same files and directories.
> 
> As for CTDB: we haven't looked at this specifically in the context for 
> Samba.  Generally speaking, anything you can do with NFS or another shared 
> file system you can do with CephFS.  IIRC last time I discussed this with 
> the Samba guys, there is more we could do here to make CTDB much more 
> efficient (by backing it with RADOS, for example), but we haven't looked 
> at any of this carefully.  We would love to see clustered Samba working 
> well on CephFS, though!  If you haven't already, plese look at our 
> samba.git cloen on github, which has patches gluing libcephfs directly 
> into Samba's VFS, allowing you to directly reexport CephFS via Samba 
> without a local mountpoint in the middle.

Oh, I didn't know about that ! Any chance of submitting it
to the main Samba repository so we can keep it up to date
with VFS changes ?

Jeremy.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CTDB Cluster Samba on Cephfs

2013-03-29 Thread Marco Aroldi 
Still trying with no success:

Sage and Ronnie:
I've tried the ping_pong tool, even with "locking=no" in my smb.conf
(no differences)

# ping_pong /mnt/ceph/samba-cluster/test 3
I have about 180 locks/second
If I start the same command from the other node, the tools stops
completely. 0 locks/second

Sage, when I start the CTDB service, the mds log says every second:
2013-03-29 16:49:34.442437 7f33fe6f3700  0 mds.0.server
handle_client_file_setlock: start: 0, length: 0, client: 5475, pid:
14795, type: 4

2013-03-29 16:49:35.440856 7f33fe6f3700  0 mds.0.server
handle_client_file_setlock: start: 0, length: 0, client: 5475, pid:
14799, type: 4

Exactly as you see it: with a blank line in between
When i start the ping_pong command i have these lines at the same rate
reported by the script (180 lines/second):

2013-03-29 17:07:50.277003 7f33fe6f3700  0 mds.0.server
handle_client_file_setlock: start: 2, length: 1, client: 5481, pid:
11011, type: 2

2013-03-29 17:07:50.281279 7f33fe6f3700  0 mds.0.server
handle_client_file_setlock: start: 1, length: 1, client: 5481, pid:
11011, type: 4

2013-03-29 17:07:50.286643 7f33fe6f3700  0 mds.0.server
handle_client_file_setlock: start: 0, length: 1, client: 5481, pid:
11011, type: 2

Finally, I've tried to lower the ctdb's RecoverBanPeriod but the
clients was unable to recover for 5 minutes (again!)
So, I've found the mds logging this:
2013-03-29 16:55:23.354854 7f33fc4ed700  0 log [INF] : closing stale
session client.5475 192.168.130.11:0/580042840 after 300.159862

I hope to find a solution.
I am at your disposal to further investigation

--
Marco Aroldi

2013/3/29 ronnie sahlberg :
> The ctdb package comes with a tool "ping pong" that is used to test
> and exercise fcntl() locking.
>
> I think a good test is using this tool and then randomly powercycling
> nodes in your fs cluster
> making sure that
> 1, fcntl() locking is still coherent and correct
> 2, always recover within 20 seconds for a single node power cycle
>
>
> That is probably a good test for CIFS serving.
>
>
> On Thu, Mar 28, 2013 at 6:22 PM, ronnie sahlberg
>  wrote:
>> On Thu, Mar 28, 2013 at 6:09 PM, Sage Weil  wrote:
>>> On Thu, 28 Mar 2013, ronnie sahlberg wrote:
 Disable the recovery lock file from ctdb completely.
 And disable fcntl locking from samba.

 To be blunt, unless your cluster filesystem is called GPFS,
 locking is probably completely broken and should be avoided.
>>>
>>> Ha!
>>>
 On Thu, Mar 28, 2013 at 8:46 AM, Marco Aroldi  
 wrote:
 > Thanks for the answer,
 >
 > I haven't yet looked at the samba.git clone, sorry. I will.
 >
 > Just a quick report on my test environment:
 > * cephfs mounted with kernel driver re-exported from 2 samba nodes
 > * If "node B" goes down, everything works like a charm: "node A" does
 > ip takeover and bring up the "node B"'s ip
 > * Instead, if "node A" goes down, "node B" can't take the rlock file
 > and gives this error:
 >
 > ctdb_recovery_lock: Failed to get recovery lock on
 > '/mnt/ceph/samba-cluster/rlock'
 > Unable to get recovery lock - aborting recovery and ban ourself for 300 
 > seconds
 >
 > * So, for 5 minutes, neither "node A" nor "node B" are active. After
 > that, the cluster recover correctly.
 > It seems that one of the 2 nodes "owns" and don't want to "release"
 > the rlock file
>>>
>>> Cephfs aims to give you coherent access between nodes.  The cost of that
>>> is that if another client goes down and it holds some lease/lock, you have
>>> to wait for it to time out.  That is supposed to happen after 60 seconds,
>>> it sounds like you've hit a bug here.  The flock/fnctl locks aren't
>>> super-well tested in the failure scenarios.
>>>
>>> Even assuming it were working, though, I'm not sure that you want to wait
>>> the 60 seconds either for the CTDB's to take over for each other.
>>
>> You do not want to wait 60 seconds. That is approaching territory where
>> CIFS clients will start causing file corruption and dataloss due to
>> them dropping writeback caches.
>>
>> You probably want to aim to try to guarantee that fcntl() locking
>> start working again after
>> ~20 seconds or so to have some headroom.
>>
>>
>> Microsoft themself state 25seconds as the absolute deadline they
>> require you guarantee before they will qualify storage.
>> That is among other things to accomodate and have some headroom for
>> some really nasty dataloss issues that will
>> happen if storage can not recover quickly enough.
>>
>>
>> CIFS is hard realtime. And you will pay dearly for missing the deadline.
>>
>>
>> regards
>> ronnie sahlberg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CTDB Cluster Samba on Cephfs

2013-03-29 Thread ronnie sahlberg 
On Fri, Mar 29, 2013 at 9:31 AM, Marco Aroldi  wrote:
> Still trying with no success:
>
> Sage and Ronnie:
> I've tried the ping_pong tool, even with "locking=no" in my smb.conf
> (no differences)
>
> # ping_pong /mnt/ceph/samba-cluster/test 3
> I have about 180 locks/second

That is very slow.

> If I start the same command from the other node, the tools stops
> completely. 0 locks/second

Looks like fcntl() locking doenst work that well.



The slow rate of fcntl() lock  will impact samba.
By default, for almost all file i/o samba will need to do at least one
fcntl(G_GETLK) in ordet to check whether some other, non-samba,
process holds a lock to the file.
If you can only do 180 fcntl(F_*LK)  per second across the cluster for
a file (I assume this is per file limitation)
this will have the effect of you only being able to do 180 i/o per
second to a file, which will make CIFS impossibly slow for any real
use.
This was all from a single node as well  so no inter-node contention!


So here you probably want to use "locking = no" in samba.  But beware,
  locking=no can have catastrophic effects on your data.
But without "locking = no"  would just become impossibly slow,
probably uselessly slow.


Using "locking = no" in samba does mean though that you no longer have
any locking coherency across protocols.
I.e.  NFS clients and samba clients are now disjoint since they can no
longer see eachothers locks.

If you only ever access the data via CIFS,  locking = no  should be safe.
But IF you access data via NFS or other NS protocols,   breaking lock
coherency across protocols like this could lead to dataloss
depending on the i/o patterns.


I would recommend only using   locking = no   if you can guarantee
that you will never export the data via other means than CIFS.
If you can not guarantee that,  you will have to reseach the use
patterns very carefully to determine whether locking = no is safe or
not.



I think for fcntl() locking,   depending on use case , is this a home
server where you can accept very poor performance?   or is this a
server for a small workgroup?
If the latter, if using locking = yes   you probably want your
filesystem to allow  >10.000 operations per second from a node with no
contention
and >1000 operations per node per second when there is contention across nodes.

If it is a big server, you probably want >> instead of > for these
numbers. At least.


But first you would need to get ping_pong working reliably  both
running in a steady state, and later running and recovering from
continous single node reboots.
It seems ping pong is not working really well for you at all at this
stage,   so that is likely a problem.



As I said,  very few cluster filesystems have fcntl() locking that is
not completely broken.




For now,   you could try "lokcing = no" in samba with the caveats above,
and you can disable using fcntl() split brain prevention in CTDB by setting
CTDB_RECOVERY_LOCK=

in /etc/sysconfig/ctdb
This will disable the split brain detection in ctdb  but allow you to
recover quicker if your cluster fs does not handle fcntl() locking
well.
(with 5 minute recovery   you will have so much dataloss due to the
way CIFS clients work and timeout  that there is probably little point
in running CIFS at all)





>
> Sage, when I start the CTDB service, the mds log says every second:
> 2013-03-29 16:49:34.442437 7f33fe6f3700  0 mds.0.server
> handle_client_file_setlock: start: 0, length: 0, client: 5475, pid:
> 14795, type: 4
>
> 2013-03-29 16:49:35.440856 7f33fe6f3700  0 mds.0.server
> handle_client_file_setlock: start: 0, length: 0, client: 5475, pid:
> 14799, type: 4
>
> Exactly as you see it: with a blank line in between
> When i start the ping_pong command i have these lines at the same rate
> reported by the script (180 lines/second):
>
> 2013-03-29 17:07:50.277003 7f33fe6f3700  0 mds.0.server
> handle_client_file_setlock: start: 2, length: 1, client: 5481, pid:
> 11011, type: 2
>
> 2013-03-29 17:07:50.281279 7f33fe6f3700  0 mds.0.server
> handle_client_file_setlock: start: 1, length: 1, client: 5481, pid:
> 11011, type: 4
>
> 2013-03-29 17:07:50.286643 7f33fe6f3700  0 mds.0.server
> handle_client_file_setlock: start: 0, length: 1, client: 5481, pid:
> 11011, type: 2
>
> Finally, I've tried to lower the ctdb's RecoverBanPeriod but the
> clients was unable to recover for 5 minutes (again!)
> So, I've found the mds logging this:
> 2013-03-29 16:55:23.354854 7f33fc4ed700  0 log [INF] : closing stale
> session client.5475 192.168.130.11:0/580042840 after 300.159862
>
> I hope to find a solution.
> I am at your disposal to further investigation
>
> --
> Marco Aroldi
>
> 2013/3/29 ronnie sahlberg :
>> The ctdb package comes with a tool "ping pong" that is used to test
>> and exercise fcntl() locking.
>>
>> I think a good test is using this tool and then randomly powercycling
>> nodes in your fs cluster
>> making sure that
>> 1, fcntl() locking is still coherent and correct
>> 2, always recover

[ceph-users] One PG active+clean+inconsistent and repair says object size mismatch

2013-03-29 Thread Wido den Hollander 

Hi,

I'd assume more people are going to encounter this, so I thought an 
e-mail to the ceph-users list would be best.


On a cluster I have one PG which is active+clean+inconsistent.

I tried this:
$ ceph pg repair 2.6a5

In my logs it showed:

2013-03-29 20:27:07.177416 osd.4 [ERR] repair 2.6a5 
93f8cea5/rb.0.2.0251/head//2 on disk size (2097152) does not 
match object info size (4194304)

2013-03-29 20:27:07.177869 osd.4 [ERR] 2.6a5 repair 1 errors, 0 fixed

osd.4, osd.22 and osd.39 are acting where 4 is primary.

On osd.4 I verified that the on-disk size of the object is indeed 
2097152 bytes.


However, on osd.22 and osd.39 the object 
rb.0.2.0251__head_93F8CEA5__2 is also 2097152 big.


According to the PG this object should be exactly 4MB big, but is that 
correct? I can't verify if it should really have been that size, since 
it could be a filesystem which only partially wrote to that object.


A stat() tells me that the last change of this file was 2012-10-17, so 
it can't be due to a recent change to the file/object.


My initial idea was to copy the object to osd.4 from one of the other 
OSDs, but the md5sum is the same on all 3 OSDs.


So my question is, why is this PG inconsistent? This object is the only 
object in that PG, so it has to be the issue.


I'm running 0.56.4 with the 3.8 kernel with btrfs.

--
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] One PG active+clean+inconsistent and repair says object size mismatch

2013-03-29 Thread Sage Weil 
On Fri, 29 Mar 2013, Wido den Hollander wrote:
> Hi,
> 
> I'd assume more people are going to encounter this, so I thought an e-mail to
> the ceph-users list would be best.
> 
> On a cluster I have one PG which is active+clean+inconsistent.
> 
> I tried this:
> $ ceph pg repair 2.6a5
> 
> In my logs it showed:
> 
> 2013-03-29 20:27:07.177416 osd.4 [ERR] repair 2.6a5
> 93f8cea5/rb.0.2.0251/head//2 on disk size (2097152) does not match
> object info size (4194304)
> 2013-03-29 20:27:07.177869 osd.4 [ERR] 2.6a5 repair 1 errors, 0 fixed
> 
> osd.4, osd.22 and osd.39 are acting where 4 is primary.
> 
> On osd.4 I verified that the on-disk size of the object is indeed 2097152
> bytes.
> 
> However, on osd.22 and osd.39 the object rb.0.2.0251__head_93F8CEA5__2
> is also 2097152 big.
> 
> According to the PG this object should be exactly 4MB big, but is that
> correct? I can't verify if it should really have been that size, since it
> could be a filesystem which only partially wrote to that object.
> 
> A stat() tells me that the last change of this file was 2012-10-17, so it
> can't be due to a recent change to the file/object.
> 
> My initial idea was to copy the object to osd.4 from one of the other OSDs,
> but the md5sum is the same on all 3 OSDs.
> 
> So my question is, why is this PG inconsistent? This object is the only object
> in that PG, so it has to be the issue.
> 
> I'm running 0.56.4 with the 3.8 kernel with btrfs.

At some point in the past, the on-disk object size got out of sync with 
the metadata.  When we've seen this, its usually been due to crashes and 
journal replay issues (not btrfs!), although there have been many other 
fixes in the argonaut -> bobtail timeframe that may have caused this.  Is 
this the first scrub in a while?  Or was the cluster doing a lot of 
recovery/rebalancing recently?  It sounds like this happened long ago but 
was just now noticed.

Usually a bug triggers on one copy, and then the change get propagated to 
the others during recovery.  If there was recent activity, it may be a 
problem that is still present...

In any case, the simplest way to repair is to truncate --size 4194304 
 on all of the replicas.

sage



> 
> -- 
> Wido den Hollander
> 42on B.V.
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] One PG active+clean+inconsistent and repair says object size mismatch

2013-03-29 Thread Wido den Hollander 

On 03/29/2013 08:56 PM, Sage Weil wrote:

On Fri, 29 Mar 2013, Wido den Hollander wrote:

Hi,

I'd assume more people are going to encounter this, so I thought an e-mail to
the ceph-users list would be best.

On a cluster I have one PG which is active+clean+inconsistent.

I tried this:
$ ceph pg repair 2.6a5

In my logs it showed:

2013-03-29 20:27:07.177416 osd.4 [ERR] repair 2.6a5
93f8cea5/rb.0.2.0251/head//2 on disk size (2097152) does not match
object info size (4194304)
2013-03-29 20:27:07.177869 osd.4 [ERR] 2.6a5 repair 1 errors, 0 fixed

osd.4, osd.22 and osd.39 are acting where 4 is primary.

On osd.4 I verified that the on-disk size of the object is indeed 2097152
bytes.

However, on osd.22 and osd.39 the object rb.0.2.0251__head_93F8CEA5__2
is also 2097152 big.

According to the PG this object should be exactly 4MB big, but is that
correct? I can't verify if it should really have been that size, since it
could be a filesystem which only partially wrote to that object.

A stat() tells me that the last change of this file was 2012-10-17, so it
can't be due to a recent change to the file/object.

My initial idea was to copy the object to osd.4 from one of the other OSDs,
but the md5sum is the same on all 3 OSDs.

So my question is, why is this PG inconsistent? This object is the only object
in that PG, so it has to be the issue.

I'm running 0.56.4 with the 3.8 kernel with btrfs.


At some point in the past, the on-disk object size got out of sync with
the metadata.  When we've seen this, its usually been due to crashes and
journal replay issues (not btrfs!), although there have been many other
fixes in the argonaut -> bobtail timeframe that may have caused this.  Is
this the first scrub in a while?  Or was the cluster doing a lot of
recovery/rebalancing recently?  It sounds like this happened long ago but
was just now noticed.


The cluster was indeed doing a lot of rebalancing and recovery lately. 
Some OSDs have been out and down for quite some time in this cluster.




Usually a bug triggers on one copy, and then the change get propagated to
the others during recovery.  If there was recent activity, it may be a
problem that is still present...

In any case, the simplest way to repair is to truncate --size 4194304
 on all of the replicas.



Yes, that worked. I ran the truncate for all 3 files and then issues a 
pg repair. That resolved it.


Thanks!

Wido


sage





--
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] I/O Speed Comparisons

2013-03-29 Thread Josh Durgin 

On 03/19/2013 07:06 PM, Josh Durgin wrote:  

On 03/18/2013 07:53 AM, Wolfgang Hennerbichler wrote:



On 03/13/2013 06:38 PM, Josh Durgin wrote:

Anyone seeing this problem, could you try the wip-rbd-cache-aio branch?


Hi,

just compiled and tested it out, unfortunately there's no big change:

  ceph --version
ceph version 0.58-375-ga4a6075 (a4a60758b7a10d51419c1961bb12f26b87672fd5)

libvirt-config:

 
   
   
 
   
   
   
 

within the VM:
cat /dev/zero > /bigfile

again trashes the virtual machine.

reading seems quick(er) though (i can't go back to bobtail without some
hassles on this machine (leveldb on monitor & such), so this is just a
guess):
dd if=/dev/vda of=/dev/null
... 82.0 MB/s

One thing that I've noticed is that the virsh start command returns very
quickly, this used to take longer on bobtail.


Thanks,
Josh


you're welcome. I'm really eager helping you out now, I've got a
testbed, if you apply a patch in git I can probably test within 24 hours.

Wolfgang


Thanks for trying it out. Yesterday I managed to reliably reproduce the
problem, but I'm still looking for the root cause. It seems there's
some subtle interaction causing the issue, since even when the cache is
write-through or flushes data immediately (i.e. not acting as a write
cache), the problem persists. I'll update
http://tracker.ceph.com/issues/3737 when I've got more information.

Josh


The issue was that the qemu rbd driver was blocking the main qemu
thread when flush was called, since it was using a synchronous flush.
Fixing this involves patches to librbd to add an asynchronous flush,
and a patch to qemu to use it.

The ceph patches are in the next and master branches, and will be
backported to bobtail. The patch for qemu is at [1] if anyone wants to
try it out.

Josh

[1] http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05367.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Object location

2013-03-29 Thread Dan Mick 
so "rados -p  put  " is one way to create an object. 
There are also C/C++/Python librados bindings for writing programs to do so.


On 03/28/2013 05:30 AM, Waed Bataineh wrote:

Object Store.


On Thu, Mar 28, 2013 at 11:51 AM, Sebastien Han
mailto:sebastien@enovance.com>> wrote:

Hi,

It depends, what do you to use from Ceph? Object store? Block
device? Distributed filesystem?


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."









PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
EMAIL : sebastien@enovance.com
 – SKYPE : han.sbastien
ADDRESS : 10, rue de la Victoire – 75009 Paris
WEB : www.enovance.com  – TWITTER : @enovance

On Mar 28, 2013, at 9:31 AM, Waed Bataineh mailto:promiselad...@gmail.com>> wrote:


Thank you both.

well , no more error message :)

If i may, i still have some confusion about how to start operating
on ceph!

_how can i start storing files that on the client at the server?
correct me if i'm wrong, at client side i'll save the files i need
to store at the mount point, then the server should deal with this
files and start ceph work!

_in case that what is really going, how i can get to the path the
files have been stored in? this is actually my major problem, dont
know the path that lead me to the files or objects!

Hope I'm making my self clear!

Thank you again.


On Thu, Mar 28, 2013 at 3:20 AM, John
Wilkins mailto:john.wilk...@inktank.com>> wrote:
Already done. Missed it at first. Thanks! :)


On Wed, Mar 27, 2013 at 5:17 PM, Sebastien
Han mailto:sebastien@enovance.com>> wrote:
Hi John,

Could you also please update the generic usage at the beginning of
the section?

Thanks!

ps: I tried to push a change, but something went wrong with Github...


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."









PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
EMAIL : sebastien@enovance.com
 – SKYPE : han.sbastien
ADDRESS : 10, rue de la Victoire – 75009 Paris
WEB : www.enovance.com  – TWITTER : @enovance

On Mar 28, 2013, at 12:52 AM, John Wilkins
mailto:john.wilk...@inktank.com>> wrote:


Sebastian,

You're correct. The usage has changed, so I've updated the doc to
reflect it. I ran through the procedure and it worked fine for
me. Sorry Waed. I hope that's all that was wrong.

On Wed, Mar 27, 2013 at 3:48 PM, Sebastien Han
mailto:sebastien@enovance.com>>
wrote:
Arf sorry, not 'odd' but 'osd' of course. (thanks autocompletion…)


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."









PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
EMAIL : sebastien@enovance.com
 – SKYPE : han.sbastien
ADDRESS : 10, rue de la Victoire – 75009 Paris
WEB : www.enovance.com  – TWITTER :
@enovance

On Mar 27, 2013, at 11:36 PM, Sebastien Han
mailto:sebastien@enovance.com>>
wrote:


Ok,

I just noticed that the documentation seems to be wrong, the
correct command to find the location of an object is:

$ ceph odd map  

Then, the error that you raised is pretty strange because even
the object doesn't exist, the command will calculate the
eventual location.

Could you please paste _all_ the steps you've made to get this
error?

Thanks.


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."










PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
EMAIL : sebastien@enovance.com
 – SKYPE : han.sbastien
ADDRESS : 10, rue de la Victoire – 75009 Paris
WEB : www.enovance.com  – TWITTER :
@enovance

On Mar 26, 2013, at 1:36 PM, Waed Bataineh
mailto:promiselad...@gmail.com>> wrote:


Pool obj_name does not exist.


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com 
(415) 425-9599
http://inktank.com





--
John Wilkins
Senior Technical Writer
Intank
john.wilk...@inktank.com

Re: [ceph-users] Object location

2013-03-29 Thread Waed Bataineh 
Dan,

well exactly if i used the mapping command line i have confusion about some
issues if i may :
_ exactly what it meant by obj in * rados -p  put  * , is
this obj consider like the first part of the file and pointing  to the
other object?
_how i can get the file back ?
_Finally, if i have two OSDs n no replica when do the mapping it always
store at osd.0, is there a way to mak it store on osd.1 or it will continue
till osd.0 will reach nearfull ?

Thank you.


On Sat, Mar 30, 2013 at 7:15 AM, Dan Mick  wrote:

> so "rados -p  put  " is one way to create an object.
> There are also C/C++/Python librados bindings for writing programs to do so.
>
>
> On 03/28/2013 05:30 AM, Waed Bataineh wrote:
>
>> Object Store.
>>
>>
>> On Thu, Mar 28, 2013 at 11:51 AM, Sebastien Han
>> > >
>> wrote:
>>
>> Hi,
>>
>> It depends, what do you to use from Ceph? Object store? Block
>> device? Distributed filesystem?
>>
>> 
>> Sébastien Han
>> Cloud Engineer
>>
>> "Always give 100%. Unless you're giving blood."
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
>> EMAIL : sebastien@enovance.com
>>  –
>> SKYPE : han.sbastien
>>
>> ADDRESS : 10, rue de la Victoire – 75009 Paris
>> WEB : www.enovance.com  – TWITTER :
>> @enovance
>>
>>
>> On Mar 28, 2013, at 9:31 AM, Waed Bataineh > > wrote:
>>
>>  Thank you both.
>>>
>>> well , no more error message :)
>>>
>>> If i may, i still have some confusion about how to start operating
>>> on ceph!
>>>
>>> _how can i start storing files that on the client at the server?
>>> correct me if i'm wrong, at client side i'll save the files i need
>>> to store at the mount point, then the server should deal with this
>>> files and start ceph work!
>>>
>>> _in case that what is really going, how i can get to the path the
>>> files have been stored in? this is actually my major problem, dont
>>> know the path that lead me to the files or objects!
>>>
>>> Hope I'm making my self clear!
>>>
>>> Thank you again.
>>>
>>>
>>> On Thu, Mar 28, 2013 at 3:20 AM, John
>>> Wilkins >> >
>>> wrote:
>>> Already done. Missed it at first. Thanks! :)
>>>
>>>
>>> On Wed, Mar 27, 2013 at 5:17 PM, Sebastien
>>> Han >> >
>>> wrote:
>>> Hi John,
>>>
>>> Could you also please update the generic usage at the beginning of
>>> the section?
>>>
>>> Thanks!
>>>
>>> ps: I tried to push a change, but something went wrong with Github...
>>>
>>> 
>>> Sébastien Han
>>> Cloud Engineer
>>>
>>> "Always give 100%. Unless you're giving blood."
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
>>> EMAIL : sebastien@enovance.com
>>> 
>>> – SKYPE : han.sbastien
>>>
>>> ADDRESS : 10, rue de la Victoire – 75009 Paris
>>> WEB : www.enovance.com  – TWITTER :
>>> @enovance
>>>
>>>
>>> On Mar 28, 2013, at 12:52 AM, John Wilkins
>>> >> >
>>> wrote:
>>>
>>>  Sebastian,

 You're correct. The usage has changed, so I've updated the doc to
 reflect it. I ran through the procedure and it worked fine for
 me. Sorry Waed. I hope that's all that was wrong.

 On Wed, Mar 27, 2013 at 3:48 PM, Sebastien Han
 >>> >

 wrote:
 Arf sorry, not 'odd' but 'osd' of course. (thanks autocompletion…)

 
 Sébastien Han
 Cloud Engineer

 "Always give 100%. Unless you're giving blood."









 PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
 EMAIL : sebastien@enovance.com
 
 – SKYPE : han.sbastien

 ADDRESS : 10, rue de la Victoire – 75009 Paris
 WEB : www.enovance.com  – TWITTER :

 @enovance

 On Mar 27, 2013, at 11:36 PM, Sebastien Han
 >>> >
 wrote:

  Ok,
>
> I just noticed that the documentation seems to be wrong, the
> correct command to find the location of an object is:
>
> $ ceph odd map  
>
> Then, the error that you raised is pretty strange because even
> the object doesn't exist, the command will calculate the
> eventual location.
>
> Could you please paste _all_ the steps you've made to get this
> error?
>
> Thanks.
>
> 
> Sébas

Re: [ceph-users] Object location

2013-03-29 Thread Dan Mick 


On Mar 29, 2013, at 9:32 PM, Waed Bataineh  wrote:

> Dan,
> 
> well exactly if i used the mapping command line i have confusion about some 
> issues if i may :
> _ exactly what it meant by obj in  rados -p  put   , is this 
> obj consider like the first part of the file and pointing  to the other 
> object?

This creates an object named  in the cluster that contains the contents of 
.  See the rados command help or manpage or docs. 

> _how i can get the file back ?

rados has another subcommand 'get' 

> _Finally, if i have two OSDs n no replica when do the mapping it always store 
> at osd.0, is there a way to mak it store on osd.1 or it will continue till 
> osd.0 will reach nearfull ? 

Objects are stored in places that depend on their names. If you create enough 
objects with different names you will see them distribute across the OSDs. 

> 
> Thank you.
> 
> 
> On Sat, Mar 30, 2013 at 7:15 AM, Dan Mick  wrote:
>> so "rados -p  put  " is one way to create an object. There 
>> are also C/C++/Python librados bindings for writing programs to do so.
>> 
>> 
>> On 03/28/2013 05:30 AM, Waed Bataineh wrote:
>>> Object Store.
>>> 
>>> 
>>> On Thu, Mar 28, 2013 at 11:51 AM, Sebastien Han
>>> mailto:sebastien@enovance.com>> wrote:
>>> 
>>> Hi,
>>> 
>>> It depends, what do you to use from Ceph? Object store? Block
>>> device? Distributed filesystem?
>>> 
>>> 
>>> Sébastien Han
>>> Cloud Engineer
>>> 
>>> "Always give 100%. Unless you're giving blood."
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
>>> EMAIL : sebastien@enovance.com
>>>  – SKYPE : han.sbastien
>>> 
>>> ADDRESS : 10, rue de la Victoire – 75009 Paris
>>> WEB : www.enovance.com  – TWITTER : @enovance
>>> 
>>> 
>>> On Mar 28, 2013, at 9:31 AM, Waed Bataineh >> > wrote:
>>> 
 Thank you both.
 
 well , no more error message :)
 
 If i may, i still have some confusion about how to start operating
 on ceph!
 
 _how can i start storing files that on the client at the server?
 correct me if i'm wrong, at client side i'll save the files i need
 to store at the mount point, then the server should deal with this
 files and start ceph work!
 
 _in case that what is really going, how i can get to the path the
 files have been stored in? this is actually my major problem, dont
 know the path that lead me to the files or objects!
 
 Hope I'm making my self clear!
 
 Thank you again.
 
 
 On Thu, Mar 28, 2013 at 3:20 AM, John
 Wilkins >>> > wrote:
 Already done. Missed it at first. Thanks! :)
 
 
 On Wed, Mar 27, 2013 at 5:17 PM, Sebastien
 Han >>> > wrote:
 Hi John,
 
 Could you also please update the generic usage at the beginning of
 the section?
 
 Thanks!
 
 ps: I tried to push a change, but something went wrong with Github...
 
 
 Sébastien Han
 Cloud Engineer
 
 "Always give 100%. Unless you're giving blood."
 
 
 
 
 
 
 
 
 
 PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
 EMAIL : sebastien@enovance.com
  – SKYPE : han.sbastien
 
 ADDRESS : 10, rue de la Victoire – 75009 Paris
 WEB : www.enovance.com  – TWITTER : @enovance
 
 
 On Mar 28, 2013, at 12:52 AM, John Wilkins
 mailto:john.wilk...@inktank.com>> wrote:
 
> Sebastian,
> 
> You're correct. The usage has changed, so I've updated the doc to
> reflect it. I ran through the procedure and it worked fine for
> me. Sorry Waed. I hope that's all that was wrong.
> 
> On Wed, Mar 27, 2013 at 3:48 PM, Sebastien Han
> mailto:sebastien@enovance.com>>
> 
> wrote:
> Arf sorry, not 'odd' but 'osd' of course. (thanks autocompletion…)
> 
> 
> Sébastien Han
> Cloud Engineer
> 
> "Always give 100%. Unless you're giving blood."
> 
> 
> 
> 
> 
> 
> 
> 
> 
> PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
> EMAIL : sebastien@enovance.com
>  – SKYPE : han.sbastien
> 
> ADDRESS : 10, rue de la Victoire – 75009 Paris
> WEB : www.enovance.com  – TWITTER :
> 
> @enovance
> 
> On Mar 27, 2013, at 11:36 PM, Sebastien Han
> mailto:sebastien@enovance.com>>
>