Re: [ceph-users] mon leveldb loss

[Sebastien Han]

Hi Mike,

Sorry to hear that, I hope this can help you to recover your RBD images:
http://www.sebastien-han.fr/blog/2015/01/29/ceph-recover-a-rbd-image-from-a-dead-cluster/

Since you don’t have your monitors, you can still walk through the OSD data dir 
and look for the rbd identifiers.
Something like this might help:

 sudo find /var/lib/ceph/osd/ -type f -name rbd*data.* | cut -d'.' -f 3 | sort 
| uniq

Hope it helps.


> On 29 Jan 2015, at 21:36, Mike Winfield  
> wrote:
> 
> Hi, I'm hoping desperately that someone can help. I have a critical issue 
> with a tiny 'cluster'...
> 
> There was a power glitch earlier today (not an outage, might have been a 
> brownout, some things went down, others didn't) and i came home to a CPU 
> machine check exception on the singular host on which i keep a trio of ceph 
> monitors. No option but to hard reset. When the system came back up, the 
> monitors didn't.
> 
> Each mon is reporting possible corruption of their leveldb stores, files are 
> missing, one might surmise an fsck decided to discard them. See attached txt 
> files for ceph-mon output and corresponding store.db directory listings.
> 
> Is there any way to recover the leveldb for the monitors? I am more than 
> capable and willing to dig into the structure of these files - or any similar 
> measures - if necessary. Perhaps correlate a compete picture between the data 
> files that are available?
> 
> I do have a relevant backup of the monitor data but it is now three months 
> old. I would prefer not to have to resort to this if there is any chance of 
> recovering monitor operability by other means.
> 
> Also, what would the consequences be of restoring such a backup when the 
> (12TB worth of) osd's are perfectly fine and contain the latest up-to-date pg 
> associations? Would there be a risk of data loss?
> 
> Unfortunately i don't have any backups of the actual user data (being poor, 
> scraping along on a shoestring budget, not exactly conducive to anything 
> approaching an ideal hardware setup), unless one counts a set of old disks 
> from a previously failed cluster from six months ago.
> 
> My last recourse will likely be to try to scavenge and piece together my most 
> important files from whatever i find on the osd's. Far from an exciting 
> prospect but i am seriously desperate.
> 
> I would be terribly grateful for any input.
> 
> Mike
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Cheers.

Sébastien Han
Cloud Architect

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72
Mail: sebastien@enovance.com
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] No auto-mount of OSDs after server reboot

[Alexis KOALLA]

Hi Lindsay and Daniel
Thanks for your replies.
Apologize for not specifying my LAB env details :
Here is the details:

OS: Ubuntu 14.04 LTS, Kernel 3.8.0-29-generic
Ceph version: Firefly 0.80.8
env: LAB

@Lindsay : I'm wonderring if putting the mount command in fstab is new 
to ceph or it is recommended since the beginning of ceph.
  Anyway I plan to  copy-paste the mount commands from  
/etc/mtab to /etc/fstab and I hope  the issue will  be fixed.


@Daniel:  I have checked in the top level of each OSDs but there is no 
file name "sysvinit" in the directory where the "whoami" file is located.
Should I create it manually or there is way to 
auto-generate this file please.


Thanks for your help
Best and kindest regards

Alexis



Le 29/01/2015 15:11, Lindsay Mathieson a écrit :

On Thu, 29 Jan 2015 03:05:41 PM Alexis KOALLA wrote:

Hi,
Today we  encountered an issue in  our Ceph cluster in  LAB.
Issue: The servers that host the OSDs have rebooted and we have observed
that after the reboot there is no auto mount of OSD devices and we need
to manually performed the mount and then start the OSD as below:

1- [root@osd.0] mount /dev/sdb2 /var/lib/ceph/osd/ceph-0
2- [root@osd.0] start ceph-osd id=0


As far as I'm aware, ceph does not handle mounting of the base filesystem - its
up to you to create an fstab entry for it.

The osd should autostart, but it will of course fail if the filesystem is not
mounted.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] No auto-mount of OSDs after server reboot

[James Eckersall]

I'm running Ubuntu 14.04 servers with Firefly and I don't have a sysvinit
file, but I do have an upstart file.

"touch /var/lib/ceph/osd/ceph-XX/upstart" should be all you need to do.
That way, the OSD's should be mounted automatically on boot.

On 30 January 2015 at 10:25, Alexis KOALLA  wrote:

> Hi Lindsay and Daniel
> Thanks for your replies.
> Apologize for not specifying my LAB env details :
> Here is the details:
>
> OS: Ubuntu 14.04 LTS, Kernel 3.8.0-29-generic
> Ceph version: Firefly 0.80.8
> env: LAB
>
> @Lindsay : I'm wonderring if putting the mount command in fstab is new to
> ceph or it is recommended since the beginning of ceph.
>   Anyway I plan to  copy-paste the mount commands from
> /etc/mtab to /etc/fstab and I hope  the issue will  be fixed.
>
> @Daniel:  I have checked in the top level of each OSDs but there is no
> file name "sysvinit" in the directory where the "whoami" file is located.
> Should I create it manually or there is way to
> auto-generate this file please.
>
> Thanks for your help
> Best and kindest regards
>
> Alexis
>
>
>
> Le 29/01/2015 15:11, Lindsay Mathieson a écrit :
>
>  On Thu, 29 Jan 2015 03:05:41 PM Alexis KOALLA wrote:
>>
>>> Hi,
>>> Today we  encountered an issue in  our Ceph cluster in  LAB.
>>> Issue: The servers that host the OSDs have rebooted and we have observed
>>> that after the reboot there is no auto mount of OSD devices and we need
>>> to manually performed the mount and then start the OSD as below:
>>>
>>> 1- [root@osd.0] mount /dev/sdb2 /var/lib/ceph/osd/ceph-0
>>> 2- [root@osd.0] start ceph-osd id=0
>>>
>>
>> As far as I'm aware, ceph does not handle mounting of the base filesystem
>> - its
>> up to you to create an fstab entry for it.
>>
>> The osd should autostart, but it will of course fail if the filesystem is
>> not
>> mounted.
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs - disabling cache on client and on OSDs

[Mudit Verma]

Hi All,

We are working on a project where we are planning to use Ceph as storage.
However, for one experiment we are required to disable the caching on OSDs
and on client.

We want any data transaction in the filesystem to be served directly from
OSDs disk, without any cache involvement in between.

For example, if we read a file content at client, its objects should be
served directly from the disk storage.

Please let me know if there is any command to any other way to disable the
READ/WRITE cache for data path.

Thanks
Mudit
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] btrfs backend with autodefrag mount option

[Luke Kao]

Dear ceph users,

Has anyone tried to add autodefrag and mount option when use btrfs as the osd 
storage?



In some previous discussion that btrfs osd startup becomes very slow after used 
for some time, just thinking about add autodefrag will help.



We will add on our test cluster first to see if there is any difference.





Please kindly share experience if available, thanks





Luke Kao

MYCOM OSI



This electronic message contains information from Mycom which may be privileged 
or confidential. The information is intended to be for the use of the 
individual(s) or entity named above. If you are not the intended recipient, be 
aware that any disclosure, copying, distribution or any other use of the 
contents of this information is prohibited. If you have received this 
electronic message in error, please notify us by post or telephone (to the 
numbers or correspondence address above) or by email (at the email address 
above) immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] btrfs backend with autodefrag mount option

[Mark Nelson]

oops, mangled the first part of that reply a bit.  Need my morning 
coffee. :)


On 01/30/2015 07:56 AM, Mark Nelson wrote:

About a year ago I was talking to j

On 01/30/2015 07:24 AM, Luke Kao wrote:

Dear ceph users,

Has anyone tried to add autodefrag and mount option when use btrfs as
the osd storage?


Sort of.  About a year ago I was looking into it, but Josef told me not
to use either defrag or autodefrag. (especially when lots of snapshots
are used) There is/was a bug that can make the box go OOM and keel over.
  I think fixing it was on the roadmap but I haven't heard if anything
ever made it in.



In some previous discussion that btrfs osd startup becomes very slow
after used for some time, just thinking about add autodefrag will help.

We will add on our test cluster first to see if there is any difference.

Please kindly share experience if available, thanks


With OSDs on BTRFS, we saw better performance across the board vs XFS
initially on a fresh deploy.  After ~30 minutes of small random writes
to RBD volumes, everything got incredibly fragmented and sequential
reads degraded by about 200%.  Presumably this is due to COW.  Even if
defrag was safe, there'd be a lot of data to clean up...



Luke Kao

MYCOM OSI




This electronic message contains information from Mycom which may be
privileged or confidential. The information is intended to be for the
use of the individual(s) or entity named above. If you are not the
intended recipient, be aware that any disclosure, copying, distribution
or any other use of the contents of this information is prohibited. If
you have received this electronic message in error, please notify us by
post or telephone (to the numbers or correspondence address above) or by
email (at the email address above) immediately.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] btrfs backend with autodefrag mount option

[Mark Nelson]

About a year ago I was talking to j

On 01/30/2015 07:24 AM, Luke Kao wrote:

Dear ceph users,

Has anyone tried to add autodefrag and mount option when use btrfs as
the osd storage?


Sort of.  About a year ago I was looking into it, but Josef told me not 
to use either defrag or autodefrag. (especially when lots of snapshots 
are used) There is/was a bug that can make the box go OOM and keel over. 
 I think fixing it was on the roadmap but I haven't heard if anything 
ever made it in.




In some previous discussion that btrfs osd startup becomes very slow
after used for some time, just thinking about add autodefrag will help.

We will add on our test cluster first to see if there is any difference.

Please kindly share experience if available, thanks


With OSDs on BTRFS, we saw better performance across the board vs XFS 
initially on a fresh deploy.  After ~30 minutes of small random writes 
to RBD volumes, everything got incredibly fragmented and sequential 
reads degraded by about 200%.  Presumably this is due to COW.  Even if 
defrag was safe, there'd be a lot of data to clean up...




Luke Kao

MYCOM OSI




This electronic message contains information from Mycom which may be
privileged or confidential. The information is intended to be for the
use of the individual(s) or entity named above. If you are not the
intended recipient, be aware that any disclosure, copying, distribution
or any other use of the contents of this information is prohibited. If
you have received this electronic message in error, please notify us by
post or telephone (to the numbers or correspondence address above) or by
email (at the email address above) immediately.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] error in sys.exitfunc

[Travis Rhoden]

Hi Karl,

Sorry that I missed this go by.  If you are still hitting this issue,
I'd like to help you and figure this one out, especially since you are
not the only person to have hit it.

Can you pass along your system details, (OS, version, etc.).

I'd also like to know how you installed ceph-deploy (via RPM, or pip?).

 - Travis

On Tue, Jan 20, 2015 at 10:46 AM, Blake, Karl D  wrote:
> Error is same as this posted link -
> http://www.spinics.net/lists/ceph-devel/msg21388.html
>
>
>
> From: Blake, Karl D
> Sent: Tuesday, January 20, 2015 4:29 AM
> To: ceph-us...@ceph.com
> Subject: RE: error in sys.exitfunc
>
>
>
> Please advise.
>
>
>
> Thanks,
>
> -Karl
>
>
>
> From: Blake, Karl D
> Sent: Monday, January 19, 2015 7:23 PM
> To: 'ceph-us...@ceph.com'
> Subject: error in sys.exitfunc
>
>
>
> Anytime I run Ceph-deploy I get the above error. Can you help resolve?
>
>
>
> Thanks,
>
> -Karl
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Moving a Ceph cluster (to a new network)

[Don Doerner]

All,

I built up a ceph system on my little development network, then tried to move 
it to a different network.  I edited the ceph.conf file, and fired it up and... 
well, I discovered that I was a bit naive.

I looked through the documentation pretty carefully, and I can't see any list 
of places that the original network addresses are stashed.  Can anyone point me 
at a procedure for changing network addresses like that?  Or point me at a list 
of what all has to be updated (e.g., I am guessing that my keys are all broken)?

In my case, I could recreate the entire cluster but later, when the OSDs have 
valuable data, that won't be an option.  So I'd like to learn how to do this 
now, when the jeopardy is low...

 
Regards,

-don-___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] btrfs backend with autodefrag mount option

[Lionel Bouton]

On 01/30/15 14:24, Luke Kao wrote:
>
> Dear ceph users,
>
> Has anyone tried to add autodefrag and mount option when use btrfs as
> the osd storage?
>
>  
>
> In some previous discussion that btrfs osd startup becomes very slow
> after used for some time, just thinking about add autodefrag will help.
>
>  
>
> We will add on our test cluster first to see if there is any difference.
>

We used autodefrag but it didn't help: performance degrades over time.
One possibility raised in previous discussions here is that BTRFS's
autodefrag isn't smart enough when snapshots are heavily used as is the
case with Ceph OSD by default.

There are some tunings available that we have yet to test :

filestore btrfs snap
filestore btrfs clone range
filestore journal parallel

All are enabled by default for BTRFS backends. snap is probably the
first you might want to disable and check how autodefrag and defrag
behave. It might be possible to use snap and defrag, BTRFS was quite
stable for us (but all our OSDs are on systems with at least 72GB RAM
which have enough CPU power so memory wasn't much of an issue).

Best regards,

Lionel Bouton
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD caching on 4K reads???

[Bruce McFarland]

I have a cluster and have created a rbd device - /dev/rbd1. It shows up as 
expected with 'rbd -image test info' and rbd showmapped. I have been looking at 
cluster performance with the usual Linux block device tools - fio and vdbench. 
When I look at writes and large block sequential reads I'm seeing what I'd 
expect with performance limited by either my cluster interconnect bandwidth or 
the backend device throughput speeds - 1 GE frontend and cluster network and 
7200rpm SATA OSDs with 1 SSD/osd for journal. Everything looks good EXCEPT 4K 
random reads. There is caching occurring somewhere in my system that I haven't 
been able to detect and suppress - yet.

I've set 'rbd_cache=false' in the [client] section of ceph.conf on the client, 
monitor, and storage nodes. I've flushed the system caches on the client and 
storage nodes before test run ie vm.drop_caches=3 and set the huge pages to the 
maximum available to consume free system memory so that it can't be used for 
system cache . I've also disabled read-ahead on all of the HDD/OSDs.

When I run a 4k randon read workload on the client the most I could expect 
would be ~100iops/osd x number of osd's - I'm seeing an order of magnitude 
greater than that AND running IOSTAT on the storage nodes show no read activity 
on the OSD disks.

Any ideas on what I've overlooked? There appears to be some read-ahead caching 
that I've missed.

Thanks,
Bruce
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD caching on 4K reads???

[Udo Lembke]

Hi Bruce,
hmm, sounds for me like the rbd cache.
Can you look, if the cache is realy disabled in the running config with

ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep cache

Udo

On 30.01.2015 21:51, Bruce McFarland wrote:
>
> I have a cluster and have created a rbd device - /dev/rbd1. It shows
> up as expected with ‘rbd –image test info’ and rbd showmapped. I have
> been looking at cluster performance with the usual Linux block device
> tools – fio and vdbench. When I look at writes and large block
> sequential reads I’m seeing what I’d expect with performance limited
> by either my cluster interconnect bandwidth or the backend device
> throughput speeds – 1 GE frontend and cluster network and 7200rpm SATA
> OSDs with 1 SSD/osd for journal. Everything looks good EXCEPT 4K
> random reads. There is caching occurring somewhere in my system that I
> haven’t been able to detect and suppress - yet.
>
>  
>
> I’ve set ‘rbd_cache=false’ in the [client] section of ceph.conf on the
> client, monitor, and storage nodes. I’ve flushed the system caches on
> the client and storage nodes before test run ie vm.drop_caches=3 and
> set the huge pages to the maximum available to consume free system
> memory so that it can’t be used for system cache . I’ve also disabled
> read-ahead on all of the HDD/OSDs.
>
>  
>
> When I run a 4k randon read workload on the client the most I could
> expect would be ~100iops/osd x number of osd’s – I’m seeing an order
> of magnitude greater than that AND running IOSTAT on the storage nodes
> show no read activity on the OSD disks.
>
>  
>
> Any ideas on what I’ve overlooked? There appears to be some read-ahead
> caching that I’ve missed.
>
>  
>
> Thanks,
>
> Bruce
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD caching on 4K reads???

[Bruce McFarland]

The ceph daemon isn't running on the client with the rbd device so I can't 
verify if it's disabled at the librbd level on the client. If you mean on the 
storage nodes I've had some issues dumping the config. Does the rbd caching 
occur on the storage nodes, client, or both?


From: Udo Lembke [mailto:ulem...@polarzone.de]
Sent: Friday, January 30, 2015 1:00 PM
To: Bruce McFarland; ceph-us...@ceph.com
Cc: Prashanth Nednoor
Subject: Re: [ceph-users] RBD caching on 4K reads???

Hi Bruce,
hmm, sounds for me like the rbd cache.
Can you look, if the cache is realy disabled in the running config with

ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep cache

Udo
On 30.01.2015 21:51, Bruce McFarland wrote:
I have a cluster and have created a rbd device - /dev/rbd1. It shows up as 
expected with 'rbd -image test info' and rbd showmapped. I have been looking at 
cluster performance with the usual Linux block device tools - fio and vdbench. 
When I look at writes and large block sequential reads I'm seeing what I'd 
expect with performance limited by either my cluster interconnect bandwidth or 
the backend device throughput speeds - 1 GE frontend and cluster network and 
7200rpm SATA OSDs with 1 SSD/osd for journal. Everything looks good EXCEPT 4K 
random reads. There is caching occurring somewhere in my system that I haven't 
been able to detect and suppress - yet.

I've set 'rbd_cache=false' in the [client] section of ceph.conf on the client, 
monitor, and storage nodes. I've flushed the system caches on the client and 
storage nodes before test run ie vm.drop_caches=3 and set the huge pages to the 
maximum available to consume free system memory so that it can't be used for 
system cache . I've also disabled read-ahead on all of the HDD/OSDs.

When I run a 4k randon read workload on the client the most I could expect 
would be ~100iops/osd x number of osd's - I'm seeing an order of magnitude 
greater than that AND running IOSTAT on the storage nodes show no read activity 
on the OSD disks.

Any ideas on what I've overlooked? There appears to be some read-ahead caching 
that I've missed.

Thanks,
Bruce




___

ceph-users mailing list

ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD caching on 4K reads???

[Udo Lembke]

Hi Bruce,
you can also look on the mon, like
ceph --admin-daemon /var/run/ceph/ceph-mon.b.asok config show | grep cache

(I guess you have an number instead of the .b. )

Udo
On 30.01.2015 22:02, Bruce McFarland wrote:
>
> The ceph daemon isn’t running on the client with the rbd device so I
> can’t verify if it’s disabled at the librbd level on the client. If
> you mean on the storage nodes I’ve had some issues dumping the config.
> Does the rbd caching occur on the storage nodes, client, or both?
>
>  
>
>  
>
> *From:*Udo Lembke [mailto:ulem...@polarzone.de]
> *Sent:* Friday, January 30, 2015 1:00 PM
> *To:* Bruce McFarland; ceph-us...@ceph.com
> *Cc:* Prashanth Nednoor
> *Subject:* Re: [ceph-users] RBD caching on 4K reads???
>
>  
>
> Hi Bruce,
> hmm, sounds for me like the rbd cache.
> Can you look, if the cache is realy disabled in the running config with
>
> ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep cache
>
> Udo
>
> On 30.01.2015 21:51, Bruce McFarland wrote:
>
> I have a cluster and have created a rbd device - /dev/rbd1. It
> shows up as expected with ‘rbd –image test info’ and rbd
> showmapped. I have been looking at cluster performance with the
> usual Linux block device tools – fio and vdbench. When I look at
> writes and large block sequential reads I’m seeing what I’d expect
> with performance limited by either my cluster interconnect
> bandwidth or the backend device throughput speeds – 1 GE frontend
> and cluster network and 7200rpm SATA OSDs with 1 SSD/osd for
> journal. Everything looks good EXCEPT 4K random reads. There is
> caching occurring somewhere in my system that I haven’t been able
> to detect and suppress - yet.
>
>  
>
> I’ve set ‘rbd_cache=false’ in the [client] section of ceph.conf on
> the client, monitor, and storage nodes. I’ve flushed the system
> caches on the client and storage nodes before test run ie
> vm.drop_caches=3 and set the huge pages to the maximum available
> to consume free system memory so that it can’t be used for system
> cache . I’ve also disabled read-ahead on all of the HDD/OSDs.
>
>  
>
> When I run a 4k randon read workload on the client the most I
> could expect would be ~100iops/osd x number of osd’s – I’m seeing
> an order of magnitude greater than that AND running IOSTAT on the
> storage nodes show no read activity on the OSD disks.
>
>  
>
> Any ideas on what I’ve overlooked? There appears to be some
> read-ahead caching that I’ve missed.
>
>  
>
> Thanks,
>
> Bruce
>
>
>
>
> ___
>
> ceph-users mailing list
>
> ceph-users@lists.ceph.com 
>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>  
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] calamari server error 503 detail rpc error lost remote after 10s heartbeat

[Tony]

Hi,

I have ceph giant installed and installed/compiled calamari but getting
"calamari server error 503 detail rpc error lost remote after 10s heartbeat"

It seems calamari doesn't have contact with ceph for some reason.

Anyway to configure  calamari manually to get status and fix the 503 error?

I'm thinking it's a rights issue with ceph and the calamari interface?

I'm running under RHEL6, Ceph Giant and current calamari from github.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs - disabling cache on client and on OSDs

[Gregory Farnum]

I don't think there's any way to force the OSDs to do that. What
exactly are you trying to do?
-Greg

On Fri, Jan 30, 2015 at 4:02 AM, Mudit Verma  wrote:
> Hi All,
>
> We are working on a project where we are planning to use Ceph as storage.
> However, for one experiment we are required to disable the caching on OSDs
> and on client.
>
> We want any data transaction in the filesystem to be served directly from
> OSDs disk, without any cache involvement in between.
>
> For example, if we read a file content at client, its objects should be
> served directly from the disk storage.
>
> Please let me know if there is any command to any other way to disable the
> READ/WRITE cache for data path.
>
> Thanks
> Mudit
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] No auto-mount of OSDs after server reboot

[Anthony D'Atri]

One thing than can cause this is messed-up partition ID's / typecodes.   Check 
out the ceph-disk script to see how they get applied.  I have a few systems 
that somehow got messed up -- at boot they don't get started, but if I mounted 
them manually on /mnt, checked out the whoami file and remounted accordingly, 
then started, they ran fine.

# for i in b c d e f g h i j k ; do sgdisk 
--typecode=1:4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D /dev/sd$i ; done

# for i in b c d e f g h i j k ; do sgdisk 
--typecode=2:45B0969E-9B03-4F30-B4C6-B4B80CEFF106 /dev/sd$i ; done

One system I botched and set all the GUID's to a constant; I went back and 
fixed that:

# for i in b c d e f g h i j k ; do sgdisk 
--typecode=2:45B0969E-9B03-4F30-B4C6-B4B80CEFF106 --partition-guid=$(uuidgen 
-r) /dev/sd$i ; done

Note that I have not yet rebooted these systems to validate this approach, so 
YMMV, proceed at your own risk, this advice is not FDIC-insured and may lose 
value.


# sgdisk -i 1 /dev/sdb
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 61397DDD-E203-4D9A-9256-24E0F5F97344
First sector: 20973568 (at 10.0 GiB)
Last sector: 5859373022 (at 2.7 TiB)
Partition size: 5838399455 sectors (2.7 TiB)
Attribute flags: 
Partition name: 'ceph data'

# sgdisk -i 2 /dev/sdb
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: EF292AB7-985E-40A2-B185-DD5911D17BD7
First sector: 2048 (at 1024.0 KiB)
Last sector: 20971520 (at 10.0 GiB)
Partition size: 20969473 sectors (10.0 GiB)
Attribute flags: 
Partition name: 'ceph journal'

--aad


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] btrfs backend with autodefrag mount option

[Luke Kao]

Thanks Lionel, we are using btrfs compression and it's also stable in our 
cluster.

Currently another minor problem of btrfs fragments is sometimes we see 
btrfs-transacti process can pause the whole OSD node I/O for seconds, impacting 
all OSDs on the server.   Especially when doing recovery / backfill.

However, I wonder restart a OSD takes 30minutes may become a problem for 
maintenance.

I will share if we have any result on testing different settings.


BR,
Luke



From: Lionel Bouton [lionel-subscript...@bouton.name]
Sent: Saturday, January 31, 2015 2:29 AM
To: Luke Kao; ceph-us...@ceph.com
Subject: Re: [ceph-users] btrfs backend with autodefrag mount option

On 01/30/15 14:24, Luke Kao wrote:

Dear ceph users,

Has anyone tried to add autodefrag and mount option when use btrfs as the osd 
storage?



In some previous discussion that btrfs osd startup becomes very slow after used 
for some time, just thinking about add autodefrag will help.



We will add on our test cluster first to see if there is any difference.

We used autodefrag but it didn't help: performance degrades over time. One 
possibility raised in previous discussions here is that BTRFS's autodefrag 
isn't smart enough when snapshots are heavily used as is the case with Ceph OSD 
by default.

There are some tunings available that we have yet to test :

filestore btrfs snap
filestore btrfs clone range
filestore journal parallel



All are enabled by default for BTRFS backends. snap is probably the first you 
might want to disable and check how autodefrag and defrag behave. It might be 
possible to use snap and defrag, BTRFS was quite stable for us (but all our 
OSDs are on systems with at least 72GB RAM which have enough CPU power so 
memory wasn't much of an issue).

Best regards,

Lionel Bouton



This electronic message contains information from Mycom which may be privileged 
or confidential. The information is intended to be for the use of the 
individual(s) or entity named above. If you are not the intended recipient, be 
aware that any disclosure, copying, distribution or any other use of the 
contents of this information is prohibited. If you have received this 
electronic message in error, please notify us by post or telephone (to the 
numbers or correspondence address above) or by email (at the email address 
above) immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs - disabling cache on client and on OSDs

[Mudit Verma]

Hi Greg,

Thanks.

We need end to end (disk-client to disk-OSD) latency/throughput for READs
and WRITEs. Writes can be made Write through but we are having difficulties
with read.

Thanks
Mudit
On 31-Jan-2015 5:03 AM, "Gregory Farnum"  wrote:

> I don't think there's any way to force the OSDs to do that. What
> exactly are you trying to do?
> -Greg
>
> On Fri, Jan 30, 2015 at 4:02 AM, Mudit Verma 
> wrote:
> > Hi All,
> >
> > We are working on a project where we are planning to use Ceph as storage.
> > However, for one experiment we are required to disable the caching on
> OSDs
> > and on client.
> >
> > We want any data transaction in the filesystem to be served directly from
> > OSDs disk, without any cache involvement in between.
> >
> > For example, if we read a file content at client, its objects should be
> > served directly from the disk storage.
> >
> > Please let me know if there is any command to any other way to disable
> the
> > READ/WRITE cache for data path.
> >
> > Thanks
> > Mudit
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com