Re: [ceph-users] osd crashes with large object size (>10GB) in luminos Rados

2017-09-26 Thread Alexander Kushnirenko
objects. Objects shouldn’t be > stored as large as that and performance will also suffer. > > > > *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf > Of *Alexander Kushnirenko > *Sent:* 26 September 2017 13:50 > *To:* ceph-users@lists.ceph.com > *Subject:*

[ceph-users] osd crashes with large object size (>10GB) in luminos Rados

2017-09-26 Thread Alexander Kushnirenko
Hello, We successfully use rados to store backup volumes in jewel version of CEPH. Typical volume size is 25-50GB. Backup software (bareos) use Rados objects as backup volumes and it works fine. Recently we tried luminous for the same purpose. In luminous developers reduced osd_max_object_size

Re: [ceph-users] OSD crashes on EC recovery

2016-08-10 Thread Brian Felton
Roeland, We're seeing the same problems in our cluster. I can't offer you a solution that gets the OSD back, but I can tell you what I did to work around it. We're running 5 0.94.6 clusters with 9 nodes / 648 HDD OSDs with a k=7, m=2 erasure coded .rgw.buckets pool. During the backfilling after

[ceph-users] OSD crashes on EC recovery

2016-08-10 Thread Roeland Mertens
Hi, we run a Ceph 10.2.1 cluster across 35 nodes with a total of 595 OSDs, we have a mixture of normally replicated volumes and EC volumes using the following erasure-code-profile: # ceph osd erasure-code-profile get rsk8m5 jerasure-per-chunk-alignment=false k=8 m=5 plugin=jerasure ruleset-fa

Re: [ceph-users] OSD Crashes

2016-05-02 Thread Varada Kari
eboots. > > An idea how to fix this. > > Thanks > Pankaj > > > > -Original Message- > From: Somnath Roy [mailto:somnath@sandisk.com] > Sent: Friday, April 29, 2016 9:03 AM > To: Garg, Pankaj; Samuel Just > Cc: ceph-users@lists.ceph.com > Subje

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj
: Friday, April 29, 2016 9:03 AM To: Garg, Pankaj; Samuel Just Cc: ceph-users@lists.ceph.com Subject: RE: [ceph-users] OSD Crashes Check system log and search for the corresponding drive. It should have the information what is failing.. Thanks & Regards Somnath -Original Message- From:

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Samuel Just
egards > Somnath > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Garg, Pankaj > Sent: Friday, April 29, 2016 8:59 AM > To: Samuel Just > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] OSD Crashes >

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Somnath Roy
muel Just Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] OSD Crashes I can see that. I guess what would that be symptomatic of? How is it doing that on 6 different systems and on multiple OSDs? -Original Message- From: Samuel Just [mailto:sj...@redhat.com] Sent: Friday, April 29, 20

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj
: [ceph-users] OSD Crashes Your fs is throwing an EIO on open. -Sam On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj wrote: > Hi, > > I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 > nodes, each with 12 HDD Drives and 2SSD Drives. All these were > initially runn

Re: [ceph-users] OSD Crashes

2016-04-29 Thread Samuel Just
Your fs is throwing an EIO on open. -Sam On Fri, Apr 29, 2016 at 8:54 AM, Garg, Pankaj wrote: > Hi, > > I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 nodes, > each with 12 HDD Drives and 2SSD Drives. All these were initially running > Hammer, and then were successfully update

[ceph-users] OSD Crashes

2016-04-29 Thread Garg, Pankaj
Hi, I had a fully functional Ceph cluster with 3 x86 Nodes and 3 ARM64 nodes, each with 12 HDD Drives and 2SSD Drives. All these were initially running Hammer, and then were successfully updated to Infernalis (9.2.0). I recently deleted all my OSDs and swapped my drives with new ones on the x86

[ceph-users] OSD crashes when starting

2015-08-07 Thread Gerd Jakobovitsch
Dear all, I got to an unrecoverable crash at one specific OSD, every time I try to restart it. It happened first at firefly 0.80.8, I updated to 0.80.10, but it continued to happen. Due to this failure, I have several PGs down+peering, that won't recover even marking the OSD out. Could som

Re: [ceph-users] OSD crashes

2015-07-22 Thread Alex Gorbachev
We have been error free for almost 3 weeks now. The following settings on all OSD nodes were changed: vm.swappiness=1 vm.min_free_kbytes=262144 My discussion on XFS list is here: http://www.spinics.net/lists/xfs/msg33645.html Thanks, Alex On Fri, Jul 3, 2015 at 6:27 AM, Jan Schermer wrote: >

Re: [ceph-users] OSD crashes

2015-07-03 Thread Alex Gorbachev
Thanks Jan. /proc/sys/vm/min_free_kbytes was set to 32M, I set it to 256M with system having 64 GB RAM. Also my swappiness was set to 0, no problems in lab tests, but I wonder if we hit some limit on 24/7 OSD operation. I will update after some days of running with these parameter. Best regards

Re: [ceph-users] OSD crashes

2015-07-03 Thread Jan Schermer
What’s the value of /proc/sys/vm/min_free_kbytes on your system? Increase it to 256M (better do it if there’s lots of free memory) and see if it helps. It can also be set too high, hard to find any formula how to set it correctly... Jan > On 03 Jul 2015, at 10:16, Alex Gorbachev wrote: > > He

[ceph-users] OSD crashes

2015-07-03 Thread Alex Gorbachev
Hello, we are experiencing severe OSD timeouts, OSDs are not taken out and we see the following in syslog on Ubuntu 14.04.2 with Firefly 0.80.9. Thank you for any advice. Alex Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261899] BUG: unable to handle kernel paging request at 0019001c J