With the low number of OSDs, you are probably satuarting the disks. Check
with `iostat -xd 2` and see what the utilization of your disks are. A lot
of SSDs don't perform well with Ceph's heavy sync writes and performance is
terrible.
If some of your drives are 100% while others are lower utilizati
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
I recently upgraded a RHCS-3.0 cluster with 4 iGW's to RHCS-3.2 on top of RHEL-
7.6
Big block size performance went from ~350MB/s to about 1100MB/s on each lun,
seen from a VM in vSphere-6.5 with data read from an ssd pool and written to a
hdd pool,
All;
I have a test and demonstration cluster running (3 hosts, MON, MGR, 2x OSD per
host), and I'm trying to add a 4th host for gateway purposes.
The radosgw process keeps dying with:
2019-06-07 15:59:50.700 7fc4ef273780 0 ceph version 14.2.1
(d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus
Hi Sergei,
Please add to your host:
For 64GB RAM, reserve 1GB.
vm.min_free_kbytes = 1048576
For 128GB RAM, reserve 2GB.
vm.min_free_kbytes = 2097152
For 256GB RAM, reserve 3GB.
vm.min_free_kbytes = 3145728
This will prevent to your OSD to use ALL memory of host and OOM act.
Regards
-M
Hi,
My OSD processes are constantly getting killed by OOM killer. My
cluster has 5 servers, each with 18 spinning disks, running 18 OSD
daemons in 48GB of memory.
I was trying to limit OSD cache, according to
http://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/
[osd]
bluest
I think I found source of my problems.
One of my monitors was on problematic disk. It was not responding for few seconds so mon was taken out and in the cluster frequently (few
times a day). I expect, that key exchange was during that time taken with this problematic monitor and was not successf
The truth of the matter is that folks try to boil this down to some kind
of hard and fast rule but it's often not that simple. With our current
default settings for pglog, rocksdb WAL buffers, etc, the OSD basically
needs about 1GB of RAM for bare-bones operation (not under recovery or
extreme
In nautilus min 4GB per disk .
In case of ssd/nvme 6-12GB per disk.
8GB per disk is a good way to get performance
+2/4GB for the OS.
Regards,
Manuel
-Mensaje original-
De: ceph-users En nombre de
jes...@krogh.cc
Enviado el: viernes, 7 de junio de 2019 19:36
Para: Jorge Garcia
CC: cep
> I'm a bit confused by the RAM recommendations for OSD servers. I have
> also seen conflicting information in the lists (1 GB RAM per OSD, 1 GB
> RAM per TB, 3-5 GB RAM per OSD, etc.). I guess I'm a lot better with a
> concrete example:
I think it depends on the usagepattern - the more the better
I'm a bit confused by the RAM recommendations for OSD servers. I have
also seen conflicting information in the lists (1 GB RAM per OSD, 1 GB
RAM per TB, 3-5 GB RAM per OSD, etc.). I guess I'm a lot better with a
concrete example:
Say this is your cluster (using Bluestore):
8 Machines serving
Paul / All
I'm not sure what warning your are referring to, I'm on Nautilus. The point
I'm getting at is if you weight out all OSD on a host with a cluster of 3
OSD hosts with 3 OSD each, crush rule = host, then write to the cluster, it
*should* imo not just say remapped but undersized / degrade
How's iowait look on your disks?
How have you configured your disks and what are your cache settings?
Did you disable cstates?
On Friday, June 7, 2019, Stolte, Felix wrote:
> Hi Sinan,
>
> thanks for the numbers. I am a little bit surprised that your SSD pool
has nearly the same stats as you SA
On Fri, Jun 7, 2019 at 7:22 AM Sakirnth Nagarasa
wrote:
>
> On 6/6/19 5:09 PM, Jason Dillaman wrote:
> > On Thu, Jun 6, 2019 at 10:13 AM Sakirnth Nagarasa
> > wrote:
> >>
> >> On 6/6/19 3:46 PM, Jason Dillaman wrote:
> >>> Can you run "rbd trash ls --all --long" and see if your image
> >>> is lis
95% of usage is CephFS. Remaining is split between RGW and RBD.
On Wed, Jun 5, 2019 at 3:05 PM Gregory Farnum wrote:
>
> I think the mimic balancer doesn't include omap data when trying to
> balance the cluster. (Because it doesn't get usable omap stats from
> the cluster anyway; in Nautilus I th
Hi Sinan,
thanks for the numbers. I am a little bit surprised that your SSD pool has
nearly the same stats as you SAS pool.
Nevertheless I would expect our pools to perform like your SAS pool, at least
regarding to writes since all our write ops should be placed on our SSDs. But
since I only
Ok it seems that OSD pools are finally to correctly restored (quite same
amount of objects/datas than before restoration) so that should explain the
situation
I will have to dig why
Vincent
Le ven. 7 juin 2019 à 08:41, Vincent Pharabot
a écrit :
> Hello Cephers,
>
>
>
> I’m trying to understand
On 6/6/19 5:09 PM, Jason Dillaman wrote:
> On Thu, Jun 6, 2019 at 10:13 AM Sakirnth Nagarasa
> wrote:
>>
>> On 6/6/19 3:46 PM, Jason Dillaman wrote:
>>> Can you run "rbd trash ls --all --long" and see if your image
>>> is listed?
>>
>> No, it is not listed.
>>
>> I did run:
>> rbd trash ls --all -
Hi Felix,
I have 2 Pools, a SSD only and a SAS only pool.
SSD pool is spread over 12 OSD servers.
SAS pool is spread over 6 OSD servers.
See results (SSD Only Pool):
# sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
--file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.
Hello Casey
We found something weird during our testing of the
rgw_crypt_default_encryption_key=""xxx" parameter.
s3cms behaves like expected:
s3cmd is then always writing encrypted objects
s3cmd can read encrypted and unencrypted objects
but swift does not support encryption:
swift can read
On Tue, Jun 4, 2019 at 1:39 PM wrote:
> >> Basically they max out at around 1000 IOPS and report 100%
> >> utilization and feel slow.
> >>
> >> Haven't seen the 5200 yet.
>
> Micron 5100s performs wonderfully!
>
> You have to just turn its write cache off:
>
> hdparm -W 0 /dev/sdX
>
> 1000 IOPS m
Hi Sinan,
that would be great. The numbers should differ a lot, since you have an all
flash pool, but it would be interesting, what we could expect from such a
configuration.
Regards
Felix
-
---
On Mon, Jun 3, 2019 at 9:02 AM Hervé Ballans
wrote:
> Hi all,
>
> For information, I updated my Luminous cluster to the latest version
> 12.2.12 two weeks ago and, since then, I no longer encounter any problems
> of inconsistent pgs :)
>
You probably were affected by https://tracker.ceph.com/iss
Hi,
ceph-iscsi 3.0 fixes a lot of problems and limitations of the older gateway.
Best way to run it on Debian/Ubuntu is to build it yourself
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
remapped no longer triggers a health warning in nautilus.
Your data is still there, it's just on the wrong OSD if that OSD is still
up and running.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.cro
Quoting Max Vernimmen (vernim...@textkernel.nl):
> Thank you for the suggestion to use the bitmap allocator. I looked at the
> ceph documentation and could find no mention of this setting. This makes me
> wonder how safe and production ready this setting really is. I'm hesitant
> to apply that to o
On Thu, Jun 6, 2019 at 8:00 PM Sage Weil wrote:
>
> Hello RBD users,
>
> Would you mind running this command on a random OSD on your RBD-oriented
> cluster?
>
> ceph-objectstore-tool \
> --data-path /var/lib/ceph/osd/ceph-NNN \
>
> '["meta",{"oid":"snapmapper","key":"","snapid":0,"hash":2758339
Hi Felix,
I can run your commands inside an OpenStack VM. Tthe storage cluster contains
of 12 OSD servers, holding each 8x 960GB SSD. Luminous FileStore. Replicated 3.
Would it help you to run your command on my cluster?
Sinan
> Op 7 jun. 2019 om 08:52 heeft Stolte, Felix het
> volgende gesc
Hi all,
Just a quick heads up, and maybe a check if anyone else is affected.
After upgrading our MDS's from v12.2.11 to v12.2.12, we started
getting crashes with
/builddir/build/BUILD/ceph-12.2.12/src/mds/MDSRank.cc: 1304:
FAILED assert(session->get_nref() == 1)
I opened a ticket here with
Hi Max,
I don't think this is allocator related issue. The symptoms that
triggered us to start using bitmap allocator over stupid one were:
- write op latency gradually increasing over time (days not hours)
- perf showing significant amount of time spent in allocator related
function
- OSD
Thank you for the suggestion to use the bitmap allocator. I looked at the
ceph documentation and could find no mention of this setting. This makes me
wonder how safe and production ready this setting really is. I'm hesitant
to apply that to our production environment.
If the allocator setting helps
Hi,
I did the talk "200 clusters vs 1 admin" on latest Cephalocon (Barcelona
2019).
There were some questions on the backstage about implementation details,
opensourcing, etc.
I just thought that if anyone has some questions then you may catch me
here via email (bartosz.rabi...@corp.ovh.
31 matches
Mail list logo
