[ceph-users] Performance issues in newly deployed Ceph cluster

[Loschwitz,Martin Gerhard]

Folks,

I am running into a very strange issue with a brand new Ceph cluster during 
initial testing. Cluster
consists of 12 nodes, 4 of them have SSDs only, the other eight have a mixture 
of SSDs and HDDs.
The latter nods are configured so that three or four HDDs use one SSDs for 
their blockdb.

Ceph version is Nautilus.

When writing to the cluster, clients will, in regular intervals, run into I/O 
stall (i.e. writes will take up
to 25 minutes to complete). Deleting RBD Images will often take forever as 
well. After several weeks
of debugging, what I can say from looking at the log files, is that what 
appears to take a lot of time
is writing stuff to OSDs:


"time": "2020-05-20 10:52:23.211006", 
"event": "reached_pg"
},
{
"time": "2020-05-20 10:52:23.211047",
"event": "waiting for ondisk"
},
{
"time": "2020-05-20 10:53:35.369081",
"event": "done"
}

But these machines are I/O idling. there is almost no I/O happening at all 
according to sysstat.
I am slowly growing a bit desperate over this, and hence I wonder whether 
anybody has ever
seen a similar issue? Or are there possibly any tips on where to carry on with 
debugging?

Servers are from Dell with PERC controllers in HBA mode.

The primary purpose of this Ceph cluster is to serve as backing storage for 
OpenStack, and to
this point, I was not able to reproduce the issue with the SSD-only nodes.

Best regards
Martin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Nautilus: (Minority of) OSDs with huge buffer_anon usage - triggering OOMkiller in worst cases.

[aoanla]

Hi Mark, thanks for your efforts on this already. 

I had to wait for my account on tracker.ceph to be approved before I could 
submit the bug - which is here: https://tracker.ceph.com/issues/45706 

Sam
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] dealing with spillovers

[thoralf schulze]

hi there,

trying to get around my head rocksdb spillovers and how to deal with
them … in particular, i have one osds which does not have any pools
associated (as per ceph pg ls-by-osd $osd ), yet it does show up in ceph
health detail as:

 osd.$osd spilled over 2.9 MiB metadata from 'db' device (49 MiB
used of 37 GiB) to slow device

compaction doesn't help. i am well aware of
https://tracker.ceph.com/issues/38745 , yet find it really
counter-intuitive that an empty osd with a more-or-less optimal sized db
volume can't fit its rockdb on the former.

is there any way to repair this, apart from re-creating the osd? fwiw,
dumping the database with

ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-$osd dump >
bluestore_kv.dump

yields a file of less than 100mb in size.

and, while we're at it, a few more related questions:

- am i right to assume that the leveldb and rocksdb arguments to
ceph-kvstore-tool are only relevant for osds with filestore-backend?
- does ceph-kvstore-tool bluestore-kv … also deal with rocksdb-items for
osds with bluestore-backend?

thank you very much & with kind regards,
thoralf.



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: mds container dies during deployment

[Simon Sutter]

Hello,


Didn't read the right one:
https://docs.ceph.com/docs/master/cephadm/install/#deploy-mdss
There it says, how to do it right.

The command I was using, was just to add a mds daemon if you have already one.


Hopes it helps others.

Cheers, Simon


Von: Simon Sutter 
Gesendet: Montag, 25. Mai 2020 16:44:54
An: ceph-users@ceph.io
Betreff: [ceph-users] mds container dies during deployment

Hello everyone


I've got a fresh ceph octopus installation and I'm trying to set up a cephfs 
with erasure code configuration.
The metadata pool was set up as default.
The erasure code pool was set up with this command:
-> ceph osd pool create ec-data_fs 128 erasure default
Enabled overwrites:
-> ceph osd pool set ec-data_fs allow_ec_overwrites true
And create fs:
-> ceph fs new ec-data_fs meta_fs ec-data_fs --force


Then I tried deploying the mds, but this fails:
-> ceph orch daemon add mds ec-data_fs magma01
returns:
-> Deployed mds.ec-data_fs.magma01.ujpcly on host 'magma01'

The mds daemon is not there.

Aparently the container dies without any information, as seen in the journal:

May 25 16:11:56 magma01 podman[9348]: 2020-05-25 16:11:56.670510456 +0200 CEST 
m=+0.186462913 container create 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)
May 25 16:11:56 magma01 systemd[1]: Started 
libpod-conmon-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope.
May 25 16:11:56 magma01 systemd[1]: Started libcontainer container 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.
May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.112182262 +0200 CEST 
m=+0.628134873 container init 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)
May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137011897 +0200 CEST 
m=+0.652964354 container start 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)
May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137110412 +0200 CEST 
m=+0.653062853 container attach 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)
May 25 16:11:57 magma01 systemd[1]: 
libpod-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope: 
Consumed 327ms CPU time
May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.182968802 +0200 CEST 
m=+0.698921275 container died 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)
May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.413743787 +0200 CEST 
m=+0.929696266 container remove 
0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 
(image=docker.io/ceph/ceph:v15, name=competent_cori)

Can someone help me debugging this?

Cheers
Simon

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
hosttech GmbH | Simon Sutter
hosttech.ch

WE LOVE TO HOST YOU.

create your own website!
more information & online-demo: 
www.website-creator.ch>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph client on rhel6?

[Simon Sutter]

Hello again,


I have a new question:
We want to upgrade a server, with an os based on rhel6.

The ceph cluster is atm on octopus.

How can I install the client packages to mount cephfs and do a backup of the 
server?
Is it even possible?

Are the client packages from hammer compatible with the octopus release?


Thanks in advance,

Simon

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs taking too much memory, for buffer_anon

[Mark Nelson]

Hi Harald,


Yeah, I suspect your issue is definitely related to what Adam has been 
investigating. FWIW, we are talking about re-introducing a periodic trim 
in Adam's PR here:



https://github.com/ceph/ceph/pull/35171


That should help on the memory growth side, but if we still have objects 
using huge amounts of memory for metadata (1MB+) it will thrash the the 
onode cache and make everything slow.  Ultimately we still need to find 
the root cause of the large per-object buffer_anon memory usage to 
really fix things.



Mark


On 5/25/20 12:25 PM, Harald Staub wrote:

Hi Mark

Thank you! This is 14.2.8, on Ubuntu Bionic. Some with kernel 4.15, 
some with 5.3, but that does not seem to make a difference here. 
Transparent Huge Pages are not used according to

grep -i AnonHugePages /proc/meminfo

Workload is a mix of OpenStack volumes (replicated) and RGW on EC 8+3. 
EC pool with 1024 PGs, 900M objects.


Around 500 hdd OSDs (4 and 8 TB), 30 ssd OSDs (2 TB). The maximum 
number of PGs per OSD is only 123. The hdd OSDs have DB on SSD, but a 
bit less than 30 GB unfortunately. I have seen 200 GB and more 
slow_bytes, compression of the DB seems to help a lot.


No BlueStore compression.

I had a look at the related thread:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JQ72K5LK3YFFETNNL4MX6HHZLF5GBYDT/ 



Today I saw a correlation that may match your thoughts. During 1 hour 
with a high number of write IOPS (not throughput) on the EC pool, 
available memory increased drastically.


Cheers
 Harry

On 20.05.20 15:15, Mark Nelson wrote:

Hi Harald,


Thanks!  So you can see from the perf dump that the target bytes are 
a little below 4GB, but the mapped bytes are around 7GB. The priority 
cache manager has reacted by setting the "cache_bytes" to 128MB which 
is the minimum global value and each cache is getting 64MB (the local 
minimum value per cache). Ultimately this means the priority cache 
manager has basically told all of the caches to shrink to their 
smallest possible values so it's doing the right thing.  So the next 
question is why buffer_anon is so huge. Looking at the mempool stats, 
there are not that many items but still a lot of memory used.  On 
average those items in buffer_anon are ~150K.  It can't be just 
buffer anon though, you've got several gigabytes of mapped memory 
being used beyond that and around 4GB of unmapped memory that 
tcmalloc should be freeing every iteration of the priority cache 
manager.



So next questions:  What version of Ceph is this, and do you have 
transparent huge pages enabled? We automatically disable it now, but 
if you are running an older version you might want to disable (or at 
least set it to madvise) manually.  Also, what kind of workload is 
hitting the OSDs?  If you can reliably make it grow you could try 
doing a heap profile at the same time the workload is going on and 
see if you can see where the memory is being used.



Mark


On 5/20/20 7:36 AM, Harald Staub wrote:

Hi Mark

Thank you for you explanations! Some numbers of this example osd below.

Cheers
 Harry

From dump mempools:

    "buffer_anon": {
    "items": 29012,
    "bytes": 4584503367
    },

From perf dump:

    "prioritycache": {
    "target_bytes": 3758096384,
    "mapped_bytes": 7146692608,
    "unmapped_bytes": 3825983488,
    "heap_bytes": 10972676096,
    "cache_bytes": 134217728
    },
    "prioritycache:data": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "prioritycache:kv": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "prioritycache:meta": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },

On 20.05.20 14:05, Mark Nelson wrote:

Hi Harald,


Any idea what the priority_cache_manger perf counters show? (or you 
can also enable debug osd / debug priority_cache_manager) The osd 
memory autotuning works by shrinking the bluestore and rocksdb 
caches to some target value to try and keep the mapped memo

[ceph-users] move bluestore wal/db

[Frank R]

Is there a safe way to move the bluestore wal and db to a new device
that doesn't involve rebuilding the entire OSD?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: move bluestore wal/db

[Eneko Lacunza]

Hi,

Yes, it can be done (shuting down the OSD but no rebuild required), we 
did it for resizing wal partition to a bigger one.


A simple Google search will help; I can paste the procedure we followed 
but it's in spanish :(


Cheers

El 26/5/20 a las 17:20, Frank R escribió:

Is there a safe way to move the bluestore wal and db to a new device
that doesn't involve rebuilding the entire OSD?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943569206
Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
www.binovo.es
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000

[Marc Roos]

Look what I have found!!! :)
https://ceph.com/geen-categorie/ceph-loves-jumbo-frames/ 



-Original Message-
From: Anthony D'Atri [mailto:anthony.da...@gmail.com] 
Sent: maandag 25 mei 2020 22:12
To: Marc Roos
Cc: kdhall; martin.verges; sstkadu; amudhan83; ceph-users; doustar
Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not 
working after setting MTU 9000

Quick and easy depends on your network infrastructure.  Sometimes it is 
difficult or impossible to retrofit a live cluster without disruption.   


> On May 25, 2020, at 1:03 AM, Marc Roos  
wrote:
> 
> 
> I am interested. I am always setting mtu to 9000. To be honest I 
> cannot imagine there is no optimization since you have less interrupt 
> requests, and you are able x times as much data. Every time there 
> something written about optimizing the first thing mention is changing 

> to the mtu 9000. Because it is quick and easy win.
> 
> 
> 
> 
> -Original Message-
> From: Dave Hall [mailto:kdh...@binghamton.edu]
> Sent: maandag 25 mei 2020 5:11
> To: Martin Verges; Suresh Rama
> Cc: Amudhan P; Khodayar Doustar; ceph-users
> Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not 
> working after setting MTU 9000
> 
> All,
> 
> Regarding Martin's observations about Jumbo Frames
> 
> I have recently been gathering some notes from various internet 
> sources regarding Linux network performance, and Linux performance in 
> general, to be applied to a Ceph cluster I manage but also to the rest 

> of the Linux server farm I'm responsible for.
> 
> In short, enabling Jumbo Frames without also tuning a number of other 
> kernel and NIC attributes will not provide the performance increases 
> we'd like to see.  I have not yet had a chance to go through the rest 
> of the testing I'd like to do, but  I can confirm (via iperf3) that 
> only enabling Jumbo Frames didn't make a significant difference.
> 
> Some of the other attributes I'm referring to are incoming and 
> outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt 
> coalescing, NIC offload functions that should or shouldn't be turned 
> on, packet queuing disciplines (tc), the best choice of TCP slow-start 

> algorithms, and other TCP features and attributes.
> 
> The most off-beat item I saw was something about adding IPTABLES rules 

> to bypass CONNTRACK table lookups.
> 
> In order to do anything meaningful to assess the effect of all of 
> these settings I'd like to figure out how to set them all via Ansible 
> - so more to learn before I can give opinions.
> 
> -->  If anybody has added this type of configuration to Ceph Ansible,
> I'd be glad for some pointers.
> 
> I have started to compile a document containing my notes.  It's rough, 

> but I'd be glad to share if anybody is interested.
> 
> -Dave
> 
> Dave Hall
> Binghamton University
> 
>> On 5/24/2020 12:29 PM, Martin Verges wrote:
>> 
>> Just save yourself the trouble. You won't have any real benefit from
> MTU
>> 9000. It has some smallish, but it is not worth the effort, problems,
> and
>> loss of reliability for most environments.
>> Try it yourself and do some benchmarks, especially with your regular 
>> workload on the cluster (not the maximum peak performance), then drop
> the
>> MTU to default ;).
>> 
>> Please if anyone has other real world benchmarks showing huge
> differences
>> in regular Ceph clusters, please feel free to post it here.
>> 
>> --
>> Martin Verges
>> Managing director
>> 
>> Mobile: +49 174 9335695
>> E-Mail: martin.ver...@croit.io
>> Chat: https://t.me/MartinVerges
>> 
>> croit GmbH, Freseniusstr. 31h, 81247 Munich
>> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht 
>> Munich HRB 231263
>> 
>> Web: https://croit.io
>> YouTube: https://goo.gl/PGE1Bx
>> 
>> 
>>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama
>> :
>> 
>>> Ping with 9000 MTU won't get response as I said and it should be
> 8972. Glad
>>> it is working but you should know what happened to avoid this issue
> later.
>>> 
 On Sun, May 24, 2020, 3:04 AM Amudhan P  
wrote:
>>> 
 No, ping with MTU size 9000 didn't work.
 
 On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar
> 
 wrote:
 
> Does your ping work or not?
> 
> 
> On Sun, May 24, 2020 at 6:53 AM Amudhan P 
> wrote:
> 
>> Yes, I have set setting on the switch side also.
>> 
>> On Sat 23 May, 2020, 6:47 PM Khodayar Doustar,
> 
>> wrote:
>> 
>>> Problem should be with network. When you change MTU it should be
 changed
>>> all over the network, any single hup on your network should 
>>> speak
> and
>>> accept 9000 MTU packets. you can check it on your hosts with
>>> "ifconfig"
>>> command and there is also equivalent commands for other
 network/security
>>> devices.
>>> 
>>> If you have just one node which it not correctly configured for
> MTU
 9000
>>> it wouldn't work.
>>> 
>>> On Sat, May 23, 

[ceph-users] Re: Prometheus Python Errors

[Ernesto Puerta]

This has been recently fixed in master
 (I just submitted backporting PRs
for octopus  and nautilus
). BTW the fix is pretty trivial

.


Ernesto Puerta

He / Him / His

Senior Software Engineer, Ceph

Red Hat 



On Tue, May 19, 2020 at 9:38 PM  wrote:

> Hello Everyone,
>
> I have installed both Prometheus and Grafana on one of my manager nodes
> (Ubuntu 18.04), and have configured both according to the documentation. I
> have visible Grafana dashboards when visiting http://mon1:3000, but no
> data exists on the dashboard. Python errors are shown for the job_name:
> ceph in Prometheus.
>
> Below is my prometheus.yaml configuration
>
> global:
>   scrape_interval: 5s
>
> scrape_configs:
> - job_name: prometheus
>   static_configs:
>   - targets: ['localhost:9090']
> - job_name: 'ceph-exporter'
>   static_configs:
> - targets: ['localhost:9100']
>   labels:
> alias: ceph-exporter
> - job_name: 'ceph'
>   static_configs:
> - targets: ['localhost:9283']
>   labels:
> alias: ceph
>
> And, these are the Python errors shown when I view the details of the
> targets in Prometheus (http://mon1:9090/targets)
>
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line
> 670, in respond
> response.body = self.handler()
>   File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line
> 220, in __call__
> self.body = self.oldhandler(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line
> 60, in __call__
> return self.callable(*self.args, **self.kwargs)
>   File "/usr/share/ceph/mgr/prometheus/module.py", line 1060, in metrics
> return self._metrics(instance)
>   File "/usr/share/ceph/mgr/prometheus/module.py", line 1074, in _metrics
> instance.collect_cache = instance.collect()
>   File "/usr/share/ceph/mgr/prometheus/module.py", line 975, in collect
> self.get_rbd_stats()
>   File "/usr/share/ceph/mgr/prometheus/module.py", line 734, in
> get_rbd_stats
> 'rbd_stats_pools_refresh_interval', 300)
> TypeError: unsupported operand type(s) for +: 'int' and 'str'
>
> If anyone has experienced this issue, and might have a solution, I would
> appreciate any assistance.
>
> Thank you,
> Todd
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [External Email] Re: Ceph Nautius not working after setting MTU 9000

[Paul Emmerich]

Don't optimize stuff without benchmarking *before and after*, don't apply
random tuning tipps from the Internet without benchmarking them.

My experience with Jumbo frames: 3% performance. On a NVMe-only setup with
100 Gbit/s network.

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Tue, May 26, 2020 at 7:02 PM Marc Roos  wrote:

>
>
> Look what I have found!!! :)
> https://ceph.com/geen-categorie/ceph-loves-jumbo-frames/
>
>
>
> -Original Message-
> From: Anthony D'Atri [mailto:anthony.da...@gmail.com]
> Sent: maandag 25 mei 2020 22:12
> To: Marc Roos
> Cc: kdhall; martin.verges; sstkadu; amudhan83; ceph-users; doustar
> Subject: Re: [ceph-users] Re: [External Email] Re: Ceph Nautius not
> working after setting MTU 9000
>
> Quick and easy depends on your network infrastructure.  Sometimes it is
> difficult or impossible to retrofit a live cluster without disruption.
>
>
> > On May 25, 2020, at 1:03 AM, Marc Roos 
> wrote:
> >
> > 
> > I am interested. I am always setting mtu to 9000. To be honest I
> > cannot imagine there is no optimization since you have less interrupt
> > requests, and you are able x times as much data. Every time there
> > something written about optimizing the first thing mention is changing
>
> > to the mtu 9000. Because it is quick and easy win.
> >
> >
> >
> >
> > -Original Message-
> > From: Dave Hall [mailto:kdh...@binghamton.edu]
> > Sent: maandag 25 mei 2020 5:11
> > To: Martin Verges; Suresh Rama
> > Cc: Amudhan P; Khodayar Doustar; ceph-users
> > Subject: [ceph-users] Re: [External Email] Re: Ceph Nautius not
> > working after setting MTU 9000
> >
> > All,
> >
> > Regarding Martin's observations about Jumbo Frames
> >
> > I have recently been gathering some notes from various internet
> > sources regarding Linux network performance, and Linux performance in
> > general, to be applied to a Ceph cluster I manage but also to the rest
>
> > of the Linux server farm I'm responsible for.
> >
> > In short, enabling Jumbo Frames without also tuning a number of other
> > kernel and NIC attributes will not provide the performance increases
> > we'd like to see.  I have not yet had a chance to go through the rest
> > of the testing I'd like to do, but  I can confirm (via iperf3) that
> > only enabling Jumbo Frames didn't make a significant difference.
> >
> > Some of the other attributes I'm referring to are incoming and
> > outgoing buffer sizes at the NIC, IP, and TCP levels, interrupt
> > coalescing, NIC offload functions that should or shouldn't be turned
> > on, packet queuing disciplines (tc), the best choice of TCP slow-start
>
> > algorithms, and other TCP features and attributes.
> >
> > The most off-beat item I saw was something about adding IPTABLES rules
>
> > to bypass CONNTRACK table lookups.
> >
> > In order to do anything meaningful to assess the effect of all of
> > these settings I'd like to figure out how to set them all via Ansible
> > - so more to learn before I can give opinions.
> >
> > -->  If anybody has added this type of configuration to Ceph Ansible,
> > I'd be glad for some pointers.
> >
> > I have started to compile a document containing my notes.  It's rough,
>
> > but I'd be glad to share if anybody is interested.
> >
> > -Dave
> >
> > Dave Hall
> > Binghamton University
> >
> >> On 5/24/2020 12:29 PM, Martin Verges wrote:
> >>
> >> Just save yourself the trouble. You won't have any real benefit from
> > MTU
> >> 9000. It has some smallish, but it is not worth the effort, problems,
> > and
> >> loss of reliability for most environments.
> >> Try it yourself and do some benchmarks, especially with your regular
> >> workload on the cluster (not the maximum peak performance), then drop
> > the
> >> MTU to default ;).
> >>
> >> Please if anyone has other real world benchmarks showing huge
> > differences
> >> in regular Ceph clusters, please feel free to post it here.
> >>
> >> --
> >> Martin Verges
> >> Managing director
> >>
> >> Mobile: +49 174 9335695
> >> E-Mail: martin.ver...@croit.io
> >> Chat: https://t.me/MartinVerges
> >>
> >> croit GmbH, Freseniusstr. 31h, 81247 Munich
> >> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht
> >> Munich HRB 231263
> >>
> >> Web: https://croit.io
> >> YouTube: https://goo.gl/PGE1Bx
> >>
> >>
> >>> Am So., 24. Mai 2020 um 15:54 Uhr schrieb Suresh Rama
> >> :
> >>
> >>> Ping with 9000 MTU won't get response as I said and it should be
> > 8972. Glad
> >>> it is working but you should know what happened to avoid this issue
> > later.
> >>>
>  On Sun, May 24, 2020, 3:04 AM Amudhan P 
> wrote:
> >>>
>  No, ping with MTU size 9000 didn't work.
> 
>  On Sun, May 24, 2020 at 12:26 PM Khodayar Doustar
> > 
>  wrote:
> 
> > Does your ping work or not?
> >
> >
> > On Sun, May 24, 2020 at 6:53 AM Amudhan P 
> > wrote:
> >

[ceph-users] Re: Multisite RADOS Gateway replication factor in zonegroup

[Konstantin Shalygin]

On 5/25/20 9:50 PM, alexander.vysoc...@megafon.ru wrote:


I didn’t find any information about the replication factor in the zone 
group. Assume  I  have three ceph clusters with Rados Gateway in one 
zone group each with replica size 3. How many replicas of an object  
I’ll get in total?


Is it possible to define several regions, each with several 
datacenters, and define maximum replication factor at region scope?


In total you get as much replicas as clusters with this object you have, 
i.e. if your 3 clusters have object with size 3 you will get 9 replicas 
of this object in total.




k

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: looking for telegram group in English or Chinese

[Konstantin Shalygin]

On 5/26/20 1:13 PM, Zhenshi Zhou wrote:

Is there any telegram group for communicating with ceph users?


AFAIK there is only Russian (CIS) group [1], but feel free to join with 
English!





[1] https://t.me/ceph_ru

k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: looking for telegram group in English or Chinese

[Martin Verges]

Hello,

as I find it a good idea and couldn't find another, I just created
https://t.me/ceph_users.
Please feel free to join and let's see to get this channel startet ;)

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx


Am Mi., 27. Mai 2020 um 07:07 Uhr schrieb Konstantin Shalygin <
k0...@k0ste.ru>:

> On 5/26/20 1:13 PM, Zhenshi Zhou wrote:
> > Is there any telegram group for communicating with ceph users?
>
> AFAIK there is only Russian (CIS) group [1], but feel free to join with
> English!
>
>
>
>
> [1] https://t.me/ceph_ru
>
> k
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: looking for telegram group in English or Chinese

[Zhenshi Zhou]

Awesome, thanks!

Martin Verges  于2020年5月27日周三 下午2:04写道：

> Hello,
>
> as I find it a good idea and couldn't find another, I just created
> https://t.me/ceph_users.
> Please feel free to join and let's see to get this channel startet ;)
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695
> E-Mail: martin.ver...@croit.io
> Chat: https://t.me/MartinVerges
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
>
> Web: https://croit.io
> YouTube: https://goo.gl/PGE1Bx
>
>
> Am Mi., 27. Mai 2020 um 07:07 Uhr schrieb Konstantin Shalygin <
> k0...@k0ste.ru>:
>
>> On 5/26/20 1:13 PM, Zhenshi Zhou wrote:
>> > Is there any telegram group for communicating with ceph users?
>>
>> AFAIK there is only Russian (CIS) group [1], but feel free to join with
>> English!
>>
>>
>>
>>
>> [1] https://t.me/ceph_ru
>>
>> k
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io