Re: [ceph-users] radosgw: S3 object retention: high usage of default.rgw.log pool
I was misled.In fact, this is not an automatic deletion, but the removal of one object per op by application. Reject. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Converting to dynamic bucket resharding in Luminous
Hi, I have a Jewel Ceph cluster with RGW index sharding enabled. I've configured the index to have 128 shards. I am upgrading to Luminous. What will happen if I enable dynamic bucket index resharding in ceph.conf? Will it maintain my 128 shards (the buckets are currently empty), and will it split them (to 256, and beyond) when they get full enough? Yes it will. However I would not recommend enabling dynamic resharding, I had some problems with it, like resharding loops where large buckets failed to reshard, and it tried resharding them over and over again. I had problems deleting some buckets that had multiple reshards done because of missing objects (Maybe objects where deleted during a dynamic reshard, and this was not recorded to the indexes). So for the time being I disabled dynamic resharding again. Micha Krause ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
Hey Nathan.
No blaming here. I'm very thankful for this great peace (ok, sometime more of a
beast ;) ) of open-source SDS and all the great work around it incl. community
and users... and happy the problem is identified and can be fixed for
others/the future as well :)
Well, yes, can confirm your found "error" also here:
[root@sds20 ~]# ceph-detect-init
Traceback (most recent call last):
File "/usr/bin/ceph-detect-init", line 9, in
load_entry_point('ceph-detect-init==1.0.1', 'console_scripts',
'ceph-detect-init')()
File "/usr/lib/python2.7/site-packages/ceph_detect_init/main.py", line 56, in
run
print(ceph_detect_init.get(args.use_rhceph).init)
File "/usr/lib/python2.7/site-packages/ceph_detect_init/__init__.py", line
42, in get
release=release)
ceph_detect_init.exc.UnsupportedPlatform: Platform is not supported.: rhel 7.5
Gesendet: Sonntag, 29. Juli 2018 um 20:33 Uhr
Von: "Nathan Cutler"
An: ceph.nov...@habmalnefrage.de, "Vasu Kulkarni"
Cc: ceph-users , "Ceph Development"
Betreff: Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
> Strange...
> - wouldn't swear, but pretty sure v13.2.0 was working ok before
> - so what do others say/see?
> - no one on v13.2.1 so far (hard to believe) OR
> - just don't have this "systemctl ceph-osd.target" problem and all just works?
>
> If you also __MIGRATED__ from Luminous (say ~ v12.2.5 or older) to Mimic (say
> v13.2.0 -> v13.2.1) and __DO NOT__ see the same systemctl problems, whats
> your Linix OS and version (I'm on RHEL 7.5 here) ? :O
Best regards
Anton
Hi ceph.novice:
I'm the one to blame for this regretful incident. Today I have
reproduced the issue in teuthology:
2018-07-29T18:20:07.288 INFO:teuthology.orchestra.run.ovh093:Running:
'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph-detect-init'
2018-07-29T18:20:07.796
INFO:teuthology.orchestra.run.ovh093.stderr:Traceback (most recent call
last):
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
File "/bin/ceph-detect-init", line 9, in
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
load_entry_point('ceph-detect-init==1.0.1', 'console_scripts',
'ceph-detect-init')()
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
File "/usr/lib/python2.7/site-packages/ceph_detect_init/main.py", line
56, in run
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
print(ceph_detect_init.get(args.use_rhceph).init)
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
File "/usr/lib/python2.7/site-packages/ceph_detect_init/__init__.py",
line 42, in get
2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
release=release)
2018-07-29T18:20:07.797
INFO:teuthology.orchestra.run.ovh093.stderr:ceph_detect_init.exc.UnsupportedPlatform:
Platform is not supported.: rhel 7.5
Just to be sure, can you confirm? (I.e. issue the command
"ceph-detect-init" on your RHEL 7.5 system. Instead of saying "systemd"
it gives an error like above?)
I'm working on a fix now at https://github.com/ceph/ceph/pull/23303
Nathan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Running 12.2.5 without problems, should I upgrade to 12.2.7 or wait for 12.2.8?
Hi, I'm Running 12.2.5 and I have no Problems at the moment. However my servers reporting daily that they want to upgrade to 12.2.7, is this save or should I wait for 12.2.8? Are there any predictions when the 12.2.8 release will be available? Micha Krause ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Converting to dynamic bucket resharding in Luminous
Hi, I also had the same issues and took to disabling this feature. Thanks On Mon, Jul 30, 2018 at 8:42 AM, Micha Krause wrote: > Hi, > > I have a Jewel Ceph cluster with RGW index sharding enabled. I've >> configured the index to have 128 shards. I am upgrading to Luminous. What >> will happen if I enable dynamic bucket index resharding in ceph.conf? Will >> it maintain my 128 shards (the buckets are currently empty), and will it >> split them (to 256, and beyond) when they get full enough? >> > Yes it will. > However I would not recommend enabling dynamic resharding, I had some > problems with it, like resharding loops where large buckets failed to > reshard, and it tried resharding > them over and over again. > I had problems deleting some buckets that had multiple reshards done > because of missing objects (Maybe objects where deleted during a dynamic > reshard, and this was not recorded to > the indexes). > > So for the time being I disabled dynamic resharding again. > > Micha Krause > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
Hi together,
for all others on this list, it might also be helpful to know which setups are
likely affected.
Does this only occur for Filestore disks, i.e. if ceph-volume has taken over
taking care of these?
Does it happen on every RHEL 7.5 system?
We're still on 13.2.0 here and ceph-detect-init works fine on our CentOS 7.5
systems (it just echoes "systemd").
We're on Bluestore.
Should we hold off on an upgrade, or are we unaffected?
Cheers,
Oliver
Am 30.07.2018 um 09:50 schrieb ceph.nov...@habmalnefrage.de:
> Hey Nathan.
>
> No blaming here. I'm very thankful for this great peace (ok, sometime more of
> a beast ;) ) of open-source SDS and all the great work around it incl.
> community and users... and happy the problem is identified and can be fixed
> for others/the future as well :)
>
> Well, yes, can confirm your found "error" also here:
>
> [root@sds20 ~]# ceph-detect-init
> Traceback (most recent call last):
> File "/usr/bin/ceph-detect-init", line 9, in
> load_entry_point('ceph-detect-init==1.0.1', 'console_scripts',
> 'ceph-detect-init')()
> File "/usr/lib/python2.7/site-packages/ceph_detect_init/main.py", line 56,
> in run
> print(ceph_detect_init.get(args.use_rhceph).init)
> File "/usr/lib/python2.7/site-packages/ceph_detect_init/__init__.py", line
> 42, in get
> release=release)
> ceph_detect_init.exc.UnsupportedPlatform: Platform is not supported.: rhel
> 7.5
>
>
> Gesendet: Sonntag, 29. Juli 2018 um 20:33 Uhr
> Von: "Nathan Cutler"
> An: ceph.nov...@habmalnefrage.de, "Vasu Kulkarni"
> Cc: ceph-users , "Ceph Development"
>
> Betreff: Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
>> Strange...
>> - wouldn't swear, but pretty sure v13.2.0 was working ok before
>> - so what do others say/see?
>> - no one on v13.2.1 so far (hard to believe) OR
>> - just don't have this "systemctl ceph-osd.target" problem and all just
>> works?
>>
>> If you also __MIGRATED__ from Luminous (say ~ v12.2.5 or older) to Mimic
>> (say v13.2.0 -> v13.2.1) and __DO NOT__ see the same systemctl problems,
>> whats your Linix OS and version (I'm on RHEL 7.5 here) ? :O
>
> Best regards
> Anton
>
>
>
> Hi ceph.novice:
>
> I'm the one to blame for this regretful incident. Today I have
> reproduced the issue in teuthology:
>
> 2018-07-29T18:20:07.288 INFO:teuthology.orchestra.run.ovh093:Running:
> 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph-detect-init'
> 2018-07-29T18:20:07.796
> INFO:teuthology.orchestra.run.ovh093.stderr:Traceback (most recent call
> last):
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> File "/bin/ceph-detect-init", line 9, in
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> load_entry_point('ceph-detect-init==1.0.1', 'console_scripts',
> 'ceph-detect-init')()
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> File "/usr/lib/python2.7/site-packages/ceph_detect_init/main.py", line
> 56, in run
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> print(ceph_detect_init.get(args.use_rhceph).init)
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> File "/usr/lib/python2.7/site-packages/ceph_detect_init/__init__.py",
> line 42, in get
> 2018-07-29T18:20:07.797 INFO:teuthology.orchestra.run.ovh093.stderr:
> release=release)
> 2018-07-29T18:20:07.797
> INFO:teuthology.orchestra.run.ovh093.stderr:ceph_detect_init.exc.UnsupportedPlatform:
> Platform is not supported.: rhel 7.5
>
> Just to be sure, can you confirm? (I.e. issue the command
> "ceph-detect-init" on your RHEL 7.5 system. Instead of saying "systemd"
> it gives an error like above?)
>
> I'm working on a fix now at https://github.com/ceph/ceph/pull/23303
>
> Nathan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
for all others on this list, it might also be helpful to know which setups are likely affected. Does this only occur for Filestore disks, i.e. if ceph-volume has taken over taking care of these? Does it happen on every RHEL 7.5 system? It affects all OSDs managed by ceph-disk on all RHEL systems (but not on CentOS), regardless of whether they are filestore or bluestore. We're still on 13.2.0 here and ceph-detect-init works fine on our CentOS 7.5 systems (it just echoes "systemd"). We're on Bluestore. Should we hold off on an upgrade, or are we unaffected? The regression does not affect CentOS - only RHEL. Nathan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
kzal t maar eens testen :) On 30/07/18 10:54, Nathan Cutler wrote: for all others on this list, it might also be helpful to know which setups are likely affected. Does this only occur for Filestore disks, i.e. if ceph-volume has taken over taking care of these? Does it happen on every RHEL 7.5 system? It affects all OSDs managed by ceph-disk on all RHEL systems (but not on CentOS), regardless of whether they are filestore or bluestore. We're still on 13.2.0 here and ceph-detect-init works fine on our CentOS 7.5 systems (it just echoes "systemd"). We're on Bluestore. Should we hold off on an upgrade, or are we unaffected? The regression does not affect CentOS - only RHEL. Nathan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph crushmap question
Hi. I'm newbie with ceph. I know that i can define custom types in crush map. And type id 0 used always for devices. But if i specify type 0 slot Does i need to specify devices with this name or use predefined name device? For example: type 0 device device 0 osd.0 class hdd or type o slot slot 0 slot1-1 class hdd My use case - i have udev rules on my servers that maps device with serial number to specific server slot. So when disk dies i know which slot needs to be replaced. (In my server led uid to identify disk does not work). And i want to use my disk names (slot1, slot2 and so) inside ceph crush map. -- Vasiliy Tolstov, e-mail: v.tols...@selfip.ru ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HELP! --> CLUSER DOWN (was "v13.2.1 Mimic released")
Hi All,
there might be a a problem on Scientific Linux 7.5 too:
after upgrading directly from 12.2.5 to 13.2.1
[root@cephr01 ~]# ceph-detect-init
Traceback (most recent call last):
File "/usr/bin/ceph-detect-init", line 9, in
load_entry_point('ceph-detect-init==1.0.1', 'console_scripts',
'ceph-detect-init')()
File "/usr/lib/python2.7/site-packages/ceph_detect_init/main.py", line
56, in run
print(ceph_detect_init.get(args.use_rhceph).init)
File "/usr/lib/python2.7/site-packages/ceph_detect_init/__init__.py",
line 42, in get
release=release)
ceph_detect_init.exc.UnsupportedPlatform: Platform is not supported.:
rhel 7.5
# cat /etc/redhat-release
Scientific Linux release 7.5 (Nitrogen)
# cat /etc/os-release
NAME="Scientific Linux"
VERSION="7.5 (Nitrogen)"
ID="rhel"
ID_LIKE="scientific centos fedora"
VERSION_ID="7.5"
PRETTY_NAME="Scientific Linux 7.5 (Nitrogen)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:scientificlinux:scientificlinux:7.5:GA"
HOME_URL="http://www.scientificlinux.org//";
BUG_REPORT_URL="mailto:scientific-linux-de...@listserv.fnal.gov";
REDHAT_BUGZILLA_PRODUCT="Scientific Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Scientific Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
Applying Nathan's one line fix to ceph_detect_init/__init__.py
works :)
# ceph-detect-init
systemd
all the best,
Jake
On 30/07/18 09:59, Kenneth Waegeman wrote:
> kzal t maar eens testen :)
>
>
> On 30/07/18 10:54, Nathan Cutler wrote:
>>> for all others on this list, it might also be helpful to know which
>>> setups are likely affected.
>>> Does this only occur for Filestore disks, i.e. if ceph-volume has
>>> taken over taking care of these?
>>> Does it happen on every RHEL 7.5 system?
>>
>> It affects all OSDs managed by ceph-disk on all RHEL systems (but not
>> on CentOS), regardless of whether they are filestore or bluestore.
>>
>>> We're still on 13.2.0 here and ceph-detect-init works fine on our
>>> CentOS 7.5 systems (it just echoes "systemd").
>>> We're on Bluestore.
>>> Should we hold off on an upgrade, or are we unaffected?
>>
>> The regression does not affect CentOS - only RHEL.
>>
>> Nathan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS configuration for millions of small files
Hello community, I am building first cluster for project, that hosts millions of small (from 20kb) and big (up to 10mb) files. Right now we are moving from local 16tb raid storage to cluster of 12 small machines. We are planning to have 11 OSD nodes, use erasure coding pool (10+1) and one host for MDS. On my local tests I see, that available space decrease unproportionally to the amount of data copied into cluster. With clean cluster I have, for example 100gb available space, but after copying 40gb in - size decreases for about 5-10%. Is that normal? Is there any term, that would specify cluster's minimal object size? I also have question if having so many small files (current number is about 50'000'000 files at least) - could have negative impact and where would be our bottleneck? As we don't have money for SSD, we will have WAL/DB on separate simple HDD. Also - would that help to put Metadata pool on separate disks, away from Data pool drives for CephFS? Regards, Anton. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] very low read performance
Dear list,
we experience very poor single thread read performance (~35 MB/s) on our 5
node ceph cluster. I first encountered it in vms transferring data via rsync,
but could reproduce the problem with rbd and rados bench on the physical
nodes.
Let me shortly give an overview on our infrastructure which is surely far from
optimal for ceph:
- 5 dell r720xd nodes running proxmox 5.2 are kvm hypervisors and also
provide the ceph infrastructure (3 are mons, all 5 are osd nodes)
- each node has 12 4 TB 7200 rpm SAS HDDs, attached via the internal H710P,
which maps each physical disk to a virtual one. Each virtual disk is one
bluestore osd (total 60). OS resides on two additional disks.
- each node has two Intel DC P3700 NVME 400 GB, each with 6 ~54 GB
partitions for rocks db
- each node has two 10 GBit/s NICs teamed together (ovs bond in slb-balance
mode, bridge to attach vms and host interfaces. The networks are vlan tagged).
Ceph's cluster and public network is the same. Ping latency between nodes is ~
0.1 ms and MTU 1500.
- we disabled cephx trying to gain performance
- deep scrubbing is restricted from 1900 to 0600 and the interval raised
from weekly to 28 days as it reduced performance even further
- we reduced several debugging options as this is often suggested to gain
performance
- replication factor is 3
- the ceph pool provides rbd images and has 2048 pgs (current distribution
is 80 to 129 pgs/osd)
Some more information at https://git.idiv.de/dsarpe/ceph-perf (etc/pve/
ceph.conf, etc/network/interfaces, ceph_osd_df, ceph_osd_tree).
Here are two example rados bench runs for write and sequential read while the
cluster was relatively idle (low cpu and memory load on nodes, ceph capacity
used < 50%, no recovery and hardly any other client io):
```
# rados bench -p rbd --run-name benchmark_t1 --no-cleanup -b 4M 300 seq -t 1
[…]
Total time run: 300.018203
Total writes made: 13468
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 179.562
Stddev Bandwidth: 18.155
Max bandwidth (MB/sec): 220
Min bandwidth (MB/sec): 108
Average IOPS: 44
Stddev IOPS:4
Max IOPS: 55
Min IOPS: 27
Average Latency(s): 0.0222748
Stddev Latency(s): 0.0134969
Max latency(s): 0.27939
Min latency(s): 0.0114312
```
```
# rados bench -p rbd --run-name benchmark_t1 300 seq -t 1
[…]
Total time run: 300.239245
Total reads made: 2612
Read size:4194304
Object size: 4194304
Bandwidth (MB/sec): 34.7989
Average IOPS: 8
Stddev IOPS: 1
Max IOPS: 15
Min IOPS: 5
Average Latency(s): 0.114472
Max latency(s): 0.471255
Min latency(s): 0.0182361
```
Performance scales with the number of threads, but I think even with spinners
the sequential read from a single thread should be higher. If I test
sequential read speed from individual ceph block partitions with dd I get
values of ~180 MB/s per partition (there is one ceph block partition per
device) even if reading from all partitions in parallel.
```
echo 3 | tee /proc/sys/vm/drop_caches
ls -l /var/lib/ceph/osd/ceph-*/ |
grep "block -" |
awk '{ print $11 }' |
while read -r partition
do dd if="$partition" of=/dev/null bs=4M count=100 &
done
```
The block.db partitions on nvme all report >=1.1 GB/s if read sequentially in
parallel it goes down to 350 MB/s (there are 6 ceph block.db partitions per
nvme device).
```
echo 3 | tee /proc/sys/vm/drop_caches
ls -l /var/lib/ceph/osd/ceph-*/ |
grep "block.db -" |
awk '{ print $11 }' |
while read -r partition
do dd if="$partition" of=/dev/null bs=4M count=100 &
done
```
During deeb scrubing iostat usually shows values of 50 - 100 MB/s per device
(not all at the same time of course). Therefore I wonder why the sequential
read is so much lower? Any pointers where to look?
On a side note is there a command to list all current ceph clients and get
their configuration on luminous?
Cheers,
Dirk
--
general it-support unit
Phone +49 341 97-33118
Email dirk.sa...@idiv.de
German Centre for Integrative Biodiversity Research (iDiv)
Halle-Jena-Leipzig
Deutscher Platz 5e 04103
Leipzig
Germany
iDiv is a research centre of the DFG - Deutsche Forschungsgemeinschaft
iDiv ist ein Forschungszentrum der Deutschen
Forschungsgemeinschaft (DFG). Es ist eine zentrale Einrichtung
der Universität Leipzig im Sinne des § 92 Abs. 1 SächsHSFG und wird
zusammen mit der Martin-Luther-Universität Halle-Wittenberg, der
Friedrich-Schiller-Universität Jena sowie dem Helmholtz-Zentrum für
Umweltforschung (UFZ) betrieben. Sieben außeruniversitäre Einrichtungen
unterstützen iDiv finanziell sowie durch ihre Expertise: das
Max-Planck-Institut für Biogeochemie (MPI BGC), das Max-Planck-Institut
für chemische Ökologie (MPI CE), das Max-Planck-Institut für
evolutionäre Anthropologie (MPI EVA),
Re: [ceph-users] Cephfs meta data pool to ssd and measuring performance difference
Something like smallfile perhaps? https://github.com/bengland2/smallfile Or you just time creating/reading lots of files With read benching you would want to ensure you've cleared your mds cache or use a dataset larger than the cache. I'd be interested in seeing your results, I this on the to do list myself. On 25 Jul 2018 15:18, "Marc Roos" wrote: >From this thread, I got how to move the meta data pool from the hdd's to the ssd's. https://www.spinics.net/lists/ceph-users/msg39498.html ceph osd pool get fs_meta crush_rule ceph osd pool set fs_meta crush_rule replicated_ruleset_ssd I guess this can be done on a live system? What would be a good test to show the performance difference between the old hdd and the new ssd? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] [Jewel 10.2.11] OSD Segmentation fault
Hello Ceph users, We have updated our cluster from 10.2.7 to 10.2.11. A few hours after the update, 1 OSD crashed. When trying to add the OSD back to the cluster, other 2 OSDs started crashing with segmentation fault. Had to mark all 3 OSDs as down as we had stuck PGs and blocked operations and the cluster status was HEALTH_ERR. We have tried various ways to re-add the OSDs back to the cluster but after a while they start crashing and won't start anymore. After a while they can be started again and marked as in but after some rebalancing they will start the crashing imediately after starting. Here are some logs: https://pastebin.com/nCRamgRU Do you know of any existing bug report that might be related? (I couldn't find anything). I will happily provide any information that would help solving this issue. Thank you, Alex Cucu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph-mgr dashboard behind reverse proxy
Hi! I want to set up the dashboard behind a reverse proxy. How do people determine which ceph-mgr is active? Is there any simple and elegant solution? Cheers, Tobias Florek signature.asc Description: signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs tell command not working
On Fri, Jul 27, 2018 at 8:35 PM Scottix wrote: > > ceph tell mds.0 client ls > 2018-07-27 12:32:40.344654 7fa5e27fc700 0 client.89408629 ms_handle_reset on > 10.10.1.63:6800/1750774943 > Error EPERM: problem getting command descriptions from mds.0 You need "mds allow *" capabilities (the default client.admin user has this) to send commands to MDS daemons. John > > mds log > 2018-07-27 12:32:40.342753 7fc9c1239700 1 mds.CephMon203 handle_command: > received command from client without `tell` capability: 10.10.1.x:0/953253037 > > > We are trying to run this and getting an error. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs tell command not working
Awww that makes more sense now. I guess I didn't quite comprehend EPERM at the time. Thank You, Scott On Mon, Jul 30, 2018 at 7:19 AM John Spray wrote: > On Fri, Jul 27, 2018 at 8:35 PM Scottix wrote: > > > > ceph tell mds.0 client ls > > 2018-07-27 12:32:40.344654 7fa5e27fc700 0 client.89408629 > ms_handle_reset on 10.10.1.63:6800/1750774943 > > Error EPERM: problem getting command descriptions from mds.0 > > You need "mds allow *" capabilities (the default client.admin user has > this) to send commands to MDS daemons. > > John > > > > > > > mds log > > 2018-07-27 12:32:40.342753 7fc9c1239700 1 mds.CephMon203 > handle_command: received command from client without `tell` capability: > 10.10.1.x:0/953253037 > > > > > > We are trying to run this and getting an error. > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS configuration for millions of small files
10+1 is a bad idea for obvious reasons (not enough coding chunks, you will be offline if even one server is offline). The real problem is that your 20kb files will be split up into 2kb chunks and the metadata overhead and bluestore min alloc size will eat up your disk space. Paul 2018-07-30 13:44 GMT+02:00 Anton Aleksandrov : > Hello community, > > I am building first cluster for project, that hosts millions of small > (from 20kb) and big (up to 10mb) files. Right now we are moving from local > 16tb raid storage to cluster of 12 small machines. We are planning to have > 11 OSD nodes, use erasure coding pool (10+1) and one host for MDS. > > On my local tests I see, that available space decrease unproportionally to > the amount of data copied into cluster. With clean cluster I have, for > example 100gb available space, but after copying 40gb in - size decreases > for about 5-10%. Is that normal? > > Is there any term, that would specify cluster's minimal object size? > > I also have question if having so many small files (current number is > about 50'000'000 files at least) - could have negative impact and where > would be our bottleneck? As we don't have money for SSD, we will have > WAL/DB on separate simple HDD. > > Also - would that help to put Metadata pool on separate disks, away from > Data pool drives for CephFS? > > Regards, > Anton. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs tell command not working
On Mon, Jul 30, 2018 at 3:55 PM Scottix wrote: > > Awww that makes more sense now. I guess I didn't quite comprehend EPERM at > the time. How could anyone misunderstand our super-clear and not-at-all-obscure error messages? ;-) Easy fix: https://github.com/ceph/ceph/pull/23330 John > > Thank You, > Scott > On Mon, Jul 30, 2018 at 7:19 AM John Spray wrote: >> >> On Fri, Jul 27, 2018 at 8:35 PM Scottix wrote: >> > >> > ceph tell mds.0 client ls >> > 2018-07-27 12:32:40.344654 7fa5e27fc700 0 client.89408629 ms_handle_reset >> > on 10.10.1.63:6800/1750774943 >> > Error EPERM: problem getting command descriptions from mds.0 >> >> You need "mds allow *" capabilities (the default client.admin user has >> this) to send commands to MDS daemons. >> >> John >> >> >> >> > >> > mds log >> > 2018-07-27 12:32:40.342753 7fc9c1239700 1 mds.CephMon203 handle_command: >> > received command from client without `tell` capability: >> > 10.10.1.x:0/953253037 >> > >> > >> > We are trying to run this and getting an error. >> > ___ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade Ceph 13.2.0 -> 13.2.1 and Windows iSCSI clients breakup
On 07/28/2018 03:59 PM, Wladimir Mutel wrote:
> Dear all,
>
> I want to share some experience of upgrading my experimental 1-host
> Ceph cluster from v13.2.0 to v13.2.1.
> First, I fetched new packages and installed them using 'apt
> dist-upgrade', which went smooth as usual.
> Then I noticed from 'lsof', that Ceph daemons were not restarted after
> the upgrade ('ceph osd versions' still showed 13.2.0).
> Using instructions on Luminous->Mimic upgrade, I decided to restart
> ceph-{mon,mgr,osd}.targets.
> And surely, on restarting ceph-osd.target, iSCSI sessions had been
> broken on tcmu-runner side ('Timing out cmd', 'Handler connection
> lost'), and Windows (2008 R2) clients lost their iSCSI devices.
> But that was only a beginning of surprises that followed.
> Looking into Windows Disk Management, I noticed that iSCSI disks were
> re-detected with size about 0.12 Gb larger, i.e. 2794.52 GB instead of
> 2794.40 GB, and of course the system lost their GPT labels from its
> sight. I quickly checked 'rbd info' on Ceph side and did not notice any
> increase in RBD images. They were still exactly 715398 4MB-objects as I
> intended initially.
> Restarting iSCSI initiator service on Windows did not help. Restarting
> the whole Windows did not help. Restarting tcmu-runner on Ceph side did
> not help. What resolved the problem, to my great surprise, was
> _removing/re-adding MPIO feature and re-adding iSCSI multipath support_.
> After that, Windows detected iSCSI disks with proper size again, and
> restored visibility of GPT partitions, dynamic disk metadata and all the
> rest.
>
> Ok, I avoided data loss at this time, but I have some remaining
> questions :
>
> 1. Can Ceph minor version upgrades be made less disruptive and
> traumatic? Like, some king of staged/rolling OSD daemons restart within
> single upgraded host, without losing librbd sessions ?
>
> 2. Is Windows (2008 R2) MPIO support really that screwed & crippled ?
> Were there any improvements in Win2012/2016 ? I have physical servers
We have not done any testing with windows 2008 R2, so we do not know any
issues in it and what windows patches/hotfixes are needed.
We have tested windows 2016 and have not seen that problem. We are
working on testing 2012 R2 now, but have not done that type of test yet.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph lvm question
On Sat, Jul 28, 2018 at 12:44 AM, Satish Patel wrote: > I have simple question i want to use LVM with bluestore (Its > recommended method), If i have only single SSD disk for osd in that > case i want to keep journal + data on same disk so how should i create > lvm to accommodate ? bluestore doesn't have a journal like filestore, but probably you mean block.db ? The naming conventions are different. In the case of bluestore, what you are describing doesn't require to have split LVs for each OSD component. You can simply do this: ceph-volume lvm create --bluestore --data /dev/sdb Behind the scenes, ceph-volume will create the vg/lv for /dev/sdb and get everything working > > Do i need to do following > > pvcreate /dev/sdb > vgcreate vg0 /dev/sdb > > Now i have vg(500GB) so i will create two lv-01 (10G Journal) and > lv-02 (490GB Data) > > I am doing correct? > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [Ceph-maintainers] download.ceph.com repository changes
On Fri, Jul 27, 2018 at 1:28 AM, Fabian Grünbichler wrote: > On Tue, Jul 24, 2018 at 10:38:43AM -0400, Alfredo Deza wrote: >> Hi all, >> >> After the 12.2.6 release went out, we've been thinking on better ways >> to remove a version from our repositories to prevent users from >> upgrading/installing a known bad release. >> >> The way our repos are structured today means every single version of >> the release is included in the repository. That is, for Luminous, >> every 12.x.x version of the binaries is in the same repo. This is true >> for both RPM and DEB repositories. >> >> However, the DEB repos don't allow pinning to a given version because >> our tooling (namely reprepro) doesn't construct the repositories in a >> way that this is allowed. For RPM repos this is fine, and version >> pinning works. > > If you mean that reprepo does not support referencing multiple versions > of packages in the Packages file, there is a patched fork that does > (that seems well-supported): > > https://github.com/profitbricks/reprepro Thanks for this link. That's great to know someone's working on this. What's the status of merging that back into the main reprepro code, or else shipping that fork as the new reprepro package in Debian / Ubuntu? The Ceph project could end up responsible for maintaining that reprepro fork if the main Ubuntu community does not pick it up :) The fork is several years old, and the latest update on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570623 was over a year ago. - Ken ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph lvm question
Thanks Alfredo,
This is what i am trying to do with ceph-ansible v3.1 and getting
following error, where i am wrong?
---
osd_objectstore: bluestore
osd_scenario: lvm
lvm_volumes:
- data: /dev/sdb
TASK [ceph-osd : include scenarios/lvm.yml]
*
Saturday 28 July 2018 17:17:18 -0400 (0:00:00.082) 0:11:08.249 *
fatal: [osd3]: FAILED! => {"failed": true, "reason": "no action
detected in task. This often indicates a misspelled module name, or
incorrect module path.\n\nThe error appears to have been in
'/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
line 3, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
here\nWe could be wrong, but this one looks like it might be an issue
with\nmissing quotes. Always quote template expression brackets when
they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
}}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
}}\"\n\n\nThe error appears to have been in
'/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
line 3, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
here\nWe could be wrong, but this one looks like it might be an issue
with\nmissing quotes. Always quote template expression brackets when
they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
}}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
}}\"\n\nexception type: \nexception: no action detected in
task. This often indicates a misspelled module name, or incorrect
module path.\n\nThe error appears to have been in
'/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
line 3, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
here\nWe could be wrong, but this one looks like it might be an issue
with\nmissing quotes. Always quote template expression brackets when
they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
}}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
}}\"\n"}
On Mon, Jul 30, 2018 at 1:11 PM, Alfredo Deza wrote:
> On Sat, Jul 28, 2018 at 12:44 AM, Satish Patel wrote:
>> I have simple question i want to use LVM with bluestore (Its
>> recommended method), If i have only single SSD disk for osd in that
>> case i want to keep journal + data on same disk so how should i create
>> lvm to accommodate ?
>
> bluestore doesn't have a journal like filestore, but probably you mean
> block.db ? The naming conventions are different.
>
> In the case of bluestore, what you are describing doesn't require to
> have split LVs for each OSD component. You can simply do this:
>
> ceph-volume lvm create --bluestore --data /dev/sdb
>
> Behind the scenes, ceph-volume will create the vg/lv for /dev/sdb and
> get everything working
>
>>
>> Do i need to do following
>>
>> pvcreate /dev/sdb
>> vgcreate vg0 /dev/sdb
>>
>> Now i have vg(500GB) so i will create two lv-01 (10G Journal) and
>> lv-02 (490GB Data)
>>
>> I am doing correct?
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph lvm question
Try using master?
Not sure really what 3.1 supports.
On Mon, Jul 30, 2018 at 2:03 PM, Satish Patel wrote:
> Thanks Alfredo,
>
> This is what i am trying to do with ceph-ansible v3.1 and getting
> following error, where i am wrong?
>
> ---
> osd_objectstore: bluestore
> osd_scenario: lvm
> lvm_volumes:
> - data: /dev/sdb
>
>
>
>
> TASK [ceph-osd : include scenarios/lvm.yml]
> *
> Saturday 28 July 2018 17:17:18 -0400 (0:00:00.082) 0:11:08.249
> *
> fatal: [osd3]: FAILED! => {"failed": true, "reason": "no action
> detected in task. This often indicates a misspelled module name, or
> incorrect module path.\n\nThe error appears to have been in
> '/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
> line 3, column 3, but may\nbe elsewhere in the file depending on the
> exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
> name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
> here\nWe could be wrong, but this one looks like it might be an issue
> with\nmissing quotes. Always quote template expression brackets when
> they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
> }}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
> }}\"\n\n\nThe error appears to have been in
> '/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
> line 3, column 3, but may\nbe elsewhere in the file depending on the
> exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
> name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
> here\nWe could be wrong, but this one looks like it might be an issue
> with\nmissing quotes. Always quote template expression brackets when
> they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
> }}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
> }}\"\n\nexception type: 'ansible.errors.AnsibleParserError'>\nexception: no action detected in
> task. This often indicates a misspelled module name, or incorrect
> module path.\n\nThe error appears to have been in
> '/etc/ansible/roles/ceph-ansible/roles/ceph-osd/tasks/scenarios/lvm.yml':
> line 3, column 3, but may\nbe elsewhere in the file depending on the
> exact syntax problem.\n\nThe offending line appears to be:\n\n\n-
> name: \"use ceph-volume to create {{ osd_objectstore }} osds\"\n ^
> here\nWe could be wrong, but this one looks like it might be an issue
> with\nmissing quotes. Always quote template expression brackets when
> they\nstart a value. For instance:\n\nwith_items:\n - {{ foo
> }}\n\nShould be written as:\n\nwith_items:\n - \"{{ foo
> }}\"\n"}
>
> On Mon, Jul 30, 2018 at 1:11 PM, Alfredo Deza wrote:
>> On Sat, Jul 28, 2018 at 12:44 AM, Satish Patel wrote:
>>> I have simple question i want to use LVM with bluestore (Its
>>> recommended method), If i have only single SSD disk for osd in that
>>> case i want to keep journal + data on same disk so how should i create
>>> lvm to accommodate ?
>>
>> bluestore doesn't have a journal like filestore, but probably you mean
>> block.db ? The naming conventions are different.
>>
>> In the case of bluestore, what you are describing doesn't require to
>> have split LVs for each OSD component. You can simply do this:
>>
>> ceph-volume lvm create --bluestore --data /dev/sdb
>>
>> Behind the scenes, ceph-volume will create the vg/lv for /dev/sdb and
>> get everything working
>>
>>>
>>> Do i need to do following
>>>
>>> pvcreate /dev/sdb
>>> vgcreate vg0 /dev/sdb
>>>
>>> Now i have vg(500GB) so i will create two lv-01 (10G Journal) and
>>> lv-02 (490GB Data)
>>>
>>> I am doing correct?
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS configuration for millions of small files
Does't 10+1 mean that one server can go offline without loosing data and functionality? We are quite short on hardware and need as much space as possible... would 9+1 sound better with one more extra node? Yes, that is what i see in my test in regard to space. Can min alloc size be changed? Anton Original message From: Paul Emmerich Date: 30/07/2018 17:55 (GMT+02:00) To: Anton Aleksandrov Cc: Ceph Users Subject: Re: [ceph-users] CephFS configuration for millions of small files 10+1 is a bad idea for obvious reasons (not enough coding chunks, you will be offline if even one server is offline). The real problem is that your 20kb files will be split up into 2kb chunks and the metadata overhead and bluestore min alloc size will eat up your disk space. Paul 2018-07-30 13:44 GMT+02:00 Anton Aleksandrov : Hello community, I am building first cluster for project, that hosts millions of small (from 20kb) and big (up to 10mb) files. Right now we are moving from local 16tb raid storage to cluster of 12 small machines. We are planning to have 11 OSD nodes, use erasure coding pool (10+1) and one host for MDS. On my local tests I see, that available space decrease unproportionally to the amount of data copied into cluster. With clean cluster I have, for example 100gb available space, but after copying 40gb in - size decreases for about 5-10%. Is that normal? Is there any term, that would specify cluster's minimal object size? I also have question if having so many small files (current number is about 50'000'000 files at least) - could have negative impact and where would be our bottleneck? As we don't have money for SSD, we will have WAL/DB on separate simple HDD. Also - would that help to put Metadata pool on separate disks, away from Data pool drives for CephFS? Regards, Anton. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Enable daemonperf - no stats selected by filters
Do you need to enable the option daemonperf?
[@c01 ~]# ceph daemonperf mds.a
Traceback (most recent call last):
File "/usr/bin/ceph", line 1122, in
retval = main()
File "/usr/bin/ceph", line 822, in main
done, ret = maybe_daemon_command(parsed_args, childargs)
File "/usr/bin/ceph", line 686, in maybe_daemon_command
return True, daemonperf(childargs, sockpath)
File "/usr/bin/ceph", line 776, in daemonperf
watcher.run(interval, count)
File "/usr/lib/python2.7/site-packages/ceph_daemon.py", line 362, in
run
self._load_schema()
File "/usr/lib/python2.7/site-packages/ceph_daemon.py", line 350, in
_load_schema
raise RuntimeError("no stats selected by filters")
RuntimeError: no stats selected by filters
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS configuration for millions of small files
Changing default 64k hdd min alloc size to 8k saved me 8 terabytes of disk space on cephfs with 150 million small files. You will need to redeploy OSDs for change to take effect. > On 30.07.2018, at 22:37, Anton Aleksandrov wrote: > > Yes, that is what i see in my test in regard to space. Can min alloc size be > changed? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Enable daemonperf - no stats selected by filters
On Mon, Jul 30, 2018 at 10:27 PM Marc Roos wrote:
>
>
> Do you need to enable the option daemonperf?
This looks strange, it's supposed to have sensible defaults -- what
version are you on?
John
> [@c01 ~]# ceph daemonperf mds.a
> Traceback (most recent call last):
> File "/usr/bin/ceph", line 1122, in
> retval = main()
> File "/usr/bin/ceph", line 822, in main
> done, ret = maybe_daemon_command(parsed_args, childargs)
> File "/usr/bin/ceph", line 686, in maybe_daemon_command
> return True, daemonperf(childargs, sockpath)
> File "/usr/bin/ceph", line 776, in daemonperf
> watcher.run(interval, count)
> File "/usr/lib/python2.7/site-packages/ceph_daemon.py", line 362, in
> run
> self._load_schema()
> File "/usr/lib/python2.7/site-packages/ceph_daemon.py", line 350, in
> _load_schema
> raise RuntimeError("no stats selected by filters")
> RuntimeError: no stats selected by filters
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Intermittent client reconnect delay following node fail
Hi.
We have recently setup our first ceph cluster (4 nodes) but our node failure
tests have revealed an intermittent problem. When we take down a node (i.e. by
powering it off) most of the time all clients reconnect to the cluster within
milliseconds, but occasionally it can take them 30 seconds or more. All clients
are Centos7 instances and have the ceph cluster mount point configured in
/etc/fstab as follows:
10.18.49.35:6789,10.18.49.204:6789,10.18.49.101:6789,10.18.49.183:6789:/
/mnt/ceph ceph name=admin,secretfile=/etc/ceph_key,noatime,_netdev0 2
On rare occasions, using the ls command, we can see that a failover has left a
client's /mnt/ceph directory with the following state: "??? ? ??
?? ceph". When this occurs, we think that the client has failed
to connect within 45 seconds (the mds_reconnect_timeout period) so the client
has been evicted. We can reproduce this circumstance by reducing the mds
reconnect timeout down to 1 second.
We'd like to know why our clients sometimes struggle to reconnect after a
cluster node failure and how to prevent this i.e. how can we ensure that all
clients consistently reconnect to the cluster quickly following a node failure.
We are using the default configuration options.
Ceph Status:
cluster:
id: ea2d9095-3deb-4482-bf6c-23229c594da4
health: HEALTH_OK
services:
mon: 4 daemons, quorum dub-ceph-01,dub-ceph-03,dub-ceph-04,dub-ceph-02
mgr: dub-ceph-02(active), standbys: dub-ceph-04.ott.local, dub-ceph-01,
dub-ceph-03
mds: cephfs-1/1/1 up {0=dub-ceph-03=up:active}, 3 up:standby
osd: 4 osds: 4 up, 4 in
data:
pools: 2 pools, 200 pgs
objects: 2.36 k objects, 8.9 GiB
usage: 31 GiB used, 1.9 TiB / 2.0 TiB avail
pgs: 200 active+clean
Thanks
William Lawton
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Mgr cephx caps to run `ceph fs status`?
Hi all, I want a non-admin client to be able to run `ceph fs status`, either via the ceph CLI or a python script. Adding `mgr "allow *"` to this client's cephx caps works, but I'd like to be more specific if possible. I can't find the complete list of mgr cephx caps anywhere, so if you could point me in the right direction, that'd be great! Cheers, Linh ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
