[ceph-users] Ubuntu18 and RBD Kernel Module

[Siegfried Höllrigl]

Hi !

We are running a ceph 12.2.7 Cluster and use it for RBDs.

We have now a few new servers installed with Ubuntu 18.
The default kernel version is v4.15.0.

When we create a new rbd and map/xfs-format/mount it, everything looks fine.
But if we want to map/mount a rbd that has already data in it, it takes a
very long time (>5minutes) - sometimes to map, sometimes to mount it.

There seems to be a process taking 100% of a cpu core during that "hang":
 3103 root  20   0   0  0  0 R 100.0  0.0   0:04.65 
kworker/11:1


With the "ukuu" tool, we have tested some other kernel versions :
v4.16.18 - same behavior
v4.18.5  - same behavior

And then an older kernel :
4.4.152-0404152-generic - rbd map/mount/umount/unmap - looks fine !

In the ceph.conf there is the line "rbd default features = 3" already 
(on all Servers).


Is there a need to further debug this, or did we miss some 
parameter/feature that

needs to be set differently on newer Kernels ?

Br,


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Installing ceph 12.2.4 via Ubuntu apt

[Thomas Bennett]

Hi,

I'm wanting to pin to an older version of Ceph Luminous (12.2.4) and I've
noticed that https://download.ceph.com/debian-luminous/ does not support
this via apt install:
apt install ceph works for 12.2.7 but
apt install ceph=12.2.4-1xenial does not work

The deb file are there, they're just not included in the package
distribution. Is this the desired behaviour or a misconfiguration?

Cheers,
Tom

--
Thomas Bennett

SARAO
Science Data Processing
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Deleting incomplete PGs from an erasure coded pool

[Maks Kowalik]

What is the correct procedure for re-creating an incomplete placement group
that belongs to an erasure coded pool?
I'm facing a situation when too many shards of 3 PGs were lost during OSD
crashes, and taking the data loss was decided, but can't force ceph to
recreate those PGs. The query output shows:
"peering_blocked_by_detail": [
{"detail": "peering_blocked_by_history_les_bound"}
What was tried:
1. manual deletion of all shards appearing in "peers" secion of PG query
output
2. marking all shards as complete using ceph-objectstore-tool
3. deleting peering history from OSDs keeping the shards
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Marc Roos]

I have a idle test cluster (centos7.5, Linux c04 
3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs. 

I tested reading a few files on this cephfs mount and get very low 
results compared to the rados bench. What could be the issue here?

[@client folder]# dd if=5GB.img of=/dev/null status=progress
954585600 bytes (955 MB) copied, 157.455633 s, 6.1 MB/s



I included this is rados bench that shows sort of that cluster 
performance is sort of as expected.
[@c01 ~]# rados bench -p fs_data 10 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 
4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_453883
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg 
lat(s)
0   0 0 0 0 0   -
   0
1  165842   167.967   1680.252071
0.323443
2  16   10690   179.967   1920.583383
0.324867
3  16   139   123   163.973   1320.170865
0.325976
4  16   183   167   166.975   1760.413676
0.361364
5  16   224   208   166.374   1640.394369
0.365956
6  16   254   238   158.642   1200.698396
0.382729
7  16   278   262   149.692960.120742
0.397625
8  16   317   301   150.478   1560.786822
0.411193
9  16   360   344   152.867   1720.601956
0.411577
   10  16   403   387   154.778   172 0.20342
0.404114
Total time run: 10.353683
Total writes made:  404
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 156.08
Stddev Bandwidth:   29.5778
Max bandwidth (MB/sec): 192
Min bandwidth (MB/sec): 96
Average IOPS:   39
Stddev IOPS:7
Max IOPS:   48
Min IOPS:   24
Average Latency(s): 0.409676
Stddev Latency(s):  0.243565
Max latency(s): 1.25028
Min latency(s): 0.0830112
Cleaning up (deleting benchmark objects)
Removed 404 objects
Clean up completed and total clean up time :0.867185




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Igor Fedotov]

Hi Marc,


In general dd isn't the best choice for benchmarking.

In you case there are at least 3 differences from rados bench :

1)If I haven't missed something then you're comparing reads vs. writes

2) Block Size is difference ( 512 bytes for dd vs . 4M for rados bench)

3) Just a single dd instance vs. 16 concurrent threads for rados bench.


Thanks,

Igor



On 8/28/2018 12:50 PM, Marc Roos wrote:

I have a idle test cluster (centos7.5, Linux c04
3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs.

I tested reading a few files on this cephfs mount and get very low
results compared to the rados bench. What could be the issue here?

[@client folder]# dd if=5GB.img of=/dev/null status=progress
954585600 bytes (955 MB) copied, 157.455633 s, 6.1 MB/s



I included this is rados bench that shows sort of that cluster
performance is sort of as expected.
[@c01 ~]# rados bench -p fs_data 10 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_453883
   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
 0   0 0 0 0 0   -
0
 1  165842   167.967   1680.252071
0.323443
 2  16   10690   179.967   1920.583383
0.324867
 3  16   139   123   163.973   1320.170865
0.325976
 4  16   183   167   166.975   1760.413676
0.361364
 5  16   224   208   166.374   1640.394369
0.365956
 6  16   254   238   158.642   1200.698396
0.382729
 7  16   278   262   149.692960.120742
0.397625
 8  16   317   301   150.478   1560.786822
0.411193
 9  16   360   344   152.867   1720.601956
0.411577
10  16   403   387   154.778   172 0.20342
0.404114
Total time run: 10.353683
Total writes made:  404
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 156.08
Stddev Bandwidth:   29.5778
Max bandwidth (MB/sec): 192
Min bandwidth (MB/sec): 96
Average IOPS:   39
Stddev IOPS:7
Max IOPS:   48
Min IOPS:   24
Average Latency(s): 0.409676
Stddev Latency(s):  0.243565
Max latency(s): 1.25028
Min latency(s): 0.0830112
Cleaning up (deleting benchmark objects)
Removed 404 objects
Clean up completed and total clean up time :0.867185




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deleting incomplete PGs from an erasure coded pool

[Paul Emmerich]

No need to delete it, that situation should be mostly salvagable by
setting osd_find_best_info_ignore_history_les temporarily on the
affected OSDs.
That should cause you to "just" lose some writes resulting in inconsistent data.


Paul

2018-08-28 11:08 GMT+02:00 Maks Kowalik :
> What is the correct procedure for re-creating an incomplete placement group
> that belongs to an erasure coded pool?
> I'm facing a situation when too many shards of 3 PGs were lost during OSD
> crashes, and taking the data loss was decided, but can't force ceph to
> recreate those PGs. The query output shows:
> "peering_blocked_by_detail": [
> {"detail": "peering_blocked_by_history_les_bound"}
> What was tried:
> 1. manual deletion of all shards appearing in "peers" secion of PG query
> output
> 2. marking all shards as complete using ceph-objectstore-tool
> 3. deleting peering history from OSDs keeping the shards
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Marc Roos]

 
I was not trying to compare the test results I know they are different. 
I am showing that reading is slow on cephfs (I am doing an rsync to 
cephfs and I assumed that rsync is just reading the file in a similar 
way)

And cluster is sort of in same ok state.

Meanwhile I did similar test with ceph-fuse, and getting what I am used 
to.


[@c04 folder]# dd if=file1 of=/dev/null status=progress
12305+1 records in
12305+1 records out
6300206 bytes (6.3 MB) copied, 0.100237 s, 62.9 MB/s
[@c04 folder]# dd if=file2 of=/dev/null status=progress
3116352000 bytes (3.1 GB) copied, 29.143809 s, 107 MB/ss
6209378+1 records in
6209378+1 records out
3179201945 bytes (3.2 GB) copied, 29.7547 s, 107 MB/s


-Original Message-
From: Igor Fedotov [mailto:ifedo...@suse.de] 
Sent: dinsdag 28 augustus 2018 11:59
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

Hi Marc,


In general dd isn't the best choice for benchmarking.

In you case there are at least 3 differences from rados bench :

1)If I haven't missed something then you're comparing reads vs. writes

2) Block Size is difference ( 512 bytes for dd vs . 4M for rados bench)

3) Just a single dd instance vs. 16 concurrent threads for rados bench.


Thanks,

Igor



On 8/28/2018 12:50 PM, Marc Roos wrote:
> I have a idle test cluster (centos7.5, Linux c04 
> 3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs.
>
> I tested reading a few files on this cephfs mount and get very low 
> results compared to the rados bench. What could be the issue here?
>
> [@client folder]# dd if=5GB.img of=/dev/null status=progress 954585600 

> bytes (955 MB) copied, 157.455633 s, 6.1 MB/s
>
>
>
> I included this is rados bench that shows sort of that cluster 
> performance is sort of as expected.
> [@c01 ~]# rados bench -p fs_data 10 write hints = 1 Maintaining 16 
> concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects Object prefix: 
> benchmark_data_c01_453883
>sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  
avg
> lat(s)
>  0   0 0 0 0 0   -
> 0
>  1  165842   167.967   1680.252071
> 0.323443
>  2  16   10690   179.967   1920.583383
> 0.324867
>  3  16   139   123   163.973   1320.170865
> 0.325976
>  4  16   183   167   166.975   1760.413676
> 0.361364
>  5  16   224   208   166.374   1640.394369
> 0.365956
>  6  16   254   238   158.642   1200.698396
> 0.382729
>  7  16   278   262   149.692960.120742
> 0.397625
>  8  16   317   301   150.478   1560.786822
> 0.411193
>  9  16   360   344   152.867   1720.601956
> 0.411577
> 10  16   403   387   154.778   172 0.20342
> 0.404114
> Total time run: 10.353683
> Total writes made:  404
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 156.08
> Stddev Bandwidth:   29.5778
> Max bandwidth (MB/sec): 192
> Min bandwidth (MB/sec): 96
> Average IOPS:   39
> Stddev IOPS:7
> Max IOPS:   48
> Min IOPS:   24
> Average Latency(s): 0.409676
> Stddev Latency(s):  0.243565
> Max latency(s): 1.25028
> Min latency(s): 0.0830112
> Cleaning up (deleting benchmark objects) Removed 404 objects Clean up 
> completed and total clean up time :0.867185
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Marc Roos]

 
Was there not some issue a while ago that was related to a kernel 
setting? Because I can remember doing some tests that ceph-fuse was 
always slower than the kernel module.



-Original Message-
From: Marc Roos 
Sent: dinsdag 28 augustus 2018 12:37
To: ceph-users; ifedotov
Subject: Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

 
I was not trying to compare the test results I know they are different. 
I am showing that reading is slow on cephfs (I am doing an rsync to 
cephfs and I assumed that rsync is just reading the file in a similar
way)

And cluster is sort of in same ok state.

Meanwhile I did similar test with ceph-fuse, and getting what I am used 
to.


[@c04 folder]# dd if=file1 of=/dev/null status=progress
12305+1 records in
12305+1 records out
6300206 bytes (6.3 MB) copied, 0.100237 s, 62.9 MB/s
[@c04 folder]# dd if=file2 of=/dev/null status=progress
3116352000 bytes (3.1 GB) copied, 29.143809 s, 107 MB/ss
6209378+1 records in
6209378+1 records out
3179201945 bytes (3.2 GB) copied, 29.7547 s, 107 MB/s


-Original Message-
From: Igor Fedotov [mailto:ifedo...@suse.de] 
Sent: dinsdag 28 augustus 2018 11:59
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

Hi Marc,


In general dd isn't the best choice for benchmarking.

In you case there are at least 3 differences from rados bench :

1)If I haven't missed something then you're comparing reads vs. writes

2) Block Size is difference ( 512 bytes for dd vs . 4M for rados bench)

3) Just a single dd instance vs. 16 concurrent threads for rados bench.


Thanks,

Igor



On 8/28/2018 12:50 PM, Marc Roos wrote:
> I have a idle test cluster (centos7.5, Linux c04 
> 3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs.
>
> I tested reading a few files on this cephfs mount and get very low 
> results compared to the rados bench. What could be the issue here?
>
> [@client folder]# dd if=5GB.img of=/dev/null status=progress 954585600 


> bytes (955 MB) copied, 157.455633 s, 6.1 MB/s
>
>
>
> I included this is rados bench that shows sort of that cluster 
> performance is sort of as expected.
> [@c01 ~]# rados bench -p fs_data 10 write hints = 1 Maintaining 16 
> concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects Object prefix: 
> benchmark_data_c01_453883
>sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  
avg
> lat(s)
>  0   0 0 0 0 0   -
> 0
>  1  165842   167.967   1680.252071
> 0.323443
>  2  16   10690   179.967   1920.583383
> 0.324867
>  3  16   139   123   163.973   1320.170865
> 0.325976
>  4  16   183   167   166.975   1760.413676
> 0.361364
>  5  16   224   208   166.374   1640.394369
> 0.365956
>  6  16   254   238   158.642   1200.698396
> 0.382729
>  7  16   278   262   149.692960.120742
> 0.397625
>  8  16   317   301   150.478   1560.786822
> 0.411193
>  9  16   360   344   152.867   1720.601956
> 0.411577
> 10  16   403   387   154.778   172 0.20342
> 0.404114
> Total time run: 10.353683
> Total writes made:  404
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 156.08
> Stddev Bandwidth:   29.5778
> Max bandwidth (MB/sec): 192
> Min bandwidth (MB/sec): 96
> Average IOPS:   39
> Stddev IOPS:7
> Max IOPS:   48
> Min IOPS:   24
> Average Latency(s): 0.409676
> Stddev Latency(s):  0.243565
> Max latency(s): 1.25028
> Min latency(s): 0.0830112
> Cleaning up (deleting benchmark objects) Removed 404 objects Clean up 
> completed and total clean up time :0.867185
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to put ceph-fuse fstab remote path?

[Marc Roos]

kernel
c01,c02,c03:/backup /home/backupceph
name=cephfs.backup,secretfile=/root/client.cephfs.backup.key,_netdev 0 0


c01,c02,c03:/backup /home/backup2   fuse.ceph   
ceph.id=cephfs.backup,_netdev 0 0
Mounts root cephfs 
c01,c02,c03:/backup /home/backup2   fuse.ceph   
ceph.id=cephfs.backup,r=/backup,_netdev 0 0
Gives error



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deleting incomplete PGs from an erasure coded pool

[Maks Kowalik]

Thank you for answering.
Where is this option documented?
Do I set it in the config file, or using "tell osd.number" or admin-daemon?
Do I set it on the primary OSD of the up set, on all OSDs of the up set, or
maybe on all historical peers holding the shards of a particular group?
Is this option dangerous to other groups on those OSDs (currently an OSD
holds about 160 pgs)?

Maks

wt., 28 sie 2018 o 12:12 Paul Emmerich  napisał(a):

> No need to delete it, that situation should be mostly salvagable by
> setting osd_find_best_info_ignore_history_les temporarily on the
> affected OSDs.
> That should cause you to "just" lose some writes resulting in inconsistent
> data.
>
>
> Paul
>
> 2018-08-28 11:08 GMT+02:00 Maks Kowalik :
> > What is the correct procedure for re-creating an incomplete placement
> group
> > that belongs to an erasure coded pool?
> > I'm facing a situation when too many shards of 3 PGs were lost during OSD
> > crashes, and taking the data loss was decided, but can't force ceph to
> > recreate those PGs. The query output shows:
> > "peering_blocked_by_detail": [
> > {"detail": "peering_blocked_by_history_les_bound"}
> > What was tried:
> > 1. manual deletion of all shards appearing in "peers" secion of PG query
> > output
> > 2. marking all shards as complete using ceph-objectstore-tool
> > 3. deleting peering history from OSDs keeping the shards
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Unrepairable PG

[Maks Kowalik]

Scrubs discovered the following inconsistency:

2018-08-23 17:21:07.933458 osd.62 osd.62 10.122.0.140:6805/77767 6 :
cluster [ERR] 9.3cd shard 113: soid
9:b3cd8d89:::.dir.default.153398310.112:head omap_digest 0xea4ba012 !=
omap_digest 0xc5acebfd from shard 62, omap_digest 0xea4ba012 != omap_digest
0xc5acebfd from auth oi
9:b3cd8d89:::.dir.default.153398310.112:head(138609'2009129
osd.250.0:64658209 dirty|omap|data_digest|omap_digest s 0 uv 1995230 dd
 od c5acebfd alloc_hint [0 0 0])

The omap_digest_mismatch appears on a non-primary OSD in a pool with 4
replicas. In this situation I decided to issue "pg repair" as I expected
ceph will repair the broken object. The command was successful but repair
on 9.3cd didn't start.

Then I have tried the procedure described here (setting a temporary key on
the object to force recalculation of omap_digest):
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47219.html
But deep-scrub on 9.3cd didn't start. The OSD marked the 9.3cd for
scrubbing, but that's all what happened:

2018-08-27 14:36:22.703848 7faa7e860700 20 osd.62 713813 OSD::ms_dispatch:
scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703869 7faa7e860700 20 osd.62 713813 _dispatch
0x55725b76d180 scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703871 7faa7e860700 10 osd.62 713813 handle_scrub
scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703878 7faa7e860700 10 osd.62 713813 marking pg[9.3cd(
v 713813'2359292 (713107'2357731,713813'2359292]
local-lis/les=711049/711050 n=41419 ec=178/178 lis/c 711049/711049 les/c/f
711050/711149/222921 711049/711049/710352) [62,53,163,113] r=0 lpr=711049
crt=713813'2359292 lcod 713813'2359291 mlcod 713813'2359291
active+clean+inconsistent MUST_DEEP_SCRUB MUST_SCRUB] for scrub

Does anyone know how to recover from inconsistency in such case?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deleting incomplete PGs from an erasure coded pool

[Paul Emmerich]

I don't think it's documented.

It won't affect PGs that are active+clean.
Takes effect during peering, easiest to set it in ceph.conf and
restart the OSDs on *all* OSDs that you want to rescue.
Important to not forget to unset it afterwards


Paul

2018-08-28 13:21 GMT+02:00 Maks Kowalik :
> Thank you for answering.
> Where is this option documented?
> Do I set it in the config file, or using "tell osd.number" or admin-daemon?
> Do I set it on the primary OSD of the up set, on all OSDs of the up set, or
> maybe on all historical peers holding the shards of a particular group?
> Is this option dangerous to other groups on those OSDs (currently an OSD
> holds about 160 pgs)?
>
> Maks
>
> wt., 28 sie 2018 o 12:12 Paul Emmerich  napisał(a):
>>
>> No need to delete it, that situation should be mostly salvagable by
>> setting osd_find_best_info_ignore_history_les temporarily on the
>> affected OSDs.
>> That should cause you to "just" lose some writes resulting in inconsistent
>> data.
>>
>>
>> Paul
>>
>> 2018-08-28 11:08 GMT+02:00 Maks Kowalik :
>> > What is the correct procedure for re-creating an incomplete placement
>> > group
>> > that belongs to an erasure coded pool?
>> > I'm facing a situation when too many shards of 3 PGs were lost during
>> > OSD
>> > crashes, and taking the data loss was decided, but can't force ceph to
>> > recreate those PGs. The query output shows:
>> > "peering_blocked_by_detail": [
>> > {"detail": "peering_blocked_by_history_les_bound"}
>> > What was tried:
>> > 1. manual deletion of all shards appearing in "peers" secion of PG query
>> > output
>> > 2. marking all shards as complete using ceph-objectstore-tool
>> > 3. deleting peering history from OSDs keeping the shards
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Delay replicate for ceph radosgw multi-site v2

[Jason Dillaman]

On Mon, Aug 27, 2018 at 11:19 PM đức phạm xuân  wrote:
>
> Hello Jason Dillaman,
>
> I'm working with Ceph Object Storage Multi-Site v2, ceph's version is mimic. 
> Now I want to delay replicate data from a master site to a slave site. I 
> don't know whether dose ceph has support the mechanism?

To be honest, I've never worked with RGW multisite so I am afraid I
can't immediately answer your question. I've CCed the ceph-users list
so that perhaps someone else that is more knowledgeable can answer.

> --
> Phạm Xuân Đức
> Sinh viên Học Viện Kỹ thuật Mật Mã - khóa AT11
> Mobile: +84165 417 1434
> Skype: pxduc96
> Email: ducp...@gmail.com

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Whole cluster flapping

[CUZA Frédéric]

Hi,

Just to inform that I finally resolved my problem few weeks ago now but I 
wanted to make sure that it was solved permanently.
I set the timeout of OSDs to a larger number of seconds and set the no out and 
no down flag on the cluster.
Basically I just waited that the “clean” ended but I noticed that some OSDs 
where doing more timeout than the others (since the datas were no longer 
important for me) I started to take OSDs out and purging them one by one.
I did this for about 1/8 of my OSDs and now there is no more flapping or OSDs 
going down but I still have some slow query popping up time to time.

Seems that the pool purged and deleted couldn’t make it till “the end” of it.
By purging the remaining OSDs where datas seemed to be helped to bring back the 
cluster to a stable state.

Thank you all for your help.

Regards,

De : Will Marley 
Envoyé : 08 August 2018 16:14
À : Webert de Souza Lima ; CUZA Frédéric 

Cc : ceph-users 
Objet : RE: [ceph-users] Whole cluster flapping

Hi again Frederic,

It may be worth looking at a recovery sleep.
osd recovery sleep
Description:

Time in seconds to sleep before next recovery or backfill op. Increasing this 
value will slow down recovery operation while client operations will be less 
impacted.

Type:

Float

Default:

0

osd recovery sleep hdd
Description:

Time in seconds to sleep before next recovery or backfill op for HDDs.

Type:

Float

Default:

0.1

osd recovery sleep ssd
Description:

Time in seconds to sleep before next recovery or backfill op for SSDs.

Type:

Float

Default:

0

osd recovery sleep hybrid
Description:

Time in seconds to sleep before next recovery or backfill op when osd data is 
on HDD and osd journal is on SSD.

Type:

Float

Default:

0.025


(Pulled from 
http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/)

When we faced similar issues, using the command ceph tell osd.* injectargs 
'--osd-recovery-sleep 2 allowed the OSDs to respond with a heartbeat whilst 
taking a break between recovery operations. I’d suggest tweaking the sleep wait 
time to find a sweet spot.

This may be worth a try, so let us know how you get on.

Regards,
Will

From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
On Behalf Of Webert de Souza Lima
Sent: 08 August 2018 15:06
To: frederic.c...@sib.fr
Cc: ceph-users mailto:ceph-users@lists.ceph.com>>
Subject: Re: [ceph-users] Whole cluster flapping

So your OSDs are really too busy to respond heartbeats.
You'll be facing this for sometime until cluster loads get lower.

I would set `ceph osd set nodeep-scrub` until the heavy disk IO stops.
maybe you can schedule it for enable during the night and disabling in the 
morning.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Wed, Aug 8, 2018 at 9:18 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Thx for the command line, I did take a look too it what I don’t really know 
what to search for, my bad….
All this flapping is due to deep-scrub when it starts on an OSD things start to 
go bad.

I set out all the OSDs that were flapping the most (1 by 1 after rebalancing) 
and it looks better even if some osds keep going down/up with the same message 
in logs :

1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fdabd897700' had timed out 
after 90

(I update it to 90 instead of 15s)

Regards,



De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 07 August 2018 16:28
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

oops, my bad, you're right.

I don't know much you can see but maybe you can dig around performance counters 
and see what's happening on those OSDs, try these:

~# ceph daemonperf osd.XX
~# ceph daemon osd.XX perf dump

change XX to your OSD numbers.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 10:47 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Pool is already deleted and no longer present in stats.

Regards,

De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 07 August 2018 15:08
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

Frédéric,

see if the number of objects is decreasing in the pool with `ceph df [detail]`

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 5:46 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
It’s been over a week now and the whole cluster keeps flapping, it is never the 
same OSDs that go down.
Is there a way to get the progress of this recovery ? (The pool hat I deleted 
is no longer present (for a while now))
In fact, there is a lot of i/o activity on the server where osds go down.

Regards,

De : ceph-users 
mailto:ceph-users-bo

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Yan, Zheng]

It's a bug. search thread "Poor CentOS 7.5 client performance" in ceph-users.
On Tue, Aug 28, 2018 at 2:50 AM Marc Roos  wrote:
>
>
> I have a idle test cluster (centos7.5, Linux c04
> 3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs.
>
> I tested reading a few files on this cephfs mount and get very low
> results compared to the rados bench. What could be the issue here?
>
> [@client folder]# dd if=5GB.img of=/dev/null status=progress
> 954585600 bytes (955 MB) copied, 157.455633 s, 6.1 MB/s
>
>
>
> I included this is rados bench that shows sort of that cluster
> performance is sort of as expected.
> [@c01 ~]# rados bench -p fs_data 10 write
> hints = 1
> Maintaining 16 concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects
> Object prefix: benchmark_data_c01_453883
>   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
> lat(s)
> 0   0 0 0 0 0   -
>0
> 1  165842   167.967   1680.252071
> 0.323443
> 2  16   10690   179.967   1920.583383
> 0.324867
> 3  16   139   123   163.973   1320.170865
> 0.325976
> 4  16   183   167   166.975   1760.413676
> 0.361364
> 5  16   224   208   166.374   1640.394369
> 0.365956
> 6  16   254   238   158.642   1200.698396
> 0.382729
> 7  16   278   262   149.692960.120742
> 0.397625
> 8  16   317   301   150.478   1560.786822
> 0.411193
> 9  16   360   344   152.867   1720.601956
> 0.411577
>10  16   403   387   154.778   172 0.20342
> 0.404114
> Total time run: 10.353683
> Total writes made:  404
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 156.08
> Stddev Bandwidth:   29.5778
> Max bandwidth (MB/sec): 192
> Min bandwidth (MB/sec): 96
> Average IOPS:   39
> Stddev IOPS:7
> Max IOPS:   48
> Min IOPS:   24
> Average Latency(s): 0.409676
> Stddev Latency(s):  0.243565
> Max latency(s): 1.25028
> Min latency(s): 0.0830112
> Cleaning up (deleting benchmark objects)
> Removed 404 objects
> Clean up completed and total clean up time :0.867185
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Marc Roos]

 
Thanks!!!

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg46212.html
echo 8192 >/sys/devices/virtual/bdi/ceph-1/read_ahead_kb



-Original Message-
From: Yan, Zheng [mailto:uker...@gmail.com] 
Sent: dinsdag 28 augustus 2018 15:44
To: Marc Roos
Cc: ceph-users
Subject: Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

It's a bug. search thread "Poor CentOS 7.5 client performance" in 
ceph-users.
On Tue, Aug 28, 2018 at 2:50 AM Marc Roos  
wrote:
>
>
> I have a idle test cluster (centos7.5, Linux c04 
> 3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs.
>
> I tested reading a few files on this cephfs mount and get very low 
> results compared to the rados bench. What could be the issue here?
>
> [@client folder]# dd if=5GB.img of=/dev/null status=progress 954585600 

> bytes (955 MB) copied, 157.455633 s, 6.1 MB/s
>
>
>
> I included this is rados bench that shows sort of that cluster 
> performance is sort of as expected.
> [@c01 ~]# rados bench -p fs_data 10 write hints = 1 Maintaining 16 
> concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects Object prefix: 
> benchmark_data_c01_453883
>   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
> lat(s)
> 0   0 0 0 0 0   -
>0
> 1  165842   167.967   1680.252071
> 0.323443
> 2  16   10690   179.967   1920.583383
> 0.324867
> 3  16   139   123   163.973   1320.170865
> 0.325976
> 4  16   183   167   166.975   1760.413676
> 0.361364
> 5  16   224   208   166.374   1640.394369
> 0.365956
> 6  16   254   238   158.642   1200.698396
> 0.382729
> 7  16   278   262   149.692960.120742
> 0.397625
> 8  16   317   301   150.478   1560.786822
> 0.411193
> 9  16   360   344   152.867   1720.601956
> 0.411577
>10  16   403   387   154.778   172 0.20342
> 0.404114
> Total time run: 10.353683
> Total writes made:  404
> Write size: 4194304
> Object size:4194304
> Bandwidth (MB/sec): 156.08
> Stddev Bandwidth:   29.5778
> Max bandwidth (MB/sec): 192
> Min bandwidth (MB/sec): 96
> Average IOPS:   39
> Stddev IOPS:7
> Max IOPS:   48
> Min IOPS:   24
> Average Latency(s): 0.409676
> Stddev Latency(s):  0.243565
> Max latency(s): 1.25028
> Min latency(s): 0.0830112
> Cleaning up (deleting benchmark objects) Removed 404 objects Clean up 
> completed and total clean up time :0.867185
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Deleting incomplete PGs from an erasure coded pool

[Maks Kowalik]

I don't want to "rescue" any OSDs. I want to clean the incomplete PGs to
make CEPH proceed with PG re-creation and making those groups active again.
In my case which OSDs should I start with the
"osd_find_best_info_ignore_history_les" option?
This is the part of query output from one of the groups to be cleared:
"probing_osds": [ "54(1)", "81(2)", "103(0)", "103(1)", "118(9)", "126(3)",
"129(4)", "141(1)", "142(2)", "147(7)", "150(1)", "153(8)",
"159(0)","165(6)", "168(5)",
"171(0)","174(3)","177(9)","180(5)","262(2)","291(5)","313(1)","314(8)","315(7)","316(0)","318(6)"],
"down_osds_we_would_probe": [4,88,91,94,112,133]

Maks

wt., 28 sie 2018 o 15:20 Paul Emmerich  napisał(a):

> I don't think it's documented.
>
> It won't affect PGs that are active+clean.
> Takes effect during peering, easiest to set it in ceph.conf and
> restart the OSDs on *all* OSDs that you want to rescue.
> Important to not forget to unset it afterwards
>
>
> Paul
>
> 2018-08-28 13:21 GMT+02:00 Maks Kowalik :
> > Thank you for answering.
> > Where is this option documented?
> > Do I set it in the config file, or using "tell osd.number" or
> admin-daemon?
> > Do I set it on the primary OSD of the up set, on all OSDs of the up set,
> or
> > maybe on all historical peers holding the shards of a particular group?
> > Is this option dangerous to other groups on those OSDs (currently an OSD
> > holds about 160 pgs)?
> >
> > Maks
> >
> > wt., 28 sie 2018 o 12:12 Paul Emmerich 
> napisał(a):
> >>
> >> No need to delete it, that situation should be mostly salvagable by
> >> setting osd_find_best_info_ignore_history_les temporarily on the
> >> affected OSDs.
> >> That should cause you to "just" lose some writes resulting in
> inconsistent
> >> data.
> >>
> >>
> >> Paul
> >>
> >> 2018-08-28 11:08 GMT+02:00 Maks Kowalik :
> >> > What is the correct procedure for re-creating an incomplete placement
> >> > group
> >> > that belongs to an erasure coded pool?
> >> > I'm facing a situation when too many shards of 3 PGs were lost during
> >> > OSD
> >> > crashes, and taking the data loss was decided, but can't force ceph to
> >> > recreate those PGs. The query output shows:
> >> > "peering_blocked_by_detail": [
> >> > {"detail": "peering_blocked_by_history_les_bound"}
> >> > What was tried:
> >> > 1. manual deletion of all shards appearing in "peers" secion of PG
> query
> >> > output
> >> > 2. marking all shards as complete using ceph-objectstore-tool
> >> > 3. deleting peering history from OSDs keeping the shards
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >>
> >>
> >>
> >> --
> >> Paul Emmerich
> >>
> >> Looking for help with your Ceph cluster? Contact us at https://croit.io
> >>
> >> croit GmbH
> >> Freseniusstr. 31h
> >> 81247 München
> >> www.croit.io
> >> Tel: +49 89 1896585 90
>
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Installing ceph 12.2.4 via Ubuntu apt

[David Turner]

That is the expected behavior of the ceph repo. In the past when I needed a
specific version I would download the packages for the version to a folder
and you can create a repo file that reads from a local directory. That's
how I would re-install my test lab after testing an upgrade procedure to
try it over again.

On Tue, Aug 28, 2018, 1:01 AM Thomas Bennett  wrote:

> Hi,
>
> I'm wanting to pin to an older version of Ceph Luminous (12.2.4) and I've
> noticed that https://download.ceph.com/debian-luminous/ does not support
> this via apt install:
> apt install ceph works for 12.2.7 but
> apt install ceph=12.2.4-1xenial does not work
>
> The deb file are there, they're just not included in the package
> distribution. Is this the desired behaviour or a misconfiguration?
>
> Cheers,
> Tom
>
> --
> Thomas Bennett
>
> SARAO
> Science Data Processing
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unrepairable PG

[David Turner]

There is a [1] tracker open for this issue. There are 2 steps that should
get a pg to scrub/repair that is just issuing the scrub, but not running
it. First is to increase osd_max_scrubs on the OSDs involved in the PG. If
that doesn't fix it, then try increasing your osd_deep_scrub_interval on
all osds in your cluster. Both settings can be injected and in my
experience that should allow your PG to repair/deep-scrub.

The idea is that your cluster isn't able to keep up with the deep-scrub
schedule and the deep-scrubs being forced to run by the cluster due to the
interval are higher priority than the ones you manually submit. That was
definitely the case when I had this problem a few weeks ago and these steps
resolved it. When I had it a few months ago I just let it run its course
and the repair finally happened about 3 weeks after I issued it. My
osd_deep_scrub_interval was set to 30 days, but apparently it was taking
closer to 7 weeks to get through all of the PGs.


[1] https://tracker.ceph.com/issues/23576#change-119460

On Tue, Aug 28, 2018, 5:16 AM Maks Kowalik  wrote:

> Scrubs discovered the following inconsistency:
>
> 2018-08-23 17:21:07.933458 osd.62 osd.62 10.122.0.140:6805/77767 6 :
> cluster [ERR] 9.3cd shard 113: soid
> 9:b3cd8d89:::.dir.default.153398310.112:head omap_digest 0xea4ba012 !=
> omap_digest 0xc5acebfd from shard 62, omap_digest 0xea4ba012 != omap_digest
> 0xc5acebfd from auth oi
> 9:b3cd8d89:::.dir.default.153398310.112:head(138609'2009129
> osd.250.0:64658209 dirty|omap|data_digest|omap_digest s 0 uv 1995230 dd
>  od c5acebfd alloc_hint [0 0 0])
>
> The omap_digest_mismatch appears on a non-primary OSD in a pool with 4
> replicas. In this situation I decided to issue "pg repair" as I expected
> ceph will repair the broken object. The command was successful but repair
> on 9.3cd didn't start.
>
> Then I have tried the procedure described here (setting a temporary key on
> the object to force recalculation of omap_digest):
> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg47219.html
> But deep-scrub on 9.3cd didn't start. The OSD marked the 9.3cd for
> scrubbing, but that's all what happened:
>
> 2018-08-27 14:36:22.703848 7faa7e860700 20 osd.62 713813 OSD::ms_dispatch:
> scrub([9.3cd] deep) v2
> 2018-08-27 14:36:22.703869 7faa7e860700 20 osd.62 713813 _dispatch
> 0x55725b76d180 scrub([9.3cd] deep) v2
> 2018-08-27 14:36:22.703871 7faa7e860700 10 osd.62 713813 handle_scrub
> scrub([9.3cd] deep) v2
> 2018-08-27 14:36:22.703878 7faa7e860700 10 osd.62 713813 marking pg[9.3cd(
> v 713813'2359292 (713107'2357731,713813'2359292]
> local-lis/les=711049/711050 n=41419 ec=178/178 lis/c 711049/711049 les/c/f
> 711050/711149/222921 711049/711049/710352) [62,53,163,113] r=0 lpr=711049
> crt=713813'2359292 lcod 713813'2359291 mlcod 713813'2359291
> active+clean+inconsistent MUST_DEEP_SCRUB MUST_SCRUB] for scrub
>
> Does anyone know how to recover from inconsistency in such case?
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Mimic - Erasure Code Plugin recommendation

[Steven Vacaroaia]

Hi,

Would you be able to recommend erasure code plugin ?
The default is jerasure but lrc appears to be more efficient

Ill appreciate any hints and/or pointers to resources   / best practices

Thanks
Steven
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to put ceph-fuse fstab remote path?

[David Turner]

This is what we have for our fstab to mount a specific subfolder using
ceph-fuse

id=cephfs-backup,client_mountpoint=/backup   /home/backup2
fuse.ceph   _netdev,noatime,rw  0 0

On Tue, Aug 28, 2018 at 4:04 AM Marc Roos  wrote:

>
> kernel
> c01,c02,c03:/backup /home/backupceph
> name=cephfs.backup,secretfile=/root/client.cephfs.backup.key,_netdev 0 0
>
>
> c01,c02,c03:/backup /home/backup2   fuse.ceph
> ceph.id=cephfs.backup,_netdev 0 0
> Mounts root cephfs
> c01,c02,c03:/backup /home/backup2   fuse.ceph
> ceph.id=cephfs.backup,r=/backup,_netdev 0 0
> Gives error
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs slow 6MB/s and rados bench sort of ok.

[Dietmar Rieder]

Try to update to kernel-3.10.0-862.11.6.el7.x86_64.rpm that should solve the 
problem.

Best
 Dietmar

Am 28. August 2018 11:50:31 MESZ schrieb Marc Roos :
>
>I have a idle test cluster (centos7.5, Linux c04 
>3.10.0-862.9.1.el7.x86_64), and a client kernel mount cephfs. 
>
>I tested reading a few files on this cephfs mount and get very low 
>results compared to the rados bench. What could be the issue here?
>
>[@client folder]# dd if=5GB.img of=/dev/null status=progress
>954585600 bytes (955 MB) copied, 157.455633 s, 6.1 MB/s
>
>
>
>I included this is rados bench that shows sort of that cluster 
>performance is sort of as expected.
>[@c01 ~]# rados bench -p fs_data 10 write
>hints = 1
>Maintaining 16 concurrent writes of 4194304 bytes to objects of size 
>4194304 for up to 10 seconds or 0 objects
>Object prefix: benchmark_data_c01_453883
>  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg 
>lat(s)
>  0   0 0 0 0 0   -
>   0
>1  165842   167.967   1680.252071
>0.323443
>2  16   10690   179.967   1920.583383
>0.324867
>3  16   139   123   163.973   1320.170865
>0.325976
>4  16   183   167   166.975   1760.413676
>0.361364
>5  16   224   208   166.374   1640.394369
>0.365956
>6  16   254   238   158.642   1200.698396
>0.382729
>7  16   278   262   149.692960.120742
>0.397625
>8  16   317   301   150.478   1560.786822
>0.411193
>9  16   360   344   152.867   1720.601956
>0.411577
>   10  16   403   387   154.778   172 0.20342
>0.404114
>Total time run: 10.353683
>Total writes made:  404
>Write size: 4194304
>Object size:4194304
>Bandwidth (MB/sec): 156.08
>Stddev Bandwidth:   29.5778
>Max bandwidth (MB/sec): 192
>Min bandwidth (MB/sec): 96
>Average IOPS:   39
>Stddev IOPS:7
>Max IOPS:   48
>Min IOPS:   24
>Average Latency(s): 0.409676
>Stddev Latency(s):  0.243565
>Max latency(s): 1.25028
>Min latency(s): 0.0830112
>Cleaning up (deleting benchmark objects)
>Removed 404 objects
>Clean up completed and total clean up time :0.867185
>
>
>
>
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
___
D i e t m a r R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Division for Bioinformatics
Innrain 80, 6020 Innsbruck
Phone: +43 512 9003 71402
Fax: +43 512 9003 73100
Email: dietmar.rie...@i-med.ac.at
Web: http://www.icbi.at
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Delay replicate for ceph radosgw multi-site v2

[Casey Bodley]

On 08/28/2018 09:24 AM, Jason Dillaman wrote:

On Mon, Aug 27, 2018 at 11:19 PM đức phạm xuân  wrote:

Hello Jason Dillaman,

I'm working with Ceph Object Storage Multi-Site v2, ceph's version is mimic. 
Now I want to delay replicate data from a master site to a slave site. I don't 
know whether dose ceph has support the mechanism?

To be honest, I've never worked with RGW multisite so I am afraid I
can't immediately answer your question. I've CCed the ceph-users list
so that perhaps someone else that is more knowledgeable can answer.


--
Phạm Xuân Đức
Sinh viên Học Viện Kỹ thuật Mật Mã - khóa AT11
Mobile: +84165 417 1434
Skype: pxduc96
Email: ducp...@gmail.com


There isn't really a mechanism for this, no. Could you provide some more 
details about what exactly you're trying to accomplish?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] SAN or DAS for Production ceph

[James Watson]

Dear cephers,

I am new to the storage domain.
Trying to get my head around the enterprise - production-ready setup.

The following article helps a lot here: (Yahoo ceph implementation)
https://yahooeng.tumblr.com/tagged/object-storage

But a couple of questions:

What HDD would they have used here? NVMe / SATA /SAS etc (with just 52
storage node they got 3.2 PB of capacity !! )
I try to calculate a similar setup with HGST Ultrastar He12 (12TB and it's
more recent ) and would need 86 HDDs that adds up to 1 PB only!!

How is the HDD drive attached is it DAS or a SAN (using Fibre Channel
Switches, Host Bus Adapters etc)?

Do we need a proprietary hashing algorithm to implement multi-cluster
based setup of ceph to contain CPU/Memory usage within the cluster when
rebuilding happens during device failure?

If proprietary hashing algorithm is required to setup multi-cluster ceph
using load balancer - then what could be the alternative setup we can
deploy to address the same issue?

The aim is to design a similar architecture but with upgraded products and
higher performance. - Any suggestions or thoughts are welcome



Thanks in advance
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SAN or DAS for Production ceph

[John Hearns]

James, well for a start don't use a SAN. I speak as someone who managed a
SAN with Brocade switches and multipathing for an F1 team. CEPH is Software
Defined Storage. You want discreet storage servers with a high bandwidth
Ethernet (or maybe Infiniband) fabric.

Fibrechannel still has it place here though if you want servers with FC
attached JBODs.

Also you ask about the choice between spinning disks, SSDs and NVMe drives.
Think about the COST for your petabyte archive.
True, these days you can argue that all SSD could be comparable to spinning
disks. But NVMe? Yes you get the best performance.. but do you really want
all that video data on $$$ NVMe? You need tiering.

Also dont forget low and slow archive tiers - shingled archive disks and
perhaps tape.

Me, I would start from the building blocks of Supermicro 36 bay storage
servers. Fill them with 12 Tbyte helium drives.
Two slots in the back for SSDs for your journaling.
For a higher performance tier, look at the 'double double' storage servers
from Supermicro. Or even nicer the new 'ruler'form factor servers.
For a higher density archiving tier the 90 bay Supermicro servers.

Please get in touch with someone for advice. If you are in the UK I am
happy to help and point you in the right direction.














On Tue, 28 Aug 2018 at 21:05, James Watson  wrote:

> Dear cephers,
>
> I am new to the storage domain.
> Trying to get my head around the enterprise - production-ready setup.
>
> The following article helps a lot here: (Yahoo ceph implementation)
> https://yahooeng.tumblr.com/tagged/object-storage
>
> But a couple of questions:
>
> What HDD would they have used here? NVMe / SATA /SAS etc (with just 52
> storage node they got 3.2 PB of capacity !! )
> I try to calculate a similar setup with HGST Ultrastar He12 (12TB and it's
> more recent ) and would need 86 HDDs that adds up to 1 PB only!!
>
> How is the HDD drive attached is it DAS or a SAN (using Fibre Channel
> Switches, Host Bus Adapters etc)?
>
> Do we need a proprietary hashing algorithm to implement multi-cluster
> based setup of ceph to contain CPU/Memory usage within the cluster when
> rebuilding happens during device failure?
>
> If proprietary hashing algorithm is required to setup multi-cluster ceph
> using load balancer - then what could be the alternative setup we can
> deploy to address the same issue?
>
> The aim is to design a similar architecture but with upgraded products and
> higher performance. - Any suggestions or thoughts are welcome
>
>
>
> Thanks in advance
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SAN or DAS for Production ceph

[Thomas White]

Hi James,

 

I can see where some of the confusion has arisen, hopefully I can put at least 
some of it to rest. In the Tumblr post from Yahoo, the keyword to look out for 
is “nodes”, which is distinct from individual hard drives which in Ceph is an 
OSD in most cases. So you would have multiple OSDs per node.

 

My quick napkin math would suggest that they are using 54 storage nodes, each 
holding 16 drives/OSDs (this doesn’t count the OS drives which aren’t specified 
in the post), as with the below math:

 

54 storage nodes providing 3.2PB of raw store requires ~59.25TB of storage per 
node

59.25TB / 12 = 4.94TB per OSD

59.25TB / 14 = 4.32TB per OSD

59.25TB / 16 = 3.70TB per OSD

 

Total OSDs per cluster = 864

EC Calculation: 8 / (8+3) = 72.73%

 

As they are using an 8/3 erasure coding configuration, that would provide an 
efficiency of 72.73% (see EC Calculation), so the usable capacity per storage 
cluster is around 2.33PB.

 

I haven’t included the calculation for anything below 12 as while it is 
possible, I find the 16 drive configuration most probable. As Ceph crush weight 
is shown using TiB, but most hard drives are marketed in TB due to the higher 
value, that would mean that 4TB drives are in use providing 3.63TiB of usable 
space on the drive. The math isn’t perfect here as you can see, but I’d think 
it is a safe assumption that they have at least a few higher capacity drives in 
there, or a wider mix of such standard commodity drive sizes with 4TB simply 
being a decent average.

 

For object storage clusters, particularly in use cases of high volumes of small 
objects, a standard OSD/node density is preferable which hovers between 10 and 
16 OSDs per server depending who you ask (some reading on the subject courtesy 
of RedHat 
https://www.redhat.com/cms/managed-files/st-ceph-storage-qct-object-storage-reference-architecture-f7901-201706-v2-en.pdf).
  As Yahoo’s workload is noting consistency and latency are some important 
metrics, they are also likely to use this density profile rather than something 
higher – this has the added benefit of quicker recovery times in the event of 
an individual OSD/host failure which is a parameter they tuned quite 
extensively.

 

For hashing algorithms and load balancing, I am not quite sure I understand 
your question, but RGW which implements object storage in Ceph has the ability 
to configure multiple zones/groups/regions, it might be best to have a read 
through the docs first:

http://docs.ceph.com/docs/luminous/radosgw/multisite/

 

Ceph is quite different from a SAN or DAS, and gives a great deal more 
flexibility too. If you are unsure on getting started and you need to hit the 
ground running strongly (ie a multi-PB production system), I’d really recommend 
getting a reliable consultant or taking out professional support services for 
it. Ceph is a piece of cake to manage when everything is working well, and very 
often this will be the case for a long time, but you will really value good 
planning and experience when you hit those rough patches.

 

Hope that helps,

 

Tom

 

 

From: ceph-users  On Behalf Of James Watson
Sent: 28 August 2018 21:05
To: ceph-users@lists.ceph.com
Subject: [ceph-users] SAN or DAS for Production ceph

 

Dear cephers, 

 

I am new to the storage domain. 

Trying to get my head around the enterprise - production-ready setup. 

 

The following article helps a lot here: (Yahoo ceph implementation)

https://yahooeng.tumblr.com/tagged/object-storage

 

But a couple of questions:

 

What HDD would they have used here? NVMe / SATA /SAS etc (with just 52 storage 
node they got 3.2 PB of capacity !! )

I try to calculate a similar setup with HGST Ultrastar He12 (12TB and it's more 
recent ) and would need 86 HDDs that adds up to 1 PB only!!

 

How is the HDD drive attached is it DAS or a SAN (using Fibre Channel Switches, 
Host Bus Adapters etc)?

 

Do we need a proprietary hashing algorithm to implement multi-cluster based 
setup of ceph to contain CPU/Memory usage within the cluster when rebuilding 
happens during device failure?

 

If proprietary hashing algorithm is required to setup multi-cluster ceph using 
load balancer - then what could be the alternative setup we can deploy to 
address the same issue?

 

The aim is to design a similar architecture but with upgraded products and 
higher performance. - Any suggestions or thoughts are welcome 

 

 

 

Thanks in advance

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SAN or DAS for Production ceph

[John Hearns]

James, you also use the words enterprise and production ready.
Is Redhat support important to you?




On Tue, 28 Aug 2018 at 23:56, John Hearns  wrote:

> James, well for a start don't use a SAN. I speak as someone who managed a
> SAN with Brocade switches and multipathing for an F1 team. CEPH is Software
> Defined Storage. You want discreet storage servers with a high bandwidth
> Ethernet (or maybe Infiniband) fabric.
>
> Fibrechannel still has it place here though if you want servers with FC
> attached JBODs.
>
> Also you ask about the choice between spinning disks, SSDs and NVMe
> drives. Think about the COST for your petabyte archive.
> True, these days you can argue that all SSD could be comparable to
> spinning disks. But NVMe? Yes you get the best performance.. but do you
> really want all that video data on $$$ NVMe? You need tiering.
>
> Also dont forget low and slow archive tiers - shingled archive disks and
> perhaps tape.
>
> Me, I would start from the building blocks of Supermicro 36 bay storage
> servers. Fill them with 12 Tbyte helium drives.
> Two slots in the back for SSDs for your journaling.
> For a higher performance tier, look at the 'double double' storage servers
> from Supermicro. Or even nicer the new 'ruler'form factor servers.
> For a higher density archiving tier the 90 bay Supermicro servers.
>
> Please get in touch with someone for advice. If you are in the UK I am
> happy to help and point you in the right direction.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, 28 Aug 2018 at 21:05, James Watson  wrote:
>
>> Dear cephers,
>>
>> I am new to the storage domain.
>> Trying to get my head around the enterprise - production-ready setup.
>>
>> The following article helps a lot here: (Yahoo ceph implementation)
>> https://yahooeng.tumblr.com/tagged/object-storage
>>
>> But a couple of questions:
>>
>> What HDD would they have used here? NVMe / SATA /SAS etc (with just 52
>> storage node they got 3.2 PB of capacity !! )
>> I try to calculate a similar setup with HGST Ultrastar He12 (12TB and
>> it's more recent ) and would need 86 HDDs that adds up to 1 PB only!!
>>
>> How is the HDD drive attached is it DAS or a SAN (using Fibre Channel
>> Switches, Host Bus Adapters etc)?
>>
>> Do we need a proprietary hashing algorithm to implement multi-cluster
>> based setup of ceph to contain CPU/Memory usage within the cluster when
>> rebuilding happens during device failure?
>>
>> If proprietary hashing algorithm is required to setup multi-cluster ceph
>> using load balancer - then what could be the alternative setup we can
>> deploy to address the same issue?
>>
>> The aim is to design a similar architecture but with upgraded products
>> and higher performance. - Any suggestions or thoughts are welcome
>>
>>
>>
>> Thanks in advance
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS Quota and ACL support

[Oliver Freyermuth]

Am 28.08.18 um 07:14 schrieb Yan, Zheng:
> On Mon, Aug 27, 2018 at 10:53 AM Oliver Freyermuth
>  wrote:
>>
>> Thanks for the replies.
>>
>> Am 27.08.18 um 19:25 schrieb Patrick Donnelly:
>>> On Mon, Aug 27, 2018 at 12:51 AM, Oliver Freyermuth
>>>  wrote:
 These features are critical for us, so right now we use the Fuse client. 
 My hope is CentOS 8 will use a recent enough kernel
 to get those features automatically, though.
>>>
>>> Your cluster needs to be running Mimic and Linux v4.17+.
>>>
>>> See also: https://github.com/ceph/ceph/pull/23728/files
>>>
>>
>> Yes, I know that it's part of the official / vanilla kernel as of 4.17.
>> However, I was wondering whether this functionality is also likely to be 
>> backported to the RedHat-maintained kernel which is also used in CentOS 7?
>> Even though the kernel version is "stone-aged", it matches CentOS 7's 
>> userspace and RedHat is taking good care to implement fixes.
>>
> 
> We have already backported quota patches to RHEL 3.10 kernel. It may
> take some time for redhat to release the new kernel.

That's great news, many thanks - looking forward to it! 
I also noted the CephFS kernel client is now mentioned as "fully supported" 
with the upcoming RHEL 7.6: 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7-beta/html-single/7.6_release_notes/index#new_features_file_systems
Those release notes still talk about missing quota support, but I guess this 
will then be added soonish :-). 

All the best,
Oliver

> 
> Regards
> Yan, Zheng
> 
>> Seeing that even features are backported, it would be really helpful if also 
>> this functionality would appear as part of CentOS 7.6 / 7.7,
>> especially since CentOS 8 still appears to be quite some time away.
>>
>> Cheers,
>> Oliver
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore crashing constantly with load on newly created cluster/host.

[Tyler Bishop]

After moving back to tcmalloc my random crash issues have been resolved.
 I would advise disabling support for jemalloc on bluestore since its not
stable or safe... seems risky to allow this?
_

*Tyler Bishop*
EST 2007


O: 513-299-7108 x1000
M: 513-646-5809
http://BeyondHosting.net 


This email is intended only for the recipient(s) above and/or
otherwise authorized personnel. The information contained herein and
attached is confidential and the property of Beyond Hosting. Any
unauthorized copying, forwarding, printing, and/or disclosing
any information related to this email is prohibited. If you received this
message in error, please contact the sender and destroy all copies of this
email and any attachment(s).


On Mon, Aug 27, 2018 at 11:15 PM Tyler Bishop <
tyler.bis...@beyondhosting.net> wrote:

> I bumped another post from earlier in the year.  I got this reply:
>
>
> Adam Tygart 
> 11:06 PM (8 minutes ago)
> to me, Kyle, Ceph
> This issue was related to using Jemalloc. Jemalloc is not as well
> tested with Bluestore and lead to lots of segfaults. We moved back to
> the default of tcmalloc with Bluestore and these stopped.
>
> Check /etc/sysconfig/ceph under RHEL based distros.
>
> ---
>
> I had enabled jemalloc in the sysconfig previously. Disabled that and now
> appear to have stable OSDs.
>
>
> On Mon, Aug 27, 2018 at 11:13 PM Alfredo Daniel Rezinovsky <
> alfredo.rezinov...@ingenieria.uncuyo.edu.ar> wrote:
>
>> Have you created the blockdb partitions or LVM manually ?
>>
>> What size?
>> On 27/08/18 23:48, Tyler Bishop wrote:
>>
>> My host has 256GB of ram.  62GB used under most heavy io workload.
>> _
>>
>> *Tyler Bishop*
>> EST 2007
>>
>>
>> O: 513-299-7108 x1000
>> M: 513-646-5809
>> http://BeyondHosting.net 
>>
>>
>> This email is intended only for the recipient(s) above and/or
>> otherwise authorized personnel. The information contained herein and
>> attached is confidential and the property of Beyond Hosting. Any
>> unauthorized copying, forwarding, printing, and/or disclosing
>> any information related to this email is prohibited. If you received this
>> message in error, please contact the sender and destroy all copies of this
>> email and any attachment(s).
>>
>>
>> On Mon, Aug 27, 2018 at 10:36 PM Alfredo Daniel Rezinovsky <
>> alfredo.rezinov...@ingenieria.uncuyo.edu.ar> wrote:
>>
>>> I had blockdb in ssd, with 3 OSDs per host (8G ram) and the default 3G
>>> bluestore_cache_size_ssd
>>>
>>> I stopped having inconsistencies dropping the cache to 1G.
>>>
>>> On 27/08/18 23:32, Tyler Bishop wrote:
>>>
>>> Having a constant segfault issue under io load with my newly created
>>> bluestore deployment.
>>>
>>> https://pastebin.com/82YjXRm7
>>>
>>> Setup is 28GB SSD LVM for block.db and 6T spinner for data.
>>>
>>> Config:
>>> [global]
>>> fsid =  REDACTED
>>> mon_initial_members = cephmon-1001, cephmon-1002, cephmon-1003
>>> mon_host = 10.20.142.5,10.20.142.6,10.20.142.7
>>> auth_cluster_required = cephx
>>> auth_service_required = cephx
>>> auth_client_required = cephx
>>> filestore_xattr_use_omap = true
>>>
>>> # Fixes issue where image is created with newer than supported features
>>> enabled.
>>> rbd_default_features = 3
>>>
>>>
>>> # Debug Tuning
>>> debug_lockdep = 0/0
>>> debug_context = 0/0
>>> debug_crush = 0/0
>>> debug_buffer = 0/0
>>> debug_timer = 0/0
>>> debug_filer = 0/0
>>> debug_objecter = 0/0
>>> debug_rados = 0/0
>>> debug_rbd = 0/0
>>> debug_journaler = 0/0
>>> debug_objectcatcher = 0/0
>>> debug_client = 0/0
>>> debug_osd = 0/0
>>> debug_optracker = 0/0
>>> debug_objclass = 0/0
>>> debug_filestore = 0/0
>>> debug_journal = 0/0
>>> debug_ms = 0/0
>>> debug_monc = 0/0
>>> debug_tp = 0/0
>>> debug_auth = 0/0
>>> debug_finisher = 0/0
>>> debug_heartbeatmap = 0/0
>>> debug_perfcounter = 0/0
>>> debug_asok = 0/0
>>> debug_throttle = 0/0
>>> debug_mon = 0/0
>>> debug_paxos = 0/0
>>> debug_rgw = 0/0
>>>
>>> [osd]
>>> osd_mkfs_type = xfs
>>> osd_mount_options_xfs =
>>> rw,noatime,,nodiratime,inode64,logbsize=256k,delaylog
>>> osd_mkfs_options_xfs = -f -i size=2048
>>> osd_journal_size = 10240
>>> filestore_queue_max_ops=1000
>>> filestore_queue_max_bytes = 1048576000
>>> filestore_max_sync_interval = 10
>>> filestore_merge_threshold = 500
>>> filestore_split_multiple = 100
>>> osd_op_shard_threads = 6
>>> journal_max_write_entries = 5000
>>> journal_max_write_bytes = 1048576000
>>> journal_queueu_max_ops = 3000
>>> journal_queue_max_bytes = 1048576000
>>> ms_dispatch_throttle_bytes = 1048576000
>>> objecter_inflight_op_bytes = 1048576000
>>> public network = 10.20.142.0/24
>>> cluster_network = 10.20.136.0/24
>>> osd_disk_thread_ioprio_priority = 7
>>> osd_disk_thread_ioprio_class = idle
>>> osd_max_backfills = 2
>>> osd_recovery_sleep = 0.10
>>>
>>>
>>> [client]
>>> rbd_cache = False
>>> rbd cache size = 33554432
>>> rbd cache target dirty

[ceph-users] New Ceph community manager: Mike Perez

[Sage Weil]

Hi everyone,

Please help me welcome Mike Perez, the new Ceph community manager!

Mike has a long history with Ceph: he started at DreamHost working on 
OpenStack and Ceph back in the early days, including work on the original 
RBD integration.  He went on to work in several roles in the OpenStack 
project, doing a mix of infrastructure, cross-project and community 
related initiatives, including serving as the Project Technical Lead for 
Cinder.

Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com, on 
IRC as thingee, or twitter as @thingee.

I am very excited to welcome Mike back to Ceph, and look forward to 
working together on building the Ceph developer and user communities!

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph community manager: Mike Perez

[Dan Mick]

On 08/28/2018 06:13 PM, Sage Weil wrote:
> Hi everyone,
> 
> Please help me welcome Mike Perez, the new Ceph community manager!
> 
> Mike has a long history with Ceph: he started at DreamHost working on 
> OpenStack and Ceph back in the early days, including work on the original 
> RBD integration.  He went on to work in several roles in the OpenStack 
> project, doing a mix of infrastructure, cross-project and community 
> related initiatives, including serving as the Project Technical Lead for 
> Cinder.
> 
> Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com, on 
> IRC as thingee, or twitter as @thingee.
> 
> I am very excited to welcome Mike back to Ceph, and look forward to 
> working together on building the Ceph developer and user communities!
> 
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

Welcome back Mike!

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph community manager: Mike Perez

[Erik McCormick]

Wherever I go, there you are ;). Glad to have you back again!

Cheers,
Erik

On Tue, Aug 28, 2018, 10:25 PM Dan Mick  wrote:

> On 08/28/2018 06:13 PM, Sage Weil wrote:
> > Hi everyone,
> >
> > Please help me welcome Mike Perez, the new Ceph community manager!
> >
> > Mike has a long history with Ceph: he started at DreamHost working on
> > OpenStack and Ceph back in the early days, including work on the
> original
> > RBD integration.  He went on to work in several roles in the OpenStack
> > project, doing a mix of infrastructure, cross-project and community
> > related initiatives, including serving as the Project Technical Lead for
> > Cinder.
> >
> > Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com, on
> > IRC as thingee, or twitter as @thingee.
> >
> > I am very excited to welcome Mike back to Ceph, and look forward to
> > working together on building the Ceph developer and user communities!
> >
> > sage
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> Welcome back Mike!
>
> --
> Dan Mick
> Red Hat, Inc.
> Ceph docs: http://ceph.com/docs
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph community manager: Mike Perez

[linghucongsong]

Welcome!




At 2018-08-29 09:13:24, "Sage Weil"  wrote:
>Hi everyone,
>
>Please help me welcome Mike Perez, the new Ceph community manager!
>
>Mike has a long history with Ceph: he started at DreamHost working on 
>OpenStack and Ceph back in the early days, including work on the original 
>RBD integration.  He went on to work in several roles in the OpenStack 
>project, doing a mix of infrastructure, cross-project and community 
>related initiatives, including serving as the Project Technical Lead for 
>Cinder.
>
>Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com, on 
>IRC as thingee, or twitter as @thingee.
>
>I am very excited to welcome Mike back to Ceph, and look forward to 
>working together on building the Ceph developer and user communities!
>
>sage
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph community manager: Mike Perez

[Alvaro Soto]

Welcome Mike!

On Tue, Aug 28, 2018 at 10:19 PM, linghucongsong 
wrote:

>
>
>
>
> Welcome!
>
>
>
> At 2018-08-29 09:13:24, "Sage Weil"  wrote:
> >Hi everyone,
> >
> >Please help me welcome Mike Perez, the new Ceph community manager!
> >
> >Mike has a long history with Ceph: he started at DreamHost working on
> >OpenStack and Ceph back in the early days, including work on the original
> >RBD integration.  He went on to work in several roles in the OpenStack
> >project, doing a mix of infrastructure, cross-project and community
> >related initiatives, including serving as the Project Technical Lead for
> >Cinder.
> >
> >Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com, on
> >IRC as thingee, or twitter as @thingee.
> >
> >I am very excited to welcome Mike back to Ceph, and look forward to
> >working together on building the Ceph developer and user communities!
> >
> >sage
> >___
> >ceph-users mailing list
> >ceph-users@lists.ceph.com
> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 

ATTE. Alvaro Soto Escobar

--
Great people talk about ideas,
average people talk about things,
small people talk ... about other people.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph community manager: Mike Perez

[ceph]

Great! Welcome Mike! 

Am 29. August 2018 05:36:20 MESZ schrieb Alvaro Soto :
>Welcome Mike!
>
>On Tue, Aug 28, 2018 at 10:19 PM, linghucongsong
>
>wrote:
>
>>
>>
>>
>>
>> Welcome!
>>
>>
>>
>> At 2018-08-29 09:13:24, "Sage Weil"  wrote:
>> >Hi everyone,
>> >
>> >Please help me welcome Mike Perez, the new Ceph community manager!
>> >
>> >Mike has a long history with Ceph: he started at DreamHost working
>on
>> >OpenStack and Ceph back in the early days, including work on the
>original
>> >RBD integration.  He went on to work in several roles in the
>OpenStack
>> >project, doing a mix of infrastructure, cross-project and community
>> >related initiatives, including serving as the Project Technical Lead
>for
>> >Cinder.
>> >
>> >Mike lives in Pasadena, CA, and can be reached at mpe...@redhat.com,
>on
>> >IRC as thingee, or twitter as @thingee.
>> >
>> >I am very excited to welcome Mike back to Ceph, and look forward to
>> >working together on building the Ceph developer and user
>communities!
>> >
>> >sage
>> >___
>> >ceph-users mailing list
>> >ceph-users@lists.ceph.com
>> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
>-- 
>
>ATTE. Alvaro Soto Escobar
>
>--
>Great people talk about ideas,
>average people talk about things,
>small people talk ... about other people.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to mount NFS-Ganesha-ressource via Proxmox-NFS-Plugin?

[Naumann, Thomas]

Hi,

because there aren´t any replies in proxmox-mailing-lists, i´ll give it
a try here - has anyone expierence about the following circumstance?
Any hints are welcome...:

for backup testing purposes we run a ceph-cluster with radosgw (S3) and
nfs-ganesha to export s3 via nfs. cluster is running on ubuntu 16.04.
package details:
nfs-ganesha  2.6.2-0-gb9685b8-xenial
nfs-ganesha-rgw:amd642.6.2-0-gb9685b8-1xenial
radosgw  12.2.7-1xenial
ceph 12.2.7-1xenial

detail of "ganesha.conf":
EXPORT
{
 Export_ID=100;
 Path = "/";
 Pseudo = /;
 Access_Type = RW;
 NFS_Protocols = 3,4;
 Transport_Protocols = UDP,TCP;
 Sectype = sys;
 Squash = No_Root_Squash;

 FSAL {
 Name = RGW;
 User_Id = "test";
 Access_Key_Id ="test";
 Secret_Access_Key = "test";
 }
}

RGW {
 ceph_conf = "/etc/ceph/ceph.conf";
 name = "client.rgw.blackhole-eh";
 cluster = "ceph";
}

LOG {
 Facility {
 name = FILE;
 destination = "/var/log/ganesha/ganesha.log";
 enable = active;
 }
}

detail of "storage.cfg":
nfs: test
path /mnt/pve/ganesha/
server x.x.x.x
export /
options vers=3,soft
content backup
maxfiles 10

result in "journalctl":
...
Aug 27 13:14:34 tr-25-3 pmxcfs[15192]: [status] notice: received log
Aug 27 13:14:48 tr-25-3 pvestatd[15777]: file /etc/pve/storage.cfg
line 
79 (section 'test') - unable to parse value of 'export': invalid
format 
- value does not look like a valid absolute path
Aug 27 13:14:48 tr-25-3 pvestatd[15777]: file /etc/pve/storage.cfg
line 
82 (skip section 'test'): missing value for required option 'export'
...

mounts via cli (mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp
x.x.x.x:/ /mnt/ganesha/) are working without issues - thats fine.

but how to mount this nfs-ressource via proxmox-nfs-plugin?

"pvesm nfsscan x.x.x.x" ends without an result.
-- 
Thomas Naumann

Otto-von-Guericke Universität Magdeburg
Universitätsrechenzentrum
Universitätsplatz 2
39106 Magdeburg

fon:   +49 391 67-58563
email: thomas.naum...@ovgu.de
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Installing ceph 12.2.4 via Ubuntu apt

[Thomas Bennett]

Hi David,

Thanks for your reply. That's how I'm currently handling it.

Kind regards,
Tom

On Tue, Aug 28, 2018 at 4:36 PM David Turner  wrote:

> That is the expected behavior of the ceph repo. In the past when I needed
> a specific version I would download the packages for the version to a
> folder and you can create a repo file that reads from a local directory.
> That's how I would re-install my test lab after testing an upgrade
> procedure to try it over again.
>
> On Tue, Aug 28, 2018, 1:01 AM Thomas Bennett  wrote:
>
>> Hi,
>>
>> I'm wanting to pin to an older version of Ceph Luminous (12.2.4) and I've
>> noticed that https://download.ceph.com/debian-luminous/ does not support
>> this via apt install:
>> apt install ceph works for 12.2.7 but
>> apt install ceph=12.2.4-1xenial does not work
>>
>> The deb file are there, they're just not included in the package
>> distribution. Is this the desired behaviour or a misconfiguration?
>>
>> Cheers,
>> Tom
>>
>> --
>> Thomas Bennett
>>
>> SARAO
>> Science Data Processing
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>

-- 
Thomas Bennett

SARAO
Science Data Processing
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com