[ceph-users] How to really change public network in ceph

[Mario Giammarco]

Hello,
I have a test proxmox/ceph cluster with four servers.
I need to change the ceph public subnet from 10.1.0.0/24 to 10.1.5.0/24.
I have read documentation and tutorials.
The most critical part seems monitor map editing.
But it seems to me that osds need to bind to new subnet too.
I tried to put 10.1.0 and 10.1.5 subnets to public but it seems it changes
nothing.
Infact official documentation is unclear: it says you can put in public
network more than one subnet. It says they must be routed. But it does not
say what happens when you use multiple subnets or why you should do it.

So I need help on these questions:
- exact sequence of operations to change public network in ceph (not only
monitors, also osds)
- details on multiple subnets in public networks in ceph

Thanks in advance for any help,
Mario
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to ceph 12.2.2, libec_jerasure.so: undefined symbol: _ZN4ceph6buffer3ptrC1ERKS1_

[Sebastian Koch - ilexius GmbH]

If anyone else faces this problem in the future: I fixed it now by
running "apt install --reinstall ceph-osd". Don't know why the normal
apt upgrade broke it.


On 18.02.2018 22:35, Sebastian Koch - ilexius GmbH wrote:
> Hello,
>
> I ran "apt upgrade" on Ubuntu 16.04 on one node, now the two OSDs on the
> node are not starting any more.
>
> From the apt log it looks like many Ceph packages were upgraded from
> 12.2.1-1xenial to 12.2.2-1xenial.
>
> In both OSD logs I can see the following:
>
> 2018-02-18 22:24:14.424425 7f5f96a18e00  0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2018-02-18 22:24:14.424443 7f5f96a18e00  0 ceph version 12.2.2
> (cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process
> (unknown), pid 6272
> 2018-02-18 22:24:14.427532 7f5f96a18e00 -1 Public network was set, but
> cluster network was not set
> 2018-02-18 22:24:14.427538 7f5f96a18e00 -1 Using public network also
> for cluster network
> 2018-02-18 22:24:14.430271 7f5f96a18e00  0 pidfile_write: ignore empty
> --pid-file
> 2018-02-18 22:24:14.431859 7f5f96a18e00 -1 load
> dlopen(/usr/lib/ceph/erasure-code/libec_jerasure.so):
> /usr/lib/ceph/erasure-code/libec_jerasure.so: undefined symbol:
> _ZN4ceph6buffer3ptrC1ERKS1_
>
> Any advice how to get the node running again? Thank you very much!!!
>
> Best regards,
> Sebastian
>

-- 
Dipl.-Wirt.-Inf. Sebastian Koch
Geschäftsführer


ilexius GmbH

Unter den Eichen 5
Haus i
65195 Wiesbaden
E-Mail s.k...@ilexius.de
Tel +49-611 - 1 80 33 49
Fax +49-611 - 236 80 84 29
Web http://www.ilexius.de

vertreten durch die Geschäftsleitung:
Sebastian Koch und Thomas Schlüter
Registergericht: Wiesbaden
Handelsregister: HRB 21723
Steuernummer: 040 236 22640
Ust-IdNr.: DE240822836
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Bluestore performance question

[Caspar Smit]

"I checked and the OSD-hosts peaked at a load average of about 22 (they
have 24+24HT cores) in our dd benchmark,
but stayed well below that (only about 20 % per OSD daemon) in the rados
bench test."

Maybe because your dd test uses bs=1M and rados bench is using 4M as
default block size?
Caspar

2018-02-18 16:03 GMT+01:00 Oliver Freyermuth 
:

> Dear Cephalopodians,
>
> we are just getting started with our first Ceph cluster (Luminous 12.2.2)
> and doing some basic benchmarking.
>
> We have two pools:
> - cephfs_metadata, living on 4 SSD devices (each is a bluestore OSD, 240
> GB) on 2 hosts (i.e. 2 SSDs each), setup as:
>   - replicated, min size 2, max size 4
>   - 128 PGs
> - cephfs_data, living on 6 hosts each of which has the following setup:
>   - 32 HDD drives (4 TB) each of which is a bluestore OSD, the LSI
> controller to which they are attached is in JBOD personality
>   - 2 SSD drives, each has 16 partitions with 7 GB per partition, used as
> block-db by the bluestore OSDs living on the HDDs.
>   - Created with:
> ceph osd erasure-code-profile set cephfs_data k=4 m=2
> crush-device-class=hdd crush-failure-domain=host
> ceph osd pool create cephfs_data 2048 2048 erasure cephfs_data
>   - So to summarize: 192 OSDs, 2048 PGs, each OSD has 4 TB data + 7 GB
> block-db
>
> The interconnect (public and cluster network)
> is made via IP over Infiniband (56 GBit bandwidth), using the software
> stack that comes with CentOS 7.
>
> This leaves us with the possibility that one of the metadata-hosts can
> fail, and still one of the disks can fail.
> For the data hosts, up to two machines total can fail.
>
> We have 40 clients connected to this cluster. We now run something like:
> dd if=/dev/zero of=some_file bs=1M count=1
> on each CPU core of each of the clients, yielding a total of 1120 writing
> processes (all 40 clients have 28+28HT cores),
> using the ceph-fuse client.
>
> This yields a write throughput of a bit below 1 GB/s (capital B), which is
> unexpectedly low.
> Running a BeeGFS on the same cluster before (disks were in RAID 6 in that
> case) yielded throughputs of about 12 GB/s,
> but came with other issues (e.g. it's not FOSS...), so we'd love to run
> Ceph :-).
>
> I performed some basic tests to try to understand the bottleneck for Ceph:
> # rados bench -p cephfs_data 10 write --no-cleanup -t 40
> Bandwidth (MB/sec): 695.952
> Stddev Bandwidth:   295.223
> Max bandwidth (MB/sec): 1088
> Min bandwidth (MB/sec): 76
> Average IOPS:   173
> Stddev IOPS:73
> Max IOPS:   272
> Min IOPS:   19
> Average Latency(s): 0.220967
> Stddev Latency(s):  0.305967
> Max latency(s): 2.88931
> Min latency(s): 0.0741061
>
> => This agrees mostly with our basic dd benchmark.
>
> Reading is a bit faster:
> # rados bench -p cephfs_data 10 rand
> => Bandwidth (MB/sec):   1108.75
>
> However, the disks are reasonably quick:
> # ceph tell osd.0 bench
> {
> "bytes_written": 1073741824,
> "blocksize": 4194304,
> "bytes_per_sec": 331850403
> }
>
> I checked and the OSD-hosts peaked at a load average of about 22 (they
> have 24+24HT cores) in our dd benchmark,
> but stayed well below that (only about 20 % per OSD daemon) in the rados
> bench test.
> One idea would be to switch from jerasure to ISA, since the machines are
> all Intel CPUs only anyways.
>
> Already tried:
> - TCP stack tuning (wmem, rmem), no huge effect.
> - changing the block sizes used by dd, no effect.
> - Testing network throughput with ib_write_bw, this revealed something
> like:
>  #bytes #iterationsBW peak[MB/sec]BW average[MB/sec]
>  MsgRate[Mpps]
>  2  5000 19.73  19.30
> 10.118121
>  4  5000 52.79  51.70
> 13.553412
>  8  5000 101.23 96.65
> 12.668371
>  16 5000 243.66 233.42
>  15.297583
>  32 5000 350.66 344.73
>  11.296089
>  64 5000 909.14 324.85 5.322323
>  1285000 1424.841401.29
> 11.479374
>  2565000 2865.242801.04
> 11.473055
>  5125000 5169.985095.08
> 10.434733
>  1024   5000 10022.759791.42
>  10.026410
>  2048   5000 10988.6410628.83
> 5.441958
>  4096   5000 11401.4011399.14
> 2.918180
> [...]
>
> So it seems the IP-over-Infiniband is not the bottleneck (BeeGFS was using
> RDMA).
> Other ideas that come to mind:
> - Testing with Ceph-RDMA, but that does not seem production-ready yet, if
> I read the list correctly.
> - Increasing osd_pool_erasure_code_stripe_width.
> - Using ISA as EC plugin.
> - Reducing the bluestore_cache_size_hdd, it seems when recovery +
> benchmark is ongoing, swap is used (but not when performing benchmarking
> only,
>  

Re: [ceph-users] mon service failed to start

[Caspar Smit]

Hi Behnam,

I would firstly recommend running a filesystem check on the monitor disk
first to see if there are any inconsistencies.

Is the disk where the monitor is running on a spinning disk or SSD?

If SSD you should check the Wear level stats through smartctl.
Maybe trim (discard) enabled on the filesystem mount? (discard could cause
problems/corruption in combination with certain SSD firmwares)

Caspar

2018-02-16 23:03 GMT+01:00 Behnam Loghmani :

> I checked the disk that monitor is on it with smartctl and it didn't
> return any error and it doesn't have any Current_Pending_Sector.
> Do you recommend any disk checks to make sure that this disk has problem
> and then I can send the report to the provider for replacing the disk
>
> On Sat, Feb 17, 2018 at 1:09 AM, Gregory Farnum 
> wrote:
>
>> The disk that the monitor is on...there isn't anything for you to
>> configure about a monitor WAL though so I'm not sure how that enters into
>> it?
>>
>> On Fri, Feb 16, 2018 at 12:46 PM Behnam Loghmani <
>> behnam.loghm...@gmail.com> wrote:
>>
>>> Thanks for your reply
>>>
>>> Do you mean, that's the problem with the disk I use for WAL and DB?
>>>
>>> On Fri, Feb 16, 2018 at 11:33 PM, Gregory Farnum 
>>> wrote:
>>>

 On Fri, Feb 16, 2018 at 7:37 AM Behnam Loghmani <
 behnam.loghm...@gmail.com> wrote:

> Hi there,
>
> I have a Ceph cluster version 12.2.2 on CentOS 7.
>
> It is a testing cluster and I have set it up 2 weeks ago.
> after some days, I see that one of the three mons has stopped(out of
> quorum) and I can't start it anymore.
> I checked the mon service log and the output shows this error:
>
> """
> mon.XX@-1(probing) e4 preinit clean up potentially inconsistent
> store state
> rocksdb: submit_transaction_sync error: Corruption: block checksum
> mismatch
>

 This bit is the important one. Your disk is bad and it’s feeding back
 corrupted data.




> code = 2 Rocksdb transaction:
>  0> 2018-02-16 17:37:07.041812 7f45a1e52e40 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/
> AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/
> MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUI
> LD/ceph-12.2.2/src/mon/MonitorDBStore.h: In function 'void
> MonitorDBStore::clear(std::set >&)' thread
> 7f45a1e52e40 time 2018-02-16 17:37:07.040846
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/
> AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/
> MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/mon/MonitorDBStore.h:
> 581: FAILE
> D assert(r >= 0)
> """
>
> the only solution I found is to remove this mon from quorum and remove
> all mon data and re-add this mon to quorum again.
> and ceph goes to the healthy status again.
>
> but now after some days this mon has stopped and I face the same
> problem again.
>
> My cluster setup is:
> 4 osd hosts
> total 8 osds
> 3 mons
> 1 rgw
>
> this cluster has setup with ceph-volume lvm and wal/db separation on
> logical volumes.
>
> Best regards,
> Behnam Loghmani
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

>>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Missing clones

[Karsten Becker]

Hi,

I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
fix this?

> 2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9 
> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected clone 
> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1 missing
> 2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9 
> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1 missing 
> clone(s)
> 2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible data 
> damage: 1 pg inconsistent (PG_DAMAGED)
> 2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0 fixed
> 2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub errors; 
> Possible data damage: 1 pg inconsistent

"ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
started scrub "ceph pg scrub 10.7b9" also.

Best from Berlin/Germany
Karsten


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), 
HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Eugen Block]

Hi,

just to rule out the obvious, which size does the pool have? You  
aren't running it with size = 2, do you?



Zitat von Karsten Becker :


Hi,

I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
fix this?

2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9  
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head  
expected clone  
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1 missing
2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9  
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1  
missing clone(s)
2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update:  
Possible data damage: 1 pg inconsistent (PG_DAMAGED)

2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0 fixed
2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub  
errors; Possible data damage: 1 pg inconsistent


"ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
started scrub "ceph pg scrub 10.7b9" also.

Best from Berlin/Germany
Karsten


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin  
(Charlottenburg), HRB 57947

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Karsten Becker]

Hi.

We have size=3 min_size=2.

But this "upgrade" has been done during the weekend. We had size=2
min_size=1 before.

Best
Karsten



On 19.02.2018 13:02, Eugen Block wrote:
> Hi,
> 
> just to rule out the obvious, which size does the pool have? You aren't
> running it with size = 2, do you?
> 
> 
> Zitat von Karsten Becker :
> 
>> Hi,
>>
>> I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
>> fix this?
>>
>>> 2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
>>> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
>>> clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
>>> missing
>>> 2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9
>>> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1
>>> missing clone(s)
>>> 2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible
>>> data damage: 1 pg inconsistent (PG_DAMAGED)
>>> 2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0 fixed
>>> 2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub
>>> errors; Possible data damage: 1 pg inconsistent
>>
>> "ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
>> started scrub "ceph pg scrub 10.7b9" also.
>>
>> Best from Berlin/Germany
>> Karsten
>>
>>
>> Ecologic Institut gemeinnuetzige GmbH
>> Pfalzburger Str. 43/44, D-10717 Berlin
>> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
>> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
>> Registergericht / Court of Registration: Amtsgericht Berlin
>> (Charlottenburg), HRB 57947
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), 
HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] "Cannot get stat of OSD" in ceph.mgr.log upon enabling influx plugin

[knawnd]

Dear Ceph users,

I am trying to enable influx plugin for ceph following http://docs.ceph.com/docs/master/mgr/influx/ 
but no data comes to influxdb DB. As soon as 'ceph mgr module enable influx' command is executed on 
one of ceph mgr node (running on CentOS 7.4.1708) there are the following messages in 
/var/log/ceph/ceph-mgr..log:


2018-02-19 17:11:05.947122 7f33c9b43600  0 ceph version 12.2.2 
(cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 96425

2018-02-19 17:11:05.947737 7f33c9b43600  0 pidfile_write: ignore empty 
--pid-file
2018-02-19 17:11:05.986676 7f33c9b43600  1 mgr send_beacon standby
2018-02-19 17:11:06.003029 7f33c0e2a700  1 mgr init Loading python module 
'balancer'
2018-02-19 17:11:06.031293 7f33c0e2a700  1 mgr init Loading python module 
'dashboard'
2018-02-19 17:11:06.119328 7f33c0e2a700  1 mgr init Loading python module 
'influx'
2018-02-19 17:11:06.220394 7f33c0e2a700  1 mgr init Loading python module 
'restful'
2018-02-19 17:11:06.398380 7f33c0e2a700  1 mgr init Loading python module 
'status'
2018-02-19 17:11:06.919109 7f33c0e2a700  1 mgr handle_mgr_map Activating!
2018-02-19 17:11:06.919454 7f33c0e2a700  1 mgr handle_mgr_map I am now 
activating
2018-02-19 17:11:06.952174 7f33a58ed700  1 mgr load Constructed class from 
module: balancer
2018-02-19 17:11:06.953259 7f33a58ed700  1 mgr load Constructed class from 
module: dashboard
2018-02-19 17:11:06.953959 7f33a58ed700  1 mgr load Constructed class from 
module: influx
2018-02-19 17:11:06.954193 7f33a58ed700  1 mgr load Constructed class from 
module: restful
2018-02-19 17:11:06.955549 7f33a58ed700  1 mgr load Constructed class from 
module: status
2018-02-19 17:11:06.955613 7f33a58ed700  1 mgr send_beacon active
2018-02-19 17:11:06.960224 7f33a58ed700  1 mgr[restful] Unknown request ''
2018-02-19 17:11:06.961912 7f33a28e7700  1 mgr[restful] server not running: no 
certificate configured
2018-02-19 17:11:06.969027 7f33a30e8700  0 Cannot get stat of OSD 0
 on so on for all 64 OSD I have in a cluster.

'ceph osd tree' shows all OSD are up. 'ceph health' gives HEALTH_OK.

python-influxdb-5.0.0-2.el7.noarch is installed on ceph mgr node. That rpm was 
rebuilt from fc28 srpm.

'ceph config-key dump|grep influx' shows reasonable info:
"mgr/influx/database": "ceph_stats",
"mgr/influx/hostname": "",
"mgr/influx/password": "",
"mgr/influx/ssl": "false",
"mgr/influx/username": "cephstat",
"mgr/influx/verify_ssl": "false"


influxdb-1.4.2-1.x86_64 is installed on influxdb host with CentOS 7.4.1708.

I would appreciate any help on that issue.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "Cannot get stat of OSD" in ceph.mgr.log upon enabling influx plugin

[knawnd]

Forgot to mentioned that influx self-test produces a reasonable output too (long json list with some 
metrics and timestamps) as well as there are the following lines in mgr log:


2018-02-19 17:35:04.208858 7f33a50ec700  1 mgr.server reply handle_command (0) 
Success
2018-02-19 17:35:04.245285 7f33a50ec700  0 log_channel(audit) log [DBG] : from='client.344950 
:0/3773014505' entity='client.admin' cmd=[{"prefix": "influx self-test"}]: dispatch

2018-02-19 17:35:04.245314 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer status'
2018-02-19 17:35:04.245319 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer mode'
2018-02-19 17:35:04.245323 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer on'
2018-02-19 17:35:04.245327 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer off'
2018-02-19 17:35:04.245331 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer eval'
2018-02-19 17:35:04.245335 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer eval-verbose'
2018-02-19 17:35:04.245339 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer optimize'
2018-02-19 17:35:04.245343 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer show'
2018-02-19 17:35:04.245347 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer rm'
2018-02-19 17:35:04.245351 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer reset'
2018-02-19 17:35:04.245354 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer dump'
2018-02-19 17:35:04.245358 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'balancer execute'
2018-02-19 17:35:04.245363 7f33a50ec700  1 mgr.server handle_command 
pyc_prefix: 'influx self-test'
2018-02-19 17:35:04.402782 7f33a58ed700  1 mgr.server reply handle_command (0) 
Success Self-test OK

kna...@gmail.com wrote on 19/02/18 17:27:

Dear Ceph users,

I am trying to enable influx plugin for ceph following http://docs.ceph.com/docs/master/mgr/influx/ 
but no data comes to influxdb DB. As soon as 'ceph mgr module enable influx' command is executed on 
one of ceph mgr node (running on CentOS 7.4.1708) there are the following messages in 
/var/log/ceph/ceph-mgr..log:


2018-02-19 17:11:05.947122 7f33c9b43600  0 ceph version 12.2.2 
(cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 96425

2018-02-19 17:11:05.947737 7f33c9b43600  0 pidfile_write: ignore empty 
--pid-file
2018-02-19 17:11:05.986676 7f33c9b43600  1 mgr send_beacon standby
2018-02-19 17:11:06.003029 7f33c0e2a700  1 mgr init Loading python module 
'balancer'
2018-02-19 17:11:06.031293 7f33c0e2a700  1 mgr init Loading python module 
'dashboard'
2018-02-19 17:11:06.119328 7f33c0e2a700  1 mgr init Loading python module 
'influx'
2018-02-19 17:11:06.220394 7f33c0e2a700  1 mgr init Loading python module 
'restful'
2018-02-19 17:11:06.398380 7f33c0e2a700  1 mgr init Loading python module 
'status'
2018-02-19 17:11:06.919109 7f33c0e2a700  1 mgr handle_mgr_map Activating!
2018-02-19 17:11:06.919454 7f33c0e2a700  1 mgr handle_mgr_map I am now 
activating
2018-02-19 17:11:06.952174 7f33a58ed700  1 mgr load Constructed class from 
module: balancer
2018-02-19 17:11:06.953259 7f33a58ed700  1 mgr load Constructed class from 
module: dashboard
2018-02-19 17:11:06.953959 7f33a58ed700  1 mgr load Constructed class from 
module: influx
2018-02-19 17:11:06.954193 7f33a58ed700  1 mgr load Constructed class from 
module: restful
2018-02-19 17:11:06.955549 7f33a58ed700  1 mgr load Constructed class from 
module: status
2018-02-19 17:11:06.955613 7f33a58ed700  1 mgr send_beacon active
2018-02-19 17:11:06.960224 7f33a58ed700  1 mgr[restful] Unknown request ''
2018-02-19 17:11:06.961912 7f33a28e7700  1 mgr[restful] server not running: no 
certificate configured
2018-02-19 17:11:06.969027 7f33a30e8700  0 Cannot get stat of OSD 0
 on so on for all 64 OSD I have in a cluster.

'ceph osd tree' shows all OSD are up. 'ceph health' gives HEALTH_OK.

python-influxdb-5.0.0-2.el7.noarch is installed on ceph mgr node. That rpm was 
rebuilt from fc28 srpm.

'ceph config-key dump|grep influx' shows reasonable info:
     "mgr/influx/database": "ceph_stats",
     "mgr/influx/hostname": "",
     "mgr/influx/password": "",
     "mgr/influx/ssl": "false",
     "mgr/influx/username": "cephstat",
     "mgr/influx/verify_ssl": "false"


influxdb-1.4.2-1.x86_64 is installed on influxdb host with CentOS 7.4.1708.

I would appreciate any help on that issue.




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "Cannot get stat of OSD" in ceph.mgr.log upon enabling influx plugin

[Benjeman Meekhof]

The 'cannot stat' messages are normal at startup, we see them also in
our working setup with mgr influx module.  Maybe they could be fixed
by delaying the module startup,  or having it check for some other
'all good' status but I haven't looked into it.  You should only be
seeing them when the mgr initially loads.

As far as not getting data, if the self-test works and outputs metrics
then the module is reading metrics ok from the mgr.  A few things you
could try:

- Check that the user you setup has rights to the destination
database, or admin rights to create database if you did not create and
setup beforehand
- Increase mgr debug and see if anything is showing up:  ceph tell
mgr.* injectargs '--debug_mgr 20'(this will be a lot of logging,
be sure to reset to 1/5 default)
- Check that your influx server is getting the traffic:   ' tcpdump -i
eth1 port 8086 and src host.example '

thanks,
Ben

On Mon, Feb 19, 2018 at 9:36 AM,   wrote:
> Forgot to mentioned that influx self-test produces a reasonable output too
> (long json list with some metrics and timestamps) as well as there are the
> following lines in mgr log:
>
> 2018-02-19 17:35:04.208858 7f33a50ec700  1 mgr.server reply handle_command
> (0) Success
> 2018-02-19 17:35:04.245285 7f33a50ec700  0 log_channel(audit) log [DBG] :
> from='client.344950 :0/3773014505' entity='client.admin'
> cmd=[{"prefix": "influx self-test"}]: dispatch
> 2018-02-19 17:35:04.245314 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer status'
> 2018-02-19 17:35:04.245319 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer mode'
> 2018-02-19 17:35:04.245323 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer on'
> 2018-02-19 17:35:04.245327 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer off'
> 2018-02-19 17:35:04.245331 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer eval'
> 2018-02-19 17:35:04.245335 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer eval-verbose'
> 2018-02-19 17:35:04.245339 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer optimize'
> 2018-02-19 17:35:04.245343 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer show'
> 2018-02-19 17:35:04.245347 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer rm'
> 2018-02-19 17:35:04.245351 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer reset'
> 2018-02-19 17:35:04.245354 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer dump'
> 2018-02-19 17:35:04.245358 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'balancer execute'
> 2018-02-19 17:35:04.245363 7f33a50ec700  1 mgr.server handle_command
> pyc_prefix: 'influx self-test'
> 2018-02-19 17:35:04.402782 7f33a58ed700  1 mgr.server reply handle_command
> (0) Success Self-test OK
>
> kna...@gmail.com wrote on 19/02/18 17:27:
>
>> Dear Ceph users,
>>
>> I am trying to enable influx plugin for ceph following
>> http://docs.ceph.com/docs/master/mgr/influx/ but no data comes to influxdb
>> DB. As soon as 'ceph mgr module enable influx' command is executed on one of
>> ceph mgr node (running on CentOS 7.4.1708) there are the following messages
>> in /var/log/ceph/ceph-mgr..log:
>>
>> 2018-02-19 17:11:05.947122 7f33c9b43600  0 ceph version 12.2.2
>> (cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process
>> (unknown), pid 96425
>> 2018-02-19 17:11:05.947737 7f33c9b43600  0 pidfile_write: ignore empty
>> --pid-file
>> 2018-02-19 17:11:05.986676 7f33c9b43600  1 mgr send_beacon standby
>> 2018-02-19 17:11:06.003029 7f33c0e2a700  1 mgr init Loading python module
>> 'balancer'
>> 2018-02-19 17:11:06.031293 7f33c0e2a700  1 mgr init Loading python module
>> 'dashboard'
>> 2018-02-19 17:11:06.119328 7f33c0e2a700  1 mgr init Loading python module
>> 'influx'
>> 2018-02-19 17:11:06.220394 7f33c0e2a700  1 mgr init Loading python module
>> 'restful'
>> 2018-02-19 17:11:06.398380 7f33c0e2a700  1 mgr init Loading python module
>> 'status'
>> 2018-02-19 17:11:06.919109 7f33c0e2a700  1 mgr handle_mgr_map Activating!
>> 2018-02-19 17:11:06.919454 7f33c0e2a700  1 mgr handle_mgr_map I am now
>> activating
>> 2018-02-19 17:11:06.952174 7f33a58ed700  1 mgr load Constructed class from
>> module: balancer
>> 2018-02-19 17:11:06.953259 7f33a58ed700  1 mgr load Constructed class from
>> module: dashboard
>> 2018-02-19 17:11:06.953959 7f33a58ed700  1 mgr load Constructed class from
>> module: influx
>> 2018-02-19 17:11:06.954193 7f33a58ed700  1 mgr load Constructed class from
>> module: restful
>> 2018-02-19 17:11:06.955549 7f33a58ed700  1 mgr load Constructed class from
>> module: status
>> 2018-02-19 17:11:06.955613 7f33a58ed700  1 mgr send_beacon active
>> 2018-02-19 17:11:06.960224 7f33a58ed700  1 mgr[restful] Unknown request ''
>> 2018-02-19 17:11:06.961912 7f33a28e7700  1 mgr[restful] server not
>> running: no certificate configured
>> 2018-02-19 17:11:06.969027 7f33a30e8700  0 Cannot get stat of OSD 

Re: [ceph-users] "Cannot get stat of OSD" in ceph.mgr.log upon enabling influx plugin

[John Spray]

On Mon, Feb 19, 2018 at 3:07 PM, Benjeman Meekhof  wrote:
> The 'cannot stat' messages are normal at startup, we see them also in
> our working setup with mgr influx module.  Maybe they could be fixed
> by delaying the module startup,  or having it check for some other
> 'all good' status but I haven't looked into it.  You should only be
> seeing them when the mgr initially loads.

The log spam was also recently reported as happening when OSDs are
down (https://tracker.ceph.com/issues/23017).  This is annoying, I've
just written a patch for that bug.

John

>
> As far as not getting data, if the self-test works and outputs metrics
> then the module is reading metrics ok from the mgr.  A few things you
> could try:
>
> - Check that the user you setup has rights to the destination
> database, or admin rights to create database if you did not create and
> setup beforehand
> - Increase mgr debug and see if anything is showing up:  ceph tell
> mgr.* injectargs '--debug_mgr 20'(this will be a lot of logging,
> be sure to reset to 1/5 default)
> - Check that your influx server is getting the traffic:   ' tcpdump -i
> eth1 port 8086 and src host.example '
>
> thanks,
> Ben
>
> On Mon, Feb 19, 2018 at 9:36 AM,   wrote:
>> Forgot to mentioned that influx self-test produces a reasonable output too
>> (long json list with some metrics and timestamps) as well as there are the
>> following lines in mgr log:
>>
>> 2018-02-19 17:35:04.208858 7f33a50ec700  1 mgr.server reply handle_command
>> (0) Success
>> 2018-02-19 17:35:04.245285 7f33a50ec700  0 log_channel(audit) log [DBG] :
>> from='client.344950 :0/3773014505' entity='client.admin'
>> cmd=[{"prefix": "influx self-test"}]: dispatch
>> 2018-02-19 17:35:04.245314 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer status'
>> 2018-02-19 17:35:04.245319 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer mode'
>> 2018-02-19 17:35:04.245323 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer on'
>> 2018-02-19 17:35:04.245327 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer off'
>> 2018-02-19 17:35:04.245331 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer eval'
>> 2018-02-19 17:35:04.245335 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer eval-verbose'
>> 2018-02-19 17:35:04.245339 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer optimize'
>> 2018-02-19 17:35:04.245343 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer show'
>> 2018-02-19 17:35:04.245347 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer rm'
>> 2018-02-19 17:35:04.245351 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer reset'
>> 2018-02-19 17:35:04.245354 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer dump'
>> 2018-02-19 17:35:04.245358 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'balancer execute'
>> 2018-02-19 17:35:04.245363 7f33a50ec700  1 mgr.server handle_command
>> pyc_prefix: 'influx self-test'
>> 2018-02-19 17:35:04.402782 7f33a58ed700  1 mgr.server reply handle_command
>> (0) Success Self-test OK
>>
>> kna...@gmail.com wrote on 19/02/18 17:27:
>>
>>> Dear Ceph users,
>>>
>>> I am trying to enable influx plugin for ceph following
>>> http://docs.ceph.com/docs/master/mgr/influx/ but no data comes to influxdb
>>> DB. As soon as 'ceph mgr module enable influx' command is executed on one of
>>> ceph mgr node (running on CentOS 7.4.1708) there are the following messages
>>> in /var/log/ceph/ceph-mgr..log:
>>>
>>> 2018-02-19 17:11:05.947122 7f33c9b43600  0 ceph version 12.2.2
>>> (cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process
>>> (unknown), pid 96425
>>> 2018-02-19 17:11:05.947737 7f33c9b43600  0 pidfile_write: ignore empty
>>> --pid-file
>>> 2018-02-19 17:11:05.986676 7f33c9b43600  1 mgr send_beacon standby
>>> 2018-02-19 17:11:06.003029 7f33c0e2a700  1 mgr init Loading python module
>>> 'balancer'
>>> 2018-02-19 17:11:06.031293 7f33c0e2a700  1 mgr init Loading python module
>>> 'dashboard'
>>> 2018-02-19 17:11:06.119328 7f33c0e2a700  1 mgr init Loading python module
>>> 'influx'
>>> 2018-02-19 17:11:06.220394 7f33c0e2a700  1 mgr init Loading python module
>>> 'restful'
>>> 2018-02-19 17:11:06.398380 7f33c0e2a700  1 mgr init Loading python module
>>> 'status'
>>> 2018-02-19 17:11:06.919109 7f33c0e2a700  1 mgr handle_mgr_map Activating!
>>> 2018-02-19 17:11:06.919454 7f33c0e2a700  1 mgr handle_mgr_map I am now
>>> activating
>>> 2018-02-19 17:11:06.952174 7f33a58ed700  1 mgr load Constructed class from
>>> module: balancer
>>> 2018-02-19 17:11:06.953259 7f33a58ed700  1 mgr load Constructed class from
>>> module: dashboard
>>> 2018-02-19 17:11:06.953959 7f33a58ed700  1 mgr load Constructed class from
>>> module: influx
>>> 2018-02-19 17:11:06.954193 7f33a58ed700  1 mgr load Constructed class from
>>> module: restful
>>> 2018-02-19 17:11:06.955549 7f33a58ed700  1 mgr load Construct

Re: [ceph-users] puppet for the deployment of ceph

[Benjeman Meekhof]

We use this one, now heavily modified in our own fork.   I'd sooner
point you at the original unless it is missing something you need.
Ours has diverged a bit and makes no attempt to support anything
outside our specific environment (RHEL7).

https://github.com/openstack/puppet-ceph
https://github.com/MI-OSiRIS/puppet-ceph

In the near future the master branch of our fork will use ceph-volume
instead of ceph-disk to setup new OSD, those changes are on
'bluestore' branch now.

There are a few on the forge which I have no experience with:
https://forge.puppet.com/tags/ceph

thanks,
Ben

On Fri, Feb 16, 2018 at 8:11 AM, Александр Пивушков  wrote:
> Colleagues, tell me please, who uses puppet for the deployment of ceph in
> production?
>  And also, where can I get the puppet modules for ceph?
>
>
> Александр Пивушков
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating to new pools

[Eugen Block]

Hi,

I created a ticket for the rbd import issue:

https://tracker.ceph.com/issues/23038

Regards,
Eugen


Zitat von Jason Dillaman :


On Fri, Feb 16, 2018 at 11:20 AM, Eugen Block  wrote:

Hi Jason,


... also forgot to mention "rbd export --export-format 2" / "rbd
import --export-format 2" that will also deeply export/import all
snapshots associated with an image and that feature is available in
the Luminous release.



thanks for that information, this could be very valuable for us. I'll have
to test that intesively, but not before next week.

But a first quick test brought up a couple of issues which I'll have to
re-check before bringing them up here.

One issue is worth mentioning, though: After I exported (rbd export
--export-format ...) a glance image and imported it back to a different pool
(rbd import --export-format ...) its snapshot was copied, but not protected.
This prevented nova from cloning the base image and leaving that instance in
error state. Protecting the snapshot manually and launch another instance
enabled nova to clone the image successfully.

Could this be worth a bug report or is it rather something I did wrong or
missed?


Definitely deserves a bug tracker ticket opened. Thanks.


I wish you all a nice weekend!

Regards
Eugen


Zitat von Jason Dillaman :


On Fri, Feb 16, 2018 at 8:08 AM, Jason Dillaman 
wrote:


On Fri, Feb 16, 2018 at 5:36 AM, Jens-U. Mozdzen  wrote:


Dear list, hello Jason,

you may have seen my message on the Ceph mailing list about RDB pool
migration - it's a common subject that pools were created in a
sub-optimum
fashion and i. e. pgnum is (not yet) reducible, so we're looking into
means
to "clone" an RBD pool into a new pool within the same cluster
(including
snapshots).

We had looked into creating a tool for this job, but soon noticed that
we're
duplicating basic functionality of rbd-mirror. So we tested the
following,
which worked out nicely:

- create a test cluster (Ceph cluster plus an Openstack cluster using an
RBD
pool) and some Openstack instances

- create a second Ceph test cluster

- stop Openstack

- use rbd-mirror to clone the RBD pool from the first to the second Ceph
cluster (IOW aborting rbd-mirror once the initial coping was done)

- recreate the RDB pool on the first cluster

- use rbd-mirror to clone the mirrored pool back to the (newly created)
pool
on the first cluster

- start Openstack and work with the (recreated) pool on the first
cluster

So using rbd-mirror, we could clone an RBD pool's content to a
differently
structured pool on the same cluster - by using an intermediate cluster.

@Jason: Looking at the commit history for rbd-mirror, it seems you might
be
able to shed some light on this: Do you see an easy way to modify
rbd-mirror
in such a fashion that instead of mirroring to a pool on a different
cluster
(having the same pool name as the original), mirroring would be to a
pool on
the *same* cluster, (obviously having a pool different name)?

From the "rbd cppool" perspective, a one-shot mode of operation would be
fully sufficient - but looking at the code, I have not even been able to
identify the spots where we might "cut away" the networking part, so
that
rbd-mirror might do an intra-cluster job.

Are you able to judge how much work would need to be done, in order to
create a one-shot, intra-cluster version of rbd-mirror? Might it even be
something that could be a simple enhancement?



You might be interested in the deep-copy feature that will be included
in the Mimic release. By running "rbd deep-copy 
", it will fully copy the image, including snapshots and
parentage, to a new image. There is also work-in-progress for online
image migration [1] that will allow you to keep using the image while
it's being migrated to a new destination image. Both of these are
probably more suited to your needs than the heavy-weight RBD mirroring
process -- especially if you are only interested in the first step
since RBD mirroring now directly utilizes the deep-copy feature for
the initial image sync.



... also forgot to mention "rbd export --export-format 2" / "rbd
import --export-format 2" that will also deeply export/import all
snapshots associated with an image and that feature is available in
the Luminous release.


Thank you for any information and / or opinion you care to share!

With regards,
Jens



[1] https://github.com/ceph/ceph/pull/15831

--
Jason





--
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 8

Re: [ceph-users] Missing clones

[Eugen Block]

When we ran our test cluster with size 2 I experienced a similar  
issue, but that was in Hammer. There I could find the corresponding PG  
data in the filesystem and copy it to the damaged PG. But now we also  
run Bluestore on Luminous, I don't know yet how to fix this kind of  
issue, maybe someone else can share some thoughts on this.



Zitat von Karsten Becker :


Hi.

We have size=3 min_size=2.

But this "upgrade" has been done during the weekend. We had size=2
min_size=1 before.

Best
Karsten



On 19.02.2018 13:02, Eugen Block wrote:

Hi,

just to rule out the obvious, which size does the pool have? You aren't
running it with size = 2, do you?


Zitat von Karsten Becker :


Hi,

I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
fix this?


2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
missing
2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1
missing clone(s)
2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible
data damage: 1 pg inconsistent (PG_DAMAGED)
2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0 fixed
2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub
errors; Possible data damage: 1 pg inconsistent


"ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
started scrub "ceph pg scrub 10.7b9" also.

Best from Berlin/Germany
Karsten


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin
(Charlottenburg), HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com







Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin  
(Charlottenburg), HRB 57947

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw + OpenLDAP = Failed the auth strategy, reason=-13

[Konstantin Shalygin]

Hi cephers.


I try rgw (Luminous 12.2.2) + OpenLDAP. My settings:

    "rgw_ldap_binddn": "cn=cisco,ou=people,dc=example,dc=local",
    "rgw_ldap_dnattr": "uid",
    "rgw_ldap_searchdn": "ou=s3_users,dc=example,dc=local",
    "rgw_ldap_searchfilter": "(objectClass=inetOrgPerson)",
    "rgw_ldap_secret": "/etc/ceph/ldap_secret",
    "rgw_ldap_uri": "ldap://ldap.example.local:389";,
    "rgw_s3_auth_use_ldap": "true",


Test with ldapsearch:


# ldapsearch -x -D "cn=cisco,ou=people,dc=example,dc=local" -H 
ldap://ldap.example.local:389 -b "ou=s3_users,dc=example,dc=local" -w 
secret "(&(objectClass=inetOrgPerson)(uid=prometheus))"

# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (&(objectClass=inetOrgPerson)(uid=prometheus))
# requesting: ALL
#

# prometheus, s3_users, example.local
dn: cn=prometheus,ou=s3_users,dc=example,dc=local
sn: Prometheus
givenName: Exporter
uid: prometheus
loginShell: /usr/bin/bash
displayName: Prometheus Exporter
uidNumber: 1129
homeDirectory: /home/prometheus
telephoneNumber: 0
mail: r...@k0ste.ru
gidNumber: 1121
objectClass: inetOrgPerson
objectClass: posixAccount
cn: prometheus

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1



I was make token as described in docs:



# export RGW_ACCESS_KEY_ID="prometheus"  # ldap uid/cn
# export RGW_SECRET_ACCESS_KEY="prometheus" # ldap passwd
# radosgw-token --encode --ttype=ldap
ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogInByb21ldGhldXMiLAogICAgICAgICJrZXkiOiAicHJvbWV0aGV1cyIKICAgIH0KfQo=




And try to auth with s3cmd:





access_key = prometheus
access_token = 
ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogInByb21ldGhldXMiLAogICAgICA

gICJrZXkiOiAicHJvbWV0aGV1cyIKICAgIH0KfQo=




# s3cmd la
WARNING: Could not refresh role
ERROR: S3 error: 403 (AccessDenied)




rgw was successfully binds to OpenLDAP server with this settings, but 
query is not actually made. Queries was rejected by libldap (?) with 
reason='-13'. rgw logs:






2018-02-19 22:20:43.870254 7f7b9e36c700  2 
RGWDataChangesLog::ChangesRenewThread: start

2018-02-19 22:20:45.562318 7f7b85134700 20 CONTENT_LENGTH=0
2018-02-19 22:20:45.562344 7f7b85134700 20 HTTP_ACCEPT_ENCODING=identity
2018-02-19 22:20:45.562346 7f7b85134700 20 HTTP_AUTHORIZATION=AWS 
prometheus:AnbSRUM96QJtSBI32EIco2Go0e4=

2018-02-19 22:20:45.562348 7f7b85134700 20 HTTP_HOST=10.10.10.1:7480
2018-02-19 22:20:45.562349 7f7b85134700 20 HTTP_X_AMZ_DATE=Mon, 19 Feb 
2018 15:20:45 +
2018-02-19 22:20:45.562351 7f7b85134700 20 
HTTP_X_AMZ_SECURITY_TOKEN=ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogInByb21ldGhldXMiLAogICAgICAgICJrZXkiOiAicHJvbWV0aGV1cyIKICAgIH0KfQo=

2018-02-19 22:20:45.562356 7f7b85134700 20 REQUEST_METHOD=GET
2018-02-19 22:20:45.562357 7f7b85134700 20 REQUEST_URI=/
2018-02-19 22:20:45.562358 7f7b85134700 20 SCRIPT_URI=/
2018-02-19 22:20:45.562359 7f7b85134700 20 SERVER_PORT=7480
2018-02-19 22:20:45.562363 7f7b85134700  1 == starting new request 
req=0x7f7b8512e190 =
2018-02-19 22:20:45.562392 7f7b85134700  2 req 1:0.29::GET 
/::initializing for trans_id = 
tx1-005a8aeb4d-83def65-default
2018-02-19 22:20:45.562404 7f7b85134700 10 rgw api priority: s3=1 
s3website=-1

2018-02-19 22:20:45.562406 7f7b85134700 10 host=10.10.10.1
2018-02-19 22:20:45.562423 7f7b85134700 20 subdomain= domain= 
in_hosted_domain=0 in_hosted_domain_s3website=0
2018-02-19 22:20:45.562433 7f7b85134700 20 final domain/bucket 
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 
s->info.domain= s->info.request_uri=/

2018-02-19 22:20:45.562449 7f7b85134700 10 meta>> HTTP_X_AMZ_DATE
2018-02-19 22:20:45.562457 7f7b85134700 10 meta>> 
HTTP_X_AMZ_SECURITY_TOKEN
2018-02-19 22:20:45.562460 7f7b85134700 10 x>> x-amz-date:Mon, 19 Feb 
2018 15:20:45 +
2018-02-19 22:20:45.562463 7f7b85134700 10 x>> 
x-amz-security-token:ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogInByb21ldGhldXMiLAogICAgICAgICJrZXkiOiAicHJvbWV0aGV1cyIKICAgIH0KfQo=
2018-02-19 22:20:45.562496 7f7b85134700 20 get_handler 
handler=26RGWHandler_REST_Service_S3
2018-02-19 22:20:45.562510 7f7b85134700 10 
handler=26RGWHandler_REST_Service_S3
2018-02-19 22:20:45.562513 7f7b85134700  2 req 1:0.000150:s3:GET 
/::getting op 0

2018-02-19 22:20:45.562520 7f7b85134700 10 op=26RGWListBuckets_ObjStore_S3
2018-02-19 22:20:45.562539 7f7b85134700  2 req 1:0.000164:s3:GET 
/:list_buckets:verifying requester
2018-02-19 22:20:45.562548 7f7b85134700 20 
rgw::auth::StrategyRegistry::s3_main_strategy_t: trying 
rgw::auth::s3::AWSAuthStrategy
2018-02-19 22:20:45.562550 7f7b85134700 20 
rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2018-02-19 22:20:45.562556 7f7b85134700 20 
rgw::auth::s3::S3AnonymousEngine denied with 

Re: [ceph-users] Migrating to new pools

[Jason Dillaman]

Thanks!

On Mon, Feb 19, 2018 at 10:33 AM, Eugen Block  wrote:
> Hi,
>
> I created a ticket for the rbd import issue:
>
> https://tracker.ceph.com/issues/23038
>
> Regards,
>
> Eugen
>
>
> Zitat von Jason Dillaman :
>
>> On Fri, Feb 16, 2018 at 11:20 AM, Eugen Block  wrote:
>>>
>>> Hi Jason,
>>>
 ... also forgot to mention "rbd export --export-format 2" / "rbd
 import --export-format 2" that will also deeply export/import all
 snapshots associated with an image and that feature is available in
 the Luminous release.
>>>
>>>
>>>
>>> thanks for that information, this could be very valuable for us. I'll
>>> have
>>> to test that intesively, but not before next week.
>>>
>>> But a first quick test brought up a couple of issues which I'll have to
>>> re-check before bringing them up here.
>>>
>>> One issue is worth mentioning, though: After I exported (rbd export
>>> --export-format ...) a glance image and imported it back to a different
>>> pool
>>> (rbd import --export-format ...) its snapshot was copied, but not
>>> protected.
>>> This prevented nova from cloning the base image and leaving that instance
>>> in
>>> error state. Protecting the snapshot manually and launch another instance
>>> enabled nova to clone the image successfully.
>>>
>>> Could this be worth a bug report or is it rather something I did wrong or
>>> missed?
>>
>>
>> Definitely deserves a bug tracker ticket opened. Thanks.
>>
>>> I wish you all a nice weekend!
>>>
>>> Regards
>>> Eugen
>>>
>>>
>>> Zitat von Jason Dillaman :
>>>
 On Fri, Feb 16, 2018 at 8:08 AM, Jason Dillaman 
 wrote:
>
>
> On Fri, Feb 16, 2018 at 5:36 AM, Jens-U. Mozdzen 
> wrote:
>>
>>
>> Dear list, hello Jason,
>>
>> you may have seen my message on the Ceph mailing list about RDB pool
>> migration - it's a common subject that pools were created in a
>> sub-optimum
>> fashion and i. e. pgnum is (not yet) reducible, so we're looking into
>> means
>> to "clone" an RBD pool into a new pool within the same cluster
>> (including
>> snapshots).
>>
>> We had looked into creating a tool for this job, but soon noticed that
>> we're
>> duplicating basic functionality of rbd-mirror. So we tested the
>> following,
>> which worked out nicely:
>>
>> - create a test cluster (Ceph cluster plus an Openstack cluster using
>> an
>> RBD
>> pool) and some Openstack instances
>>
>> - create a second Ceph test cluster
>>
>> - stop Openstack
>>
>> - use rbd-mirror to clone the RBD pool from the first to the second
>> Ceph
>> cluster (IOW aborting rbd-mirror once the initial coping was done)
>>
>> - recreate the RDB pool on the first cluster
>>
>> - use rbd-mirror to clone the mirrored pool back to the (newly
>> created)
>> pool
>> on the first cluster
>>
>> - start Openstack and work with the (recreated) pool on the first
>> cluster
>>
>> So using rbd-mirror, we could clone an RBD pool's content to a
>> differently
>> structured pool on the same cluster - by using an intermediate
>> cluster.
>>
>> @Jason: Looking at the commit history for rbd-mirror, it seems you
>> might
>> be
>> able to shed some light on this: Do you see an easy way to modify
>> rbd-mirror
>> in such a fashion that instead of mirroring to a pool on a different
>> cluster
>> (having the same pool name as the original), mirroring would be to a
>> pool on
>> the *same* cluster, (obviously having a pool different name)?
>>
>> From the "rbd cppool" perspective, a one-shot mode of operation would
>> be
>> fully sufficient - but looking at the code, I have not even been able
>> to
>> identify the spots where we might "cut away" the networking part, so
>> that
>> rbd-mirror might do an intra-cluster job.
>>
>> Are you able to judge how much work would need to be done, in order to
>> create a one-shot, intra-cluster version of rbd-mirror? Might it even
>> be
>> something that could be a simple enhancement?
>
>
>
> You might be interested in the deep-copy feature that will be included
> in the Mimic release. By running "rbd deep-copy 
> ", it will fully copy the image, including snapshots and
> parentage, to a new image. There is also work-in-progress for online
> image migration [1] that will allow you to keep using the image while
> it's being migrated to a new destination image. Both of these are
> probably more suited to your needs than the heavy-weight RBD mirroring
> process -- especially if you are only interested in the first step
> since RBD mirroring now directly utilizes the deep-copy feature for
> the initial image sync.



 ... also forgot to mention "rbd export --export-format 2" / "rbd
 import --export-format 2" that will also deeply 

Re: [ceph-users] Missing clones

[Eugen Block]

Could [1] be of interest?
Exporting the intact PG and importing it back to the rescpective OSD  
sounds promising.


[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html


Zitat von Karsten Becker :


Hi.

We have size=3 min_size=2.

But this "upgrade" has been done during the weekend. We had size=2
min_size=1 before.

Best
Karsten



On 19.02.2018 13:02, Eugen Block wrote:

Hi,

just to rule out the obvious, which size does the pool have? You aren't
running it with size = 2, do you?


Zitat von Karsten Becker :


Hi,

I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
fix this?


2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
missing
2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1
missing clone(s)
2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible
data damage: 1 pg inconsistent (PG_DAMAGED)
2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0 fixed
2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub
errors; Possible data damage: 1 pg inconsistent


"ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
started scrub "ceph pg scrub 10.7b9" also.

Best from Berlin/Germany
Karsten


Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin
(Charlottenburg), HRB 57947
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com







Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin  
(Charlottenburg), HRB 57947

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

[Daniel Gryniewicz]

To my knowledge, no one has done any work on ganesha + ceph and selinux. 
 Fedora (and RHEL) includes config in it's selinux package for ganesha 
+ gluster, but I'm sure there's missing bits for ceph.


Daniel

On 02/17/2018 03:15 PM, Oliver Freyermuth wrote:

Hi together,

many thanks for the RPMs provided at:
   http://download.ceph.com/nfs-ganesha/
They are very much appreciated!


Since the statement was that they will also be maintained in the future, and 
NFS Ganesha seems an important project for the future of Ceph,
let me do the first "packaging" bug report.

It seems that the current packages do not play so well with SELinux. I'm 
currently using an SELinux module with the following allows, found by
iterative use of audit2allow (full ".te" module added at the end of the mail):

allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;

"cyphesis_port_t" is probably needed since its range (tcp: 6767, 6769, 
6780-6799) overlaps with the default ports
recommended for use by OSDs and nfs-ganesha uses libcephfs to talk to them, the 
other caps appear to be needed by nfs-ganesha itself.

With these in place, it seems my setup is working well. Without the "setgid" 
cap, for example, nfs-ganesha just segfaults after the permission denied failure.
Of course, it would be best if they were installed by the package (potentially, 
more restrictive allows are possible with some care).


Please include me in replies, I am not subscribed to the list.

Cheers and all the best,
Oliver



module nfs_ganesha-fix-perms 1.0;

require {
 type proc_net_t;
 type cyphesis_port_t;
 type ganesha_t;
 class capability setuid;
 class capability setgid;
 class capability dac_override;
 class tcp_socket name_connect;
 class file { getattr open read };
}

#= ganesha_t ==
allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Signature check failures.

[Cary]

Gregory,


I greatly appreciate your assistance. I recompiled Ceph with -ssl and
the nss USE flags set, which is opposite what I was using. I am now
able to export from our pools without signature check failures. Thank
you for pointing me in the right direction.

Cary
-Dynamic



On Fri, Feb 16, 2018 at 11:29 PM, Gregory Farnum  wrote:
> On Thu, Feb 15, 2018 at 10:28 AM Cary  wrote:
>>
>> Hello,
>>
>> I have enabled debugging on my MONs and OSDs to help troubleshoot
>> these signature check failures. I was watching ods.4's log and saw
>> these errors when the signature check failure happened.
>>
>> 2018-02-15 18:06:29.235791 7f8bca7de700  1 --
>> 192.168.173.44:6806/72264 >> 192.168.173.42:0/4264467021
>> conn(0x55f802746000 :6806 s=STATE_OPEN pgs=7 cs=1 l=1).read_bulk peer
>> close file descriptor 81
>> 2018-02-15 18:06:29.235832 7f8bca7de700  1 --
>> 192.168.173.44:6806/72264 >> 192.168.173.42:0/4264467021
>> conn(0x55f802746000 :6806 s=STATE_OPEN pgs=7 cs=1 l=1).read_until read
>> failed
>> 2018-02-15 18:06:29.235841 7f8bca7de700  1 --
>> 192.168.173.44:6806/72264 >> 192.168.173.42:0/4264467021
>> conn(0x55f802746000 :6806 s=STATE_OPEN pgs=7 cs=1 l=1).process read
>> tag failed
>> 2018-02-15 18:06:29.235848 7f8bca7de700  1 --
>> 192.168.173.44:6806/72264 >> 192.168.173.42:0/4264467021
>> conn(0x55f802746000 :6806 s=STATE_OPEN pgs=7 cs=1 l=1).fault on lossy
>> channel, failing
>> 2018-02-15 18:06:29.235966 7f8bc0853700  2 osd.8 27498 ms_handle_reset
>> con 0x55f802746000 session 0x55f8063b3180
>>
>>
>>  Could someone please look at this? We have 3 different Ceph clusters
>> setup and they all have this issue. This cluster is running Gentoo and
>> Ceph version 12.2.2-r1. The other two clusters are 12.2.2. Exporting
>> images causes signature check failures and with larger files it seg
>> faults as well.
>>
>> When exporting the image from osd.4 This message shows up as well.
>> Exporting image: 1% complete...2018-02-15 18:14:05.283708 7f6834277700
>>  0 -- 192.168.173.44:0/122241099 >> 192.168.173.44:6801/72152
>> conn(0x7f681400ff10 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH
>> pgs=0 cs=0 l=1).handle_connect_reply connect got BADAUTHORIZER
>>
>> The error below show up on all OSD/MGR/MON nodes when exporting an image.
>> Exporting image: 8% complete...2018-02-15 18:15:51.419437 7f2b64ac0700
>>  0 SIGN: MSG 28 Message signature does not match contents.
>> 2018-02-15 18:15:51.419459 7f2b64ac0700  0 SIGN: MSG 28Signature on
>> message:
>> 2018-02-15 18:15:51.419460 7f2b64ac0700  0 SIGN: MSG 28sig:
>> 8338581684421737157
>> 2018-02-15 18:15:51.419469 7f2b64ac0700  0 SIGN: MSG 28Locally
>> calculated signature:
>> 2018-02-15 18:15:51.419470 7f2b64ac0700  0 SIGN: MSG 28
>> sig_check:5913182128308244
>> 2018-02-15 18:15:51.419471 7f2b64ac0700  0 Signature failed.
>> 2018-02-15 18:15:51.419472 7f2b64ac0700  0 --
>> 192.168.173.44:0/3919097436 >> 192.168.173.44:6801/72152
>> conn(0x7f2b4800ff10 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
>> pgs=39 cs=1 l=1).process Signature check failed
>>
>> Our VMs crash when writing to disk. Libvirt's logs just say the VM
>> crashed.   This is a blocker. Has anyone else seen this? This seems to
>> be an issue with Ceph Luminous, as we were not having these problem
>> with Jewel.
>
>
> When I search through my email, the only two reports of failed signatures
> are people who in fact had misconfiguration issues resulting in one end
> using signatures and the other side not.
>
> Given that, and since you're on Gentoo and presumably compiled the packages
> yourself, the most likely explanation I can think of is something that went
> wrong between your packages and the compilation. :/
>
> I guess you could try switching from libnss to libcryptopp (or vice versa)
> by recompiling with the relevant makeflags if you want to do something that
> only involves the Ceph code. Otherwise, do a rebuild?
>
> Sadly I don't think there's much else we can suggest given that nobody has
> seen this with binary packages blessed by the upstream or a distribution.
> -Greg
>
>>
>>
>> Cary
>> -Dynamic
>>
>> On Thu, Feb 1, 2018 at 7:04 PM, Cary  wrote:
>> > Hello,
>> >
>> > I did not do anything special that I know of. I was just exporting an
>> > image from Openstack. We have recently upgraded from Jewel 10.2.3 to
>> > Luminous 12.2.2.
>> >
>> > Caps for admin:
>> > client.admin
>> > key: CENSORED
>> > auid: 0
>> > caps: [mgr] allow *
>> > caps: [mon] allow *
>> > caps: [osd] allow *
>> >
>> > Caps for Cinder:
>> > client.cinder
>> > key: CENSORED
>> > caps: [mgr] allow r
>> > caps: [mon] profile rbd, allow command "osd blacklist"
>> > caps: [osd] profile rbd pool=vms, profile rbd pool=volumes,
>> > profile rbd pool=images
>> >
>> > Caps for MGR:
>> > mgr.0
>> > key: CENSORED
>> > caps: [mon] allow *
>> >
>> > I believe this is causing the virtual machines we have running to
>> > crash. Any advice wou

Re: [ceph-users] Missing clones

[Karsten Becker]

Hi.

Thank you for the tip. I just tried... but unfortunately the import aborts:

> Write #10:9de96eca:::rbd_data.f5b8603d1b58ba.1d82:head#
> snapset 0=[]:{}
> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:18#
> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:24#
> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:head#
> snapset 628=[24,21,17]:{18=[17],24=[24,21]}
> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: In function 'void 
> SnapMapper::add_oid(const hobject_t&, const std::set&, 
> MapCacher::Transaction, 
> ceph::buffer::list>*)' thread 7facba7de400 time 2018-02-19 19:24:18.917515
> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: 246: FAILED assert(r 
> == -2)
>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
> (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
> const*)+0x102) [0x7facb0c2a8f2]
>  2: (SnapMapper::add_oid(hobject_t const&, std::set std::less, std::allocator > const&, 
> MapCacher::Transaction std::char_traits, std::allocator >, ceph::buffer::list>*)+0x8e9) 
> [0x55eef3894fe9]
>  3: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, 
> ceph::buffer::list&, OSDriver&, SnapMapper&)+0xafb) [0x55eef35f901b]
>  4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, 
> OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
>  5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
> std::__cxx11::basic_string, std::allocator 
> >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
>  6: (main()+0x3909) [0x55eef3561349]
>  7: (__libc_start_main()+0xf1) [0x7facae0892b1]
>  8: (_start()+0x2a) [0x55eef35e901a]
>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
> interpret this.
> *** Caught signal (Aborted) **
>  in thread 7facba7de400 thread_name:ceph-objectstor
>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
> (stable)
>  1: (()+0x913f14) [0x55eef3c10f14]
>  2: (()+0x110c0) [0x7facaf5020c0]
>  3: (gsignal()+0xcf) [0x7facae09bfcf]
>  4: (abort()+0x16a) [0x7facae09d3fa]
>  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
> const*)+0x28e) [0x7facb0c2aa7e]
>  6: (SnapMapper::add_oid(hobject_t const&, std::set std::less, std::allocator > const&, 
> MapCacher::Transaction std::char_traits, std::allocator >, ceph::buffer::list>*)+0x8e9) 
> [0x55eef3894fe9]
>  7: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, 
> ceph::buffer::list&, OSDriver&, SnapMapper&)+0xafb) [0x55eef35f901b]
>  8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, 
> OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
>  9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
> std::__cxx11::basic_string, std::allocator 
> >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
>  10: (main()+0x3909) [0x55eef3561349]
>  11: (__libc_start_main()+0xf1) [0x7facae0892b1]
>  12: (_start()+0x2a) [0x55eef35e901a]
> Aborted

Best
Karsten

On 19.02.2018 17:09, Eugen Block wrote:
> Could [1] be of interest?
> Exporting the intact PG and importing it back to the rescpective OSD
> sounds promising.
> 
> [1]
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html
> 
> 
> Zitat von Karsten Becker :
> 
>> Hi.
>>
>> We have size=3 min_size=2.
>>
>> But this "upgrade" has been done during the weekend. We had size=2
>> min_size=1 before.
>>
>> Best
>> Karsten
>>
>>
>>
>> On 19.02.2018 13:02, Eugen Block wrote:
>>> Hi,
>>>
>>> just to rule out the obvious, which size does the pool have? You aren't
>>> running it with size = 2, do you?
>>>
>>>
>>> Zitat von Karsten Becker :
>>>
 Hi,

 I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
 fix this?

> 2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
> clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
> missing
> 2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9
> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1
> missing clone(s)
> 2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible
> data damage: 1 pg inconsistent (PG_DAMAGED)
> 2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0
> fixed
> 2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub
> errors; Possible data damage: 1 pg inconsistent

 "ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
 started scrub "ceph pg scrub 10.7b9" also.

 Best from Berlin/Germany
 Karsten


 Ecologic Institut gemeinnuetzige GmbH
 Pfalzburger Str. 43/44, D-10717 Berlin
 Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
 Sitz der Gesellschaft / Registered Office: Berlin (Germany)
 Registergericht / Court of Registration: Amt

Re: [ceph-users] ceph df: Raw used vs. used vs. actual bytes in cephfs

[Pavel Shub]

Could you be running into block size (minimum allocation unit)
overhead? Default bluestore block size is 4k for hdd and 64k for ssd.
This is exacerbated if you have tons of small files. I tend to see
this when "ceph df detail" sum of raw used in pools is less than the
global raw bytes used.

On Mon, Feb 19, 2018 at 2:09 AM, Flemming Frandsen
 wrote:
> Each OSD lives on a separate HDD in bluestore with the journals on 2GB
> partitions on a shared SSD.
>
>
> On 16/02/18 21:08, Gregory Farnum wrote:
>
> What does the cluster deployment look like? Usually this happens when you’re
> sharing disks with the OS, or have co-located file journals or something.
> On Fri, Feb 16, 2018 at 4:02 AM Flemming Frandsen
>  wrote:
>>
>> I'm trying out cephfs and I'm in the process of copying over some
>> real-world data to see what happens.
>>
>> I have created a number of cephfs file systems, the only one I've
>> started working on is the one called jenkins specifically the one named
>> jenkins which lives in fs_jenkins_data and fs_jenkins_metadata.
>>
>> According to ceph df I have about 1387 GB of data in all of the pools,
>> while the raw used space is 5918 GB, which gives a ratio of about 4.3, I
>> would have expected a ratio around 2 as the pool size has been set to 2.
>>
>>
>> Can anyone explain where half my space has been squandered?
>>
>>  > ceph df
>> GLOBAL:
>>  SIZE  AVAIL RAW USED %RAW USED
>>  8382G 2463G5918G 70.61
>> POOLS:
>>  NAME ID USED   %USED MAX
>> AVAIL OBJECTS
>>  .rgw.root11113 0 258G
>> 4
>>  default.rgw.control  2   0 0 258G
>> 8
>>  default.rgw.meta 3   0 0 258G
>> 0
>>  default.rgw.log  4   0 0 258G
>> 207
>>  fs_docker-nexus_data 5  66120M 11.09 258G
>> 22655
>>  fs_docker-nexus_metadata 6  39463k 0 258G
>> 2376
>>  fs_meta_data 7 330 0 258G
>> 4
>>  fs_meta_metadata 8567k 0 258G
>> 22
>>  fs_jenkins_data  9   1321G 71.84 258G
>> 28576278
>>  fs_jenkins_metadata  10 52178k 0 258G
>> 2285493
>>  fs_nexus_data11  0 0 258G
>> 0
>>  fs_nexus_metadata12   4181 0 258G
>> 21
>>
>> --
>>   Regards Flemming Frandsen - Stibo Systems - DK - STEP Release Manager
>>   Please use rele...@stibo.com for all Release Management requests
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
>  Regards Flemming Frandsen - Stibo Systems - DK - STEP Release Manager
>  Please use rele...@stibo.com for all Release Management requests
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Luminous : performance degrade while read operations (ceph-volume)

[nokia ceph]

Hi All,

We have 5 node clusters with EC 4+1 and use bluestore since last year from
Kraken.
Recently we migrated all our platforms to luminous 12.2.2 and finally all
OSDs migrated to ceph-volume simple type and on few platforms installed
ceph using ceph-volume .

Now we see two times more traffic in read compare to client traffic on
migrated platform and newly created platforms . This was not the case in
older releases where ceph status read B/W will be same as client read
traffic.

Some network graphs :

*Client network interface* towards ceph public interface : shows *4.3Gbps*
read


[image: Inline image 2]

*Ceph Node Public interface* : Each node around 960Mbps * 5 node =* 4.6
Gbps *- this matches.
[image: Inline image 3]

Ceph status output : show  1032 MB/s =* 8.06 Gbps*

cn6.chn6us1c1.cdn ~# ceph status
  cluster:
id: abda22db-3658-4d33-9681-e3ff10690f88
health: HEALTH_OK

  services:
mon: 5 daemons, quorum cn6,cn7,cn8,cn9,cn10
mgr: cn6(active), standbys: cn7, cn9, cn10, cn8
osd: 340 osds: 340 up, 340 in

  data:
pools:   1 pools, 8192 pgs
objects: 270M objects, 426 TB
usage:   581 TB used, 655 TB / 1237 TB avail
pgs: 8160 active+clean
 32   active+clean+scrubbing

  io:
client:   *1032 MB/s rd*, 168 MB/s wr, 1908 op/s rd, 1594 op/s wr


Write operation we don't see this issue. Client traffic and this matches.
Is this expected behavior in Luminous and ceph-volume lvm or a bug ?
Wrong calculation in ceph status read B/W ?

Please provide your feedback.

Thanks,
Muthu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous : performance degrade while read operations (ceph-volume)

[Alfredo Deza]

On Mon, Feb 19, 2018 at 2:01 PM, nokia ceph 
wrote:

> Hi All,
>
> We have 5 node clusters with EC 4+1 and use bluestore since last year from
> Kraken.
> Recently we migrated all our platforms to luminous 12.2.2 and finally all
> OSDs migrated to ceph-volume simple type and on few platforms installed
> ceph using ceph-volume .
>
> Now we see two times more traffic in read compare to client traffic on
> migrated platform and newly created platforms . This was not the case in
> older releases where ceph status read B/W will be same as client read
> traffic.
>
> Some network graphs :
>
> *Client network interface* towards ceph public interface : shows *4.3Gbps*
> read
>
>
> [image: Inline image 2]
>
> *Ceph Node Public interface* : Each node around 960Mbps * 5 node =* 4.6
> Gbps *- this matches.
> [image: Inline image 3]
>
> Ceph status output : show  1032 MB/s =* 8.06 Gbps*
>
> cn6.chn6us1c1.cdn ~# ceph status
>   cluster:
> id: abda22db-3658-4d33-9681-e3ff10690f88
> health: HEALTH_OK
>
>   services:
> mon: 5 daemons, quorum cn6,cn7,cn8,cn9,cn10
> mgr: cn6(active), standbys: cn7, cn9, cn10, cn8
> osd: 340 osds: 340 up, 340 in
>
>   data:
> pools:   1 pools, 8192 pgs
> objects: 270M objects, 426 TB
> usage:   581 TB used, 655 TB / 1237 TB avail
> pgs: 8160 active+clean
>  32   active+clean+scrubbing
>
>   io:
> client:   *1032 MB/s rd*, 168 MB/s wr, 1908 op/s rd, 1594 op/s wr
>
>
> Write operation we don't see this issue. Client traffic and this matches.
> Is this expected behavior in Luminous and ceph-volume lvm or a bug ?
> Wrong calculation in ceph status read B/W ?
>

You mentioned `ceph-volume simple` but here you say lvm. With LVM
ceph-volume will create the OSDs from scratch, while "simple" will keep
whatever OSD was created before.

Have you created the OSDs from scratch with ceph-volume? or is it just
using "simple" , managing a previously deployed OSD?

>
> Please provide your feedback.
>
> Thanks,
> Muthu
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Karsten Becker]

BTW - how can I find out, which RBDs are affected by this problem. Maybe
a copy/remove of the affected RBDs could help? But how to find out to
which RBDs this PG belongs to?

Best
Karsten

On 19.02.2018 19:26, Karsten Becker wrote:
> Hi.
> 
> Thank you for the tip. I just tried... but unfortunately the import aborts:
> 
>> Write #10:9de96eca:::rbd_data.f5b8603d1b58ba.1d82:head#
>> snapset 0=[]:{}
>> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:18#
>> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:24#
>> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:head#
>> snapset 628=[24,21,17]:{18=[17],24=[24,21]}
>> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: In function 'void 
>> SnapMapper::add_oid(const hobject_t&, const std::set&, 
>> MapCacher::Transaction, 
>> ceph::buffer::list>*)' thread 7facba7de400 time 2018-02-19 19:24:18.917515
>> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: 246: FAILED assert(r 
>> == -2)
>>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
>> (stable)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
>> const*)+0x102) [0x7facb0c2a8f2]
>>  2: (SnapMapper::add_oid(hobject_t const&, std::set> std::less, std::allocator > const&, 
>> MapCacher::Transaction> std::char_traits, std::allocator >, ceph::buffer::list>*)+0x8e9) 
>> [0x55eef3894fe9]
>>  3: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, 
>> ceph::buffer::list&, OSDriver&, SnapMapper&)+0xafb) [0x55eef35f901b]
>>  4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, 
>> OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
>>  5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
>> std::__cxx11::basic_string, 
>> std::allocator >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
>>  6: (main()+0x3909) [0x55eef3561349]
>>  7: (__libc_start_main()+0xf1) [0x7facae0892b1]
>>  8: (_start()+0x2a) [0x55eef35e901a]
>>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
>> interpret this.
>> *** Caught signal (Aborted) **
>>  in thread 7facba7de400 thread_name:ceph-objectstor
>>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
>> (stable)
>>  1: (()+0x913f14) [0x55eef3c10f14]
>>  2: (()+0x110c0) [0x7facaf5020c0]
>>  3: (gsignal()+0xcf) [0x7facae09bfcf]
>>  4: (abort()+0x16a) [0x7facae09d3fa]
>>  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
>> const*)+0x28e) [0x7facb0c2aa7e]
>>  6: (SnapMapper::add_oid(hobject_t const&, std::set> std::less, std::allocator > const&, 
>> MapCacher::Transaction> std::char_traits, std::allocator >, ceph::buffer::list>*)+0x8e9) 
>> [0x55eef3894fe9]
>>  7: (get_attrs(ObjectStore*, coll_t, ghobject_t, ObjectStore::Transaction*, 
>> ceph::buffer::list&, OSDriver&, SnapMapper&)+0xafb) [0x55eef35f901b]
>>  8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, ceph::buffer::list&, 
>> OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
>>  9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
>> std::__cxx11::basic_string, 
>> std::allocator >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
>>  10: (main()+0x3909) [0x55eef3561349]
>>  11: (__libc_start_main()+0xf1) [0x7facae0892b1]
>>  12: (_start()+0x2a) [0x55eef35e901a]
>> Aborted
> 
> Best
> Karsten
> 
> On 19.02.2018 17:09, Eugen Block wrote:
>> Could [1] be of interest?
>> Exporting the intact PG and importing it back to the rescpective OSD
>> sounds promising.
>>
>> [1]
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html
>>
>>
>> Zitat von Karsten Becker :
>>
>>> Hi.
>>>
>>> We have size=3 min_size=2.
>>>
>>> But this "upgrade" has been done during the weekend. We had size=2
>>> min_size=1 before.
>>>
>>> Best
>>> Karsten
>>>
>>>
>>>
>>> On 19.02.2018 13:02, Eugen Block wrote:
 Hi,

 just to rule out the obvious, which size does the pool have? You aren't
 running it with size = 2, do you?


 Zitat von Karsten Becker :

> Hi,
>
> I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
> fix this?
>
>> 2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
>> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
>> clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
>> missing
>> 2018-02-19 11:00:23.183707 osd.29 [INF] repair 10.7b9
>> 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head 1
>> missing clone(s)
>> 2018-02-19 11:01:18.074666 mon.0 [ERR] Health check update: Possible
>> data damage: 1 pg inconsistent (PG_DAMAGED)
>> 2018-02-19 11:01:11.856529 osd.29 [ERR] 10.7b9 repair 1 errors, 0
>> fixed
>> 2018-02-19 11:01:24.333533 mon.0 [ERR] overall HEALTH_ERR 1 scrub
>> errors; Possible data damage: 1 pg inconsistent
>
> "ceph pg repair 10.7b9" fails and is not able to fix ist. A manually
> s

Re: [ceph-users] Understanding/correcting sudden onslaught of unfound objects

[Graham Allan]

On 02/17/2018 12:48 PM, David Zafman wrote:


The commits below came after v12.2.2 and may impact this issue. When a 
pg is active+clean+inconsistent means that scrub has detected issues 
with 1 or more replicas of 1 or more objects .  An unfound object is a 
potentially temporary state in which the current set of available OSDs 
doesn't allow an object to be recovered/backfilled/repaired.  When the 
primary OSD restarts, any unfound objects ( an in memory structure) are 
reset so that the new set of peered OSDs can determine again what 
objects are unfound.


I'm not clear in this scenario whether recovery failed to start, 
recovery hung before due to a bug or if recovery stopped (as designed) 
because of the unfound object.  The new recovery_unfound and 
backfill_unfound states indicates that recovery has stopped due to 
unfound objects.


Thanks for your comments David. I could certainly enable any additional 
logging that might help to clarify what's going on here - perhaps on the 
primary OSD for a given pg?


I am still having a hard time understanding why these objects repeatedly 
get flagged as unfound, when they are downloadable and contain correct 
data whenever they are not in this state. It is a 4+2 EC pool, so I 
would think it possible to reconstruct any missing EC chunks.


It's an extensive problem; while I have been focusing on examining a 
couple of specific pgs, the pool in general is showing 2410 pgs 
inconsistent (out of 4096).


Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

Sounds like the go sdk adds a location constraint to requests that
don't go to us-east-1. RGW itself is definitely isn't tied to
us-east-1, and does not know anything about it (unless you happen to
have a zonegroup named us-east-1). Maybe there's a way to configure
the sdk to avoid doing that?

Yehuda

On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:
> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to talk
> to my RGW instance using the s3 interface. I am running ceph in docker using
> the ceph/daemon docker images in demo mode. The RGW is started with a
> zonegroup and zone with their names set to an empty string by the scripts in
> the image.
>
> I have ForcePathStyle for the client set to true, because I want to access
> all my buckets using the path: myrgw.instance:8080/somebucket.
>
> I noticed that if I set the region for the client to anything other than
> us-east-1, I get this error when creating a bucket:
> InvalidLocationConstraint: The specified location-constraint is not valid.
>
> If I set the region in the client to something made up, such as "ceph" and
> the LocationConstraint to "ceph", I still get the same error.
>
> The only way to get my buckets to create successfully is to set the client's
> region to us-east-1. I have grepped the ceph code base and cannot find any
> references to us-east-1. In addition, I looked at the AWS docs for
> calculating v4 signatures and us-east-1 is the default region but I can see
> that the region string is used in the calculation (i.e. the region is not
> ignored when calculating the signature if it is set to us-east-1).
>
> Why do my buckets create successfully if I set the region in my s3 client to
> us-east-1, but not otherwise? If I do not want to use us-east-1 as my
> default region, for example, if I want us-west-1 as my default region, what
> should I be configuring in ceph?
>
> Thanks,
>
> Francis
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Eugen Block]

BTW - how can I find out, which RBDs are affected by this problem. Maybe
a copy/remove of the affected RBDs could help? But how to find out to
which RBDs this PG belongs to?


Depending on how many PGs your cluster/pool has, you could dump your  
osdmap and then run the osdmaptool [1] for every rbd object in your  
pool and grep for the affected PG. That would be quick for a few  
objects, I guess:


ceph1:~ # ceph osd getmap -o /tmp/osdmap

ceph1:~ # osdmaptool --test-map-object image1 --pool 5 /tmp/osdmap
osdmaptool: osdmap file '/tmp/osdmap'
 object 'image1' -> 5.2 -> [0]

ceph1:~ # osdmaptool --test-map-object image2 --pool 5 /tmp/osdmap
osdmaptool: osdmap file '/tmp/osdmap'
 object 'image2' -> 5.f -> [0]


[1]  
https://www.hastexo.com/resources/hints-and-kinks/which-osd-stores-specific-rados-object/


Zitat von Karsten Becker :


BTW - how can I find out, which RBDs are affected by this problem. Maybe
a copy/remove of the affected RBDs could help? But how to find out to
which RBDs this PG belongs to?

Best
Karsten

On 19.02.2018 19:26, Karsten Becker wrote:

Hi.

Thank you for the tip. I just tried... but unfortunately the import aborts:


Write #10:9de96eca:::rbd_data.f5b8603d1b58ba.1d82:head#
snapset 0=[]:{}
Write #10:9de973fe:::rbd_data.966489238e1f29.250b:18#
Write #10:9de973fe:::rbd_data.966489238e1f29.250b:24#
Write #10:9de973fe:::rbd_data.966489238e1f29.250b:head#
snapset 628=[24,21,17]:{18=[17],24=[24,21]}
/home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: In  
function 'void SnapMapper::add_oid(const hobject_t&, const  
std::set&,  
MapCacher::Transaction,  
ceph::buffer::list>*)' thread 7facba7de400 time 2018-02-19  
19:24:18.917515
/home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: 246:  
FAILED assert(r == -2)
 ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf)  
luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char  
const*)+0x102) [0x7facb0c2a8f2]
 2: (SnapMapper::add_oid(hobject_t const&, std::setstd::less, std::allocator > const&,  
MapCacher::Transactionstd::char_traits, std::allocator >,  
ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
 3: (get_attrs(ObjectStore*, coll_t, ghobject_t,  
ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&,  
SnapMapper&)+0xafb) [0x55eef35f901b]
 4: (ObjectStoreTool::get_object(ObjectStore*, coll_t,  
ceph::buffer::list&, OSDMap&, bool*,  
ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
 5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&,  
bool, std::__cxx11::basic_string,  
std::allocator >, ObjectStore::Sequencer&)+0x1135)  
[0x55eef36002f5]

 6: (main()+0x3909) [0x55eef3561349]
 7: (__libc_start_main()+0xf1) [0x7facae0892b1]
 8: (_start()+0x2a) [0x55eef35e901a]
 NOTE: a copy of the executable, or `objdump -rdS ` is  
needed to interpret this.

*** Caught signal (Aborted) **
 in thread 7facba7de400 thread_name:ceph-objectstor
 ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf)  
luminous (stable)

 1: (()+0x913f14) [0x55eef3c10f14]
 2: (()+0x110c0) [0x7facaf5020c0]
 3: (gsignal()+0xcf) [0x7facae09bfcf]
 4: (abort()+0x16a) [0x7facae09d3fa]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char  
const*)+0x28e) [0x7facb0c2aa7e]
 6: (SnapMapper::add_oid(hobject_t const&, std::setstd::less, std::allocator > const&,  
MapCacher::Transactionstd::char_traits, std::allocator >,  
ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
 7: (get_attrs(ObjectStore*, coll_t, ghobject_t,  
ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&,  
SnapMapper&)+0xafb) [0x55eef35f901b]
 8: (ObjectStoreTool::get_object(ObjectStore*, coll_t,  
ceph::buffer::list&, OSDMap&, bool*,  
ObjectStore::Sequencer&)+0x738) [0x55eef35f9ae8]
 9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&,  
bool, std::__cxx11::basic_string,  
std::allocator >, ObjectStore::Sequencer&)+0x1135)  
[0x55eef36002f5]

 10: (main()+0x3909) [0x55eef3561349]
 11: (__libc_start_main()+0xf1) [0x7facae0892b1]
 12: (_start()+0x2a) [0x55eef35e901a]
Aborted


Best
Karsten

On 19.02.2018 17:09, Eugen Block wrote:

Could [1] be of interest?
Exporting the intact PG and importing it back to the rescpective OSD
sounds promising.

[1]
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019673.html


Zitat von Karsten Becker :


Hi.

We have size=3 min_size=2.

But this "upgrade" has been done during the weekend. We had size=2
min_size=1 before.

Best
Karsten



On 19.02.2018 13:02, Eugen Block wrote:

Hi,

just to rule out the obvious, which size does the pool have? You aren't
running it with size = 2, do you?


Zitat von Karsten Becker :


Hi,

I have one damaged PG in my cluster. All OSDs are BlueStore. How do I
fix this?


2018-02-19 11:00:23.183695 osd.29 [ERR] repair 10.7b9
10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:head expected
clone 10:9defb021:::rbd_data.2313975238e1f29.0002cbb5:64e 1
missing
2018-02-19 11:00:23.183707 osd.29 [INF] r

Re: [ceph-users] rgw bucket inaccessible - appears to be using incorrect index pool?

[Graham Allan]

Thanks Robin,

Of the two issues, this seems to me like it must be #22928.

Since the majority of index entries for this bucket are in the 
.rgw.buckets pool, but newer entries have been created in 
.rgw.buckets.index, it's clearly failing to use the explicit placement 
pool - and with the index data split across two pools I don't see how 
resharding could correct this.


I can get the object names from the "new" (incorrect) indexes with 
something like:



for i in `rados -p .rgw.buckets.index ls - | grep "default.2049236.2"`; do rados -p 
.rgw.buckets.index listomapkeys $i|grep "^[a-zA-Z0-9]"; done


Fortunately there are only ~20, in this bucket (my grep is just a stupid 
way to skip what I assume are the multipart parts, which have a 
non-ascii first char).


... and these files are downloadable, at least using s3cmd (minio client 
fails, it seems to try and check the index first).


Once I have these newer files downloaded, then to restore access to the 
older index I like the suggestion in the issue tracker to create a new 
placement target in the zone, and modify the bucket's placement rule to 
match. It seems like it might be safer than copying objects from one 
index pool to the other (though the latter certainly sounds faster and 
easier!)


From a quick check, I suspect I probably have 40 or so other buckets 
with this problem... will need to check them more closely.


Actually it looks like a lot of the affected buckets were created around 
10/2016 - I suspect the placement policies were incorrect for a short 
time due to confusion over the hammer->jewel upgrade (the 
realm/period/zonegroup/zone conversion didn't really go smoothly!)


On 02/16/2018 11:39 PM, Robin H. Johnson wrote:

On Fri, Feb 16, 2018 at 07:06:21PM -0600, Graham Allan wrote:
[snip great debugging]

This seems similar to two open issues, could be either of them depending
on how old that bucket is.
http://tracker.ceph.com/issues/22756
http://tracker.ceph.com/issues/22928

- I have a mitigation posted to 22756.
- There's a PR posted for 22928, but it'll probably only be in v12.2.4.



--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Mykola Golub]

On Mon, Feb 19, 2018 at 10:17:55PM +0100, Karsten Becker wrote:
> BTW - how can I find out, which RBDs are affected by this problem. Maybe
> a copy/remove of the affected RBDs could help? But how to find out to
> which RBDs this PG belongs to?

In this case rbd_data.966489238e1f29.250b looks like the
problem object. To find out which RBD image it belongs to you can run
`rbd info /` command for every image in the pool, looking at
block_name_prefix field, until you find 'rbd_data.966489238e1f29'.

> 
> Best
> Karsten
> 
> On 19.02.2018 19:26, Karsten Becker wrote:
> > Hi.
> > 
> > Thank you for the tip. I just tried... but unfortunately the import aborts:
> > 
> >> Write #10:9de96eca:::rbd_data.f5b8603d1b58ba.1d82:head#
> >> snapset 0=[]:{}
> >> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:18#
> >> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:24#
> >> Write #10:9de973fe:::rbd_data.966489238e1f29.250b:head#
> >> snapset 628=[24,21,17]:{18=[17],24=[24,21]}
> >> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: In function 'void 
> >> SnapMapper::add_oid(const hobject_t&, const std::set&, 
> >> MapCacher::Transaction, 
> >> ceph::buffer::list>*)' thread 7facba7de400 time 2018-02-19 19:24:18.917515
> >> /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: 246: FAILED 
> >> assert(r == -2)
> >>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
> >> (stable)
> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
> >> const*)+0x102) [0x7facb0c2a8f2]
> >>  2: (SnapMapper::add_oid(hobject_t const&, std::set >> std::less, std::allocator > const&, 
> >> MapCacher::Transaction >> std::char_traits, std::allocator >, 
> >> ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
> >>  3: (get_attrs(ObjectStore*, coll_t, ghobject_t, 
> >> ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, 
> >> SnapMapper&)+0xafb) [0x55eef35f901b]
> >>  4: (ObjectStoreTool::get_object(ObjectStore*, coll_t, 
> >> ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) 
> >> [0x55eef35f9ae8]
> >>  5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
> >> std::__cxx11::basic_string, 
> >> std::allocator >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
> >>  6: (main()+0x3909) [0x55eef3561349]
> >>  7: (__libc_start_main()+0xf1) [0x7facae0892b1]
> >>  8: (_start()+0x2a) [0x55eef35e901a]
> >>  NOTE: a copy of the executable, or `objdump -rdS ` is needed 
> >> to interpret this.
> >> *** Caught signal (Aborted) **
> >>  in thread 7facba7de400 thread_name:ceph-objectstor
> >>  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf) luminous 
> >> (stable)
> >>  1: (()+0x913f14) [0x55eef3c10f14]
> >>  2: (()+0x110c0) [0x7facaf5020c0]
> >>  3: (gsignal()+0xcf) [0x7facae09bfcf]
> >>  4: (abort()+0x16a) [0x7facae09d3fa]
> >>  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
> >> const*)+0x28e) [0x7facb0c2aa7e]
> >>  6: (SnapMapper::add_oid(hobject_t const&, std::set >> std::less, std::allocator > const&, 
> >> MapCacher::Transaction >> std::char_traits, std::allocator >, 
> >> ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
> >>  7: (get_attrs(ObjectStore*, coll_t, ghobject_t, 
> >> ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&, 
> >> SnapMapper&)+0xafb) [0x55eef35f901b]
> >>  8: (ObjectStoreTool::get_object(ObjectStore*, coll_t, 
> >> ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0x738) 
> >> [0x55eef35f9ae8]
> >>  9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool, 
> >> std::__cxx11::basic_string, 
> >> std::allocator >, ObjectStore::Sequencer&)+0x1135) [0x55eef36002f5]
> >>  10: (main()+0x3909) [0x55eef3561349]
> >>  11: (__libc_start_main()+0xf1) [0x7facae0892b1]
> >>  12: (_start()+0x2a) [0x55eef35e901a]
> >> Aborted

-- 
Mykola Golub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[David Turner]

I wasn't using the Go SDK.  I was using s3cmd when I came across this.

On Mon, Feb 19, 2018 at 4:42 PM Yehuda Sadeh-Weinraub 
wrote:

> Sounds like the go sdk adds a location constraint to requests that
> don't go to us-east-1. RGW itself is definitely isn't tied to
> us-east-1, and does not know anything about it (unless you happen to
> have a zonegroup named us-east-1). Maybe there's a way to configure
> the sdk to avoid doing that?
>
> Yehuda
>
> On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:
> > I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to
> talk
> > to my RGW instance using the s3 interface. I am running ceph in docker
> using
> > the ceph/daemon docker images in demo mode. The RGW is started with a
> > zonegroup and zone with their names set to an empty string by the
> scripts in
> > the image.
> >
> > I have ForcePathStyle for the client set to true, because I want to
> access
> > all my buckets using the path: myrgw.instance:8080/somebucket.
> >
> > I noticed that if I set the region for the client to anything other than
> > us-east-1, I get this error when creating a bucket:
> > InvalidLocationConstraint: The specified location-constraint is not
> valid.
> >
> > If I set the region in the client to something made up, such as "ceph"
> and
> > the LocationConstraint to "ceph", I still get the same error.
> >
> > The only way to get my buckets to create successfully is to set the
> client's
> > region to us-east-1. I have grepped the ceph code base and cannot find
> any
> > references to us-east-1. In addition, I looked at the AWS docs for
> > calculating v4 signatures and us-east-1 is the default region but I can
> see
> > that the region string is used in the calculation (i.e. the region is not
> > ignored when calculating the signature if it is set to us-east-1).
> >
> > Why do my buckets create successfully if I set the region in my s3
> client to
> > us-east-1, but not otherwise? If I do not want to use us-east-1 as my
> > default region, for example, if I want us-west-1 as my default region,
> what
> > should I be configuring in ceph?
> >
> > Thanks,
> >
> > Francis
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[David Turner]

Specifically my issue was having problems without this set in the .s3cfg
file. `bucket_location = US`

On Mon, Feb 19, 2018 at 5:04 PM David Turner  wrote:

> I wasn't using the Go SDK.  I was using s3cmd when I came across this.
>
> On Mon, Feb 19, 2018 at 4:42 PM Yehuda Sadeh-Weinraub 
> wrote:
>
>> Sounds like the go sdk adds a location constraint to requests that
>> don't go to us-east-1. RGW itself is definitely isn't tied to
>> us-east-1, and does not know anything about it (unless you happen to
>> have a zonegroup named us-east-1). Maybe there's a way to configure
>> the sdk to avoid doing that?
>>
>> Yehuda
>>
>> On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:
>> > I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to
>> talk
>> > to my RGW instance using the s3 interface. I am running ceph in docker
>> using
>> > the ceph/daemon docker images in demo mode. The RGW is started with a
>> > zonegroup and zone with their names set to an empty string by the
>> scripts in
>> > the image.
>> >
>> > I have ForcePathStyle for the client set to true, because I want to
>> access
>> > all my buckets using the path: myrgw.instance:8080/somebucket.
>> >
>> > I noticed that if I set the region for the client to anything other than
>> > us-east-1, I get this error when creating a bucket:
>> > InvalidLocationConstraint: The specified location-constraint is not
>> valid.
>> >
>> > If I set the region in the client to something made up, such as "ceph"
>> and
>> > the LocationConstraint to "ceph", I still get the same error.
>> >
>> > The only way to get my buckets to create successfully is to set the
>> client's
>> > region to us-east-1. I have grepped the ceph code base and cannot find
>> any
>> > references to us-east-1. In addition, I looked at the AWS docs for
>> > calculating v4 signatures and us-east-1 is the default region but I can
>> see
>> > that the region string is used in the calculation (i.e. the region is
>> not
>> > ignored when calculating the signature if it is set to us-east-1).
>> >
>> > Why do my buckets create successfully if I set the region in my s3
>> client to
>> > us-east-1, but not otherwise? If I do not want to use us-east-1 as my
>> > default region, for example, if I want us-west-1 as my default region,
>> what
>> > should I be configuring in ceph?
>> >
>> > Thanks,
>> >
>> > Francis
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing clones

[Karsten Becker]

Hi Eugen,

hmmm, that should be :

> rbd -p cpVirtualMachines list | while read LINE; do osdmaptool 
> --test-map-object $LINE --pool 10 osdmap 2>&1; rbd snap ls 
> cpVirtualMachines/$LINE | grep -v SNAPID | awk '{ print $2 }' | while read 
> LINE2; do echo "$LINE"; osdmaptool --test-map-object $LINE2 --pool 10 osdmap 
> 2>&1; done; done | less

It's a Proxmox system. There were only two snapshots on the PG, which I
deleted now. Now nothing gets displayed on the PG... is that possible? A
repair still fails unfortunately...

Best & thank you for the hint!
Karsten



On 19.02.2018 22:42, Eugen Block wrote:
>> BTW - how can I find out, which RBDs are affected by this problem. Maybe
>> a copy/remove of the affected RBDs could help? But how to find out to
>> which RBDs this PG belongs to?
> 
> Depending on how many PGs your cluster/pool has, you could dump your
> osdmap and then run the osdmaptool [1] for every rbd object in your pool
> and grep for the affected PG. That would be quick for a few objects, I
> guess:
> 
> ceph1:~ # ceph osd getmap -o /tmp/osdmap
> 
> ceph1:~ # osdmaptool --test-map-object image1 --pool 5 /tmp/osdmap
> osdmaptool: osdmap file '/tmp/osdmap'
>  object 'image1' -> 5.2 -> [0]
> 
> ceph1:~ # osdmaptool --test-map-object image2 --pool 5 /tmp/osdmap
> osdmaptool: osdmap file '/tmp/osdmap'
>  object 'image2' -> 5.f -> [0]
> 
> 
> [1]
> https://www.hastexo.com/resources/hints-and-kinks/which-osd-stores-specific-rados-object/
> 
> 
> Zitat von Karsten Becker :
> 
>> BTW - how can I find out, which RBDs are affected by this problem. Maybe
>> a copy/remove of the affected RBDs could help? But how to find out to
>> which RBDs this PG belongs to?
>>
>> Best
>> Karsten
>>
>> On 19.02.2018 19:26, Karsten Becker wrote:
>>> Hi.
>>>
>>> Thank you for the tip. I just tried... but unfortunately the import
>>> aborts:
>>>
 Write #10:9de96eca:::rbd_data.f5b8603d1b58ba.1d82:head#
 snapset 0=[]:{}
 Write #10:9de973fe:::rbd_data.966489238e1f29.250b:18#
 Write #10:9de973fe:::rbd_data.966489238e1f29.250b:24#
 Write #10:9de973fe:::rbd_data.966489238e1f29.250b:head#
 snapset 628=[24,21,17]:{18=[17],24=[24,21]}
 /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: In function
 'void SnapMapper::add_oid(const hobject_t&, const
 std::set&,
 MapCacher::Transaction,
 ceph::buffer::list>*)' thread 7facba7de400 time 2018-02-19
 19:24:18.917515
 /home/builder/source/ceph-12.2.2/src/osd/SnapMapper.cc: 246: FAILED
 assert(r == -2)
  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf)
 luminous (stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
 const*)+0x102) [0x7facb0c2a8f2]
  2: (SnapMapper::add_oid(hobject_t const&, std::set>>> std::less, std::allocator > const&,
 MapCacher::Transaction>>> std::char_traits, std::allocator >,
 ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
  3: (get_attrs(ObjectStore*, coll_t, ghobject_t,
 ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&,
 SnapMapper&)+0xafb) [0x55eef35f901b]
  4: (ObjectStoreTool::get_object(ObjectStore*, coll_t,
 ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0x738)
 [0x55eef35f9ae8]
  5: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool,
 std::__cxx11::basic_string,
 std::allocator >, ObjectStore::Sequencer&)+0x1135)
 [0x55eef36002f5]
  6: (main()+0x3909) [0x55eef3561349]
  7: (__libc_start_main()+0xf1) [0x7facae0892b1]
  8: (_start()+0x2a) [0x55eef35e901a]
  NOTE: a copy of the executable, or `objdump -rdS ` is
 needed to interpret this.
 *** Caught signal (Aborted) **
  in thread 7facba7de400 thread_name:ceph-objectstor
  ceph version 12.2.2 (215dd7151453fae88e6f968c975b6ce309d42dcf)
 luminous (stable)
  1: (()+0x913f14) [0x55eef3c10f14]
  2: (()+0x110c0) [0x7facaf5020c0]
  3: (gsignal()+0xcf) [0x7facae09bfcf]
  4: (abort()+0x16a) [0x7facae09d3fa]
  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
 const*)+0x28e) [0x7facb0c2aa7e]
  6: (SnapMapper::add_oid(hobject_t const&, std::set>>> std::less, std::allocator > const&,
 MapCacher::Transaction>>> std::char_traits, std::allocator >,
 ceph::buffer::list>*)+0x8e9) [0x55eef3894fe9]
  7: (get_attrs(ObjectStore*, coll_t, ghobject_t,
 ObjectStore::Transaction*, ceph::buffer::list&, OSDriver&,
 SnapMapper&)+0xafb) [0x55eef35f901b]
  8: (ObjectStoreTool::get_object(ObjectStore*, coll_t,
 ceph::buffer::list&, OSDMap&, bool*, ObjectStore::Sequencer&)+0x738)
 [0x55eef35f9ae8]
  9: (ObjectStoreTool::do_import(ObjectStore*, OSDSuperblock&, bool,
 std::__cxx11::basic_string,
 std::allocator >, ObjectStore::Sequencer&)+0x1135)
 [0x55eef36002f5]
  10: (main()+0x3909) [0x55eef3561349]
  11: (__libc_start_main()+0xf1) [0x7facae0892b1]
  12: (_start()+0

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[F21]

I've done some debugging and the LocationConstraint is not being set by 
the SDK by default.


I do, however, need to set the region on the client to us-east-1 for it 
to work. Anything else will return an InvalidLocationConstraint error.


Francis

On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:

Sounds like the go sdk adds a location constraint to requests that
don't go to us-east-1. RGW itself is definitely isn't tied to
us-east-1, and does not know anything about it (unless you happen to
have a zonegroup named us-east-1). Maybe there's a way to configure
the sdk to avoid doing that?

Yehuda

On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:

I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to talk
to my RGW instance using the s3 interface. I am running ceph in docker using
the ceph/daemon docker images in demo mode. The RGW is started with a
zonegroup and zone with their names set to an empty string by the scripts in
the image.

I have ForcePathStyle for the client set to true, because I want to access
all my buckets using the path: myrgw.instance:8080/somebucket.

I noticed that if I set the region for the client to anything other than
us-east-1, I get this error when creating a bucket:
InvalidLocationConstraint: The specified location-constraint is not valid.

If I set the region in the client to something made up, such as "ceph" and
the LocationConstraint to "ceph", I still get the same error.

The only way to get my buckets to create successfully is to set the client's
region to us-east-1. I have grepped the ceph code base and cannot find any
references to us-east-1. In addition, I looked at the AWS docs for
calculating v4 signatures and us-east-1 is the default region but I can see
that the region string is used in the calculation (i.e. the region is not
ignored when calculating the signature if it is set to us-east-1).

Why do my buckets create successfully if I set the region in my s3 client to
us-east-1, but not otherwise? If I do not want to use us-east-1 as my
default region, for example, if I want us-west-1 as my default region, what
should I be configuring in ceph?

Thanks,

Francis

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[Yehuda Sadeh-Weinraub]

What is the name of your zonegroup?

On Mon, Feb 19, 2018 at 3:29 PM, F21  wrote:
> I've done some debugging and the LocationConstraint is not being set by the
> SDK by default.
>
> I do, however, need to set the region on the client to us-east-1 for it to
> work. Anything else will return an InvalidLocationConstraint error.
>
> Francis
>
>
> On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:
>>
>> Sounds like the go sdk adds a location constraint to requests that
>> don't go to us-east-1. RGW itself is definitely isn't tied to
>> us-east-1, and does not know anything about it (unless you happen to
>> have a zonegroup named us-east-1). Maybe there's a way to configure
>> the sdk to avoid doing that?
>>
>> Yehuda
>>
>> On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:
>>>
>>> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to
>>> talk
>>> to my RGW instance using the s3 interface. I am running ceph in docker
>>> using
>>> the ceph/daemon docker images in demo mode. The RGW is started with a
>>> zonegroup and zone with their names set to an empty string by the scripts
>>> in
>>> the image.
>>>
>>> I have ForcePathStyle for the client set to true, because I want to
>>> access
>>> all my buckets using the path: myrgw.instance:8080/somebucket.
>>>
>>> I noticed that if I set the region for the client to anything other than
>>> us-east-1, I get this error when creating a bucket:
>>> InvalidLocationConstraint: The specified location-constraint is not
>>> valid.
>>>
>>> If I set the region in the client to something made up, such as "ceph"
>>> and
>>> the LocationConstraint to "ceph", I still get the same error.
>>>
>>> The only way to get my buckets to create successfully is to set the
>>> client's
>>> region to us-east-1. I have grepped the ceph code base and cannot find
>>> any
>>> references to us-east-1. In addition, I looked at the AWS docs for
>>> calculating v4 signatures and us-east-1 is the default region but I can
>>> see
>>> that the region string is used in the calculation (i.e. the region is not
>>> ignored when calculating the signature if it is set to us-east-1).
>>>
>>> Why do my buckets create successfully if I set the region in my s3 client
>>> to
>>> us-east-1, but not otherwise? If I do not want to use us-east-1 as my
>>> default region, for example, if I want us-west-1 as my default region,
>>> what
>>> should I be configuring in ceph?
>>>
>>> Thanks,
>>>
>>> Francis
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

[F21]

I am using the official ceph/daemon docker image. It starts RGW and 
creates a zonegroup and zone with their names set to an empty string: 
https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/start_rgw.sh#L36:54


$RGW_ZONEGROUP and $RGW_ZONE are both empty strings by default: 
https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/variables_entrypoint.sh#L46


Here's what I get when I query RGW:

$ radosgw-admin zonegroup list
{
    "default_info": "",
    "zonegroups": [
    "default"
    ]
}

$ radosgw-admin zone list
{
    "default_info": "",
    "zones": [
    "default"
    ]
}

On 20/02/2018 10:33 AM, Yehuda Sadeh-Weinraub wrote:

What is the name of your zonegroup?

On Mon, Feb 19, 2018 at 3:29 PM, F21  wrote:

I've done some debugging and the LocationConstraint is not being set by the
SDK by default.

I do, however, need to set the region on the client to us-east-1 for it to
work. Anything else will return an InvalidLocationConstraint error.

Francis


On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:

Sounds like the go sdk adds a location constraint to requests that
don't go to us-east-1. RGW itself is definitely isn't tied to
us-east-1, and does not know anything about it (unless you happen to
have a zonegroup named us-east-1). Maybe there's a way to configure
the sdk to avoid doing that?

Yehuda

On Sun, Feb 18, 2018 at 1:54 PM, F21  wrote:

I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2) to
talk
to my RGW instance using the s3 interface. I am running ceph in docker
using
the ceph/daemon docker images in demo mode. The RGW is started with a
zonegroup and zone with their names set to an empty string by the scripts
in
the image.

I have ForcePathStyle for the client set to true, because I want to
access
all my buckets using the path: myrgw.instance:8080/somebucket.

I noticed that if I set the region for the client to anything other than
us-east-1, I get this error when creating a bucket:
InvalidLocationConstraint: The specified location-constraint is not
valid.

If I set the region in the client to something made up, such as "ceph"
and
the LocationConstraint to "ceph", I still get the same error.

The only way to get my buckets to create successfully is to set the
client's
region to us-east-1. I have grepped the ceph code base and cannot find
any
references to us-east-1. In addition, I looked at the AWS docs for
calculating v4 signatures and us-east-1 is the default region but I can
see
that the region string is used in the calculation (i.e. the region is not
ignored when calculating the signature if it is set to us-east-1).

Why do my buckets create successfully if I set the region in my s3 client
to
us-east-1, but not otherwise? If I do not want to use us-east-1 as my
default region, for example, if I want us-west-1 as my default region,
what
should I be configuring in ceph?

Thanks,

Francis

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw bucket inaccessible - appears to be using incorrect index pool?

[Graham Allan]

Sorry to send another long followup, but actually... I'm not sure how to 
change the placement_rule for a bucket - or at least what I tried does 
not seem to work. Using a different (more disposable) bucket, my attempt 
went like this::


first created a new placement rule "old-placement" in both the zonegroup 
and zone commit new period - this looks ok.



{
"key": "old-placement",
"val": {
"index_pool": ".rgw.buckets",
"data_pool": ".rgw.buckets",
"data_extra_pool": "",
"index_type": 0,
"compression": ""
}



The current metadata for the test bucket looks like this:


{
"key": "bucket.instance:boto-demo-100:default.2170793.10",
"ver": {
"tag": "_BFMtrwjsyFbhU-65IielO3q",
"ver": 652
},
"mtime": "2018-02-14 13:46:37.993218Z",
"data": {
"bucket_info": {
"bucket": {
"name": "boto-demo-100",
"marker": "default.2170793.10",
"bucket_id": "default.2170793.10",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.buckets",
"data_extra_pool": "",
"index_pool": ".rgw.buckets"
}
},
"creation_time": "0.00",
"owner": "xx",
"flags": 2,
"zonegroup": "default",
"placement_rule": "",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"num_shards": 32,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": "",
"index_type": 0,
"mdsearch_config": [],
"reshard_status": 0,
"new_bucket_instance_id": ""
},
"attrs": [
{
"key": "user.rgw.acl",
"val": 
"AgKFAwIXBgAAAGJseW5jaAkAAABCZW4gTHluY2gDA2IBAQYAAABibHluY2gPAQYAAABibHluY2gEAzcCAgQABgAAAGJseW5jagIEDwkAAABCZW4gTHluY2gAAA=="
},
{
"key": "user.rgw.idtag",
"val": ""
}
]
}
}


I edit "placement_rule": to change "" -> "old-placement" and write it 
back using:



radosgw-admin metadata put bucket.instance:boto-demo-100:default.2170793.10 < 
boto-demo-100.json


Now when I run "radosgw-admin bucket list --bucket=boto-demo-100" I am 
getting an empty list, though I'm pretty sure the bucket contains some 
objects.


When I read the bucket instance metadata back again, it still reads 
"placement_rule": "" so I wonder if the bucket_info change is really 
taking effect.


A quick debug session seems to show it still querying the wrong pool 
(100) for the index, so it seems that my attempt to update the 
bucket_info is either failing or incorrect!


Graham

On 02/19/2018 04:00 PM, Graham Allan wrote:

Thanks Robin,

Of the two issues, this seems to me like it must be #22928.

Since the majority of index entries for this bucket are in the 
.rgw.buckets pool, but newer entries have been created in 
.rgw.buckets.index, it's clearly failing to use the explicit placement 
pool - and with the index data split across two pools I don't see how 
resharding could correct this.


I can get the object names from the "new" (incorrect) indexes with 
something like:


for i in `rados -p .rgw.buckets.index ls - | grep 
"default.2049236.2"`; do rados -p .rgw.buckets.index listomapkeys 
$i|grep "^[a-zA-Z0-9]"; done


Fortunately there are only ~20, in this bucket (my grep is just a stupid 
way to skip what I assume are the multipart parts, which have a 
non-ascii first char).


... and these files are downloadable, at least using s3cmd (minio client 
fails, it seems to try and check the index first).


Once I have these newer files downloaded, then to restore access to the 
older index I like the suggestion in the issue tracker to create a new 
placement target in the zone, and modify the bucket's placement rule to 
match. It seems like it might be safer than copying objects from one 
index pool to the other (though the latter certainly sounds faster and 
easier!)


 From a quick check, I suspect I probably have 40 or so other buckets 
with this problem... will need to check them more closely.


Actually it looks like a lot of the affected buckets were created around 
10/2016 - I suspect the placement policies were incorrect for a short 
time due to confusion over the hammer->jewel upgrade (the 
realm/period/zonegroup/zone conversion didn't really go smoothly!)


On 02/16/2018 11:39 PM, Robin H. Johnson wrote

Re: [ceph-users] Newbie question: stretch ceph cluster

[ST Wong (ITSC)]

Hi,

Thanks for your advice.  Will try it out.

Best Regards,
/ST Wong

From: Maged Mokhtar [mailto:mmokh...@petasan.org]
Sent: Wednesday, February 14, 2018 4:20 PM
To: ST Wong (ITSC)
Cc: Luis Periquito; Kai Wagner; Ceph Users
Subject: Re: [ceph-users] Newbie question: stretch ceph cluster


Hi,

You need to set the min_size to 2 in crush rule.

The exact location and replication flow when a client writes data depends on 
the object name and num of pgs. the crush rule determines which osds will serve 
a pg, the first is the primary osd for that pg. The client computes the pg from 
the object name and writes the object to the primary osd for that pg, then 
primary osd is then responsible to replicate with the other osds serving this 
pg. So for the same client, some objects will be sent to datacenter 1 and some 
to 2 and the osds will do the rest.

The other point is regarding how to setup monitors across 2 datacenters and be 
able to function if one goes down, this is tricky since monitors do require an 
odd number and form a quorum. This link my is quite interesting, i am not sure 
if there are better ways to do it:

https://www.sebastien-han.fr/blog/2013/01/28/ceph-geo-replication-sort-of/



Maged

On 2018-02-14 04:12, ST Wong (ITSC) wrote:
Hi,

Thanks for your advice,

-Original Message-
From: ceph-users 
[mailto:ceph-users-boun...@lists.ceph.com]
 On Behalf Of Luis Periquito
Sent: Friday, February 09, 2018 11:34 PM
To: Kai Wagner
Cc: Ceph Users
Subject: Re: [ceph-users] Newbie question: stretch ceph cluster

On Fri, Feb 9, 2018 at 2:59 PM, Kai Wagner 
mailto:kwag...@suse.com>> wrote:
Hi and welcome,


On 09.02.2018 15:46, ST Wong (ITSC) wrote:

Hi, I'm new to CEPH and got a task to setup CEPH with kind of DR feature.
We've 2 10Gb connected data centers in the same campus.I wonder if it's
possible to setup a CEPH cluster with following components in each
data
center:


3 x mon + mds + mgr
In this scenario you wouldn't be any better, as loosing a room means loosing 
half of your cluster. Can you run the MON somewhere else that would be able to 
continue if you loose one of the rooms?

Will it be okay to have 3 x MON per DC so that we still have 3 x MON in case of 
losing 1 DC ?  Or need more in case of double fault - losing 1 DC and failure 
of any MON in remaining DC will make the cluster stop working?



As for MGR and MDS they're (recommended) active/passive; so one per room would 
be enough.

3 x OSD (replicated factor=2, between data center)


replicated with size=2 is a bad idea. You can have size=4 and
min_size=2 and have a crush map with rules something like:


rule crosssite {
id 0
type replicated
min_size 4
max_size 4
step take default
step choose firstn 2 type room
step chooseleaf firstn 2 type host
step emit
}


this will store 4 copies, 2 in different hosts and 2 different rooms.

Does it mean for new data write to hostA:roomA, replication will take place as 
following?
1. from hostA:roomA to hostB:roomA
2. from hostA:roomA to hostA, roomB
3. from hostB:roomA to hostB, roomB

If it works in this way, can copy in 3 be skipped so that for each piece of 
data, there are 3 replicas - original one, replica in same room, and replica in 
other room, in order to save some space?

Besides, would also like to ask if it's correct that the cluster will continue 
to work (degraded) if one room is lost?

Will there be any better way to setup such 'stretched' cluster between 2 DCs?  
They're extension instead of real DR site...

Sorry for the newbie questions and we'll proceed to have more study and 
experiment on this.

Thanks a lot.







So that any one of following failure won't affect the cluster's
operation and data availability:

any one component in either data center failure of either one of the
data center


Is it possible?

In general this is possible, but I would consider that replica=2 is
not a good idea. In case of a failure scenario or just maintenance and
one DC is powered off and just one single disk fails on the other DC,
this can already lead to data loss. My advice here would be, if anyhow
possible, please don't do replica=2.

In case one data center failure case, seems replication can't occur any
more.   Any CRUSH rule can achieve this purpose?


Sorry for the newbie question.


Thanks a lot.

Regards

/st wong





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB
21284 (AG Nürnberg)


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@list

Re: [ceph-users] Luminous : performance degrade while read operations (ceph-volume)

[nokia ceph]

Hi Alfredo Deza,

We have 5 node platforms with lvm osd created from scratch and another 5
node platform migrated from kraken which is ceph volume simple. Both has
same issue . Both platform has only hdd for osd.

We also noticed 2 times disk iops more compare to kraken , this causes less
read performance. During rocksdb compaction the situation is worse.


Meanwhile we are building another platform creating osd using ceph-disk and
analyse on this.

Thanks,
Muthu



On Tuesday, February 20, 2018, Alfredo Deza  wrote:

>
>
> On Mon, Feb 19, 2018 at 2:01 PM, nokia ceph 
> wrote:
>
>> Hi All,
>>
>> We have 5 node clusters with EC 4+1 and use bluestore since last year
>> from Kraken.
>> Recently we migrated all our platforms to luminous 12.2.2 and finally all
>> OSDs migrated to ceph-volume simple type and on few platforms installed
>> ceph using ceph-volume .
>>
>> Now we see two times more traffic in read compare to client traffic on
>> migrated platform and newly created platforms . This was not the case in
>> older releases where ceph status read B/W will be same as client read
>> traffic.
>>
>> Some network graphs :
>>
>> *Client network interface* towards ceph public interface : shows
>> *4.3Gbps* read
>>
>>
>> [image: Inline image 2]
>>
>> *Ceph Node Public interface* : Each node around 960Mbps * 5 node =* 4.6
>> Gbps *- this matches.
>> [image: Inline image 3]
>>
>> Ceph status output : show  1032 MB/s =* 8.06 Gbps*
>>
>> cn6.chn6us1c1.cdn ~# ceph status
>>   cluster:
>> id: abda22db-3658-4d33-9681-e3ff10690f88
>> health: HEALTH_OK
>>
>>   services:
>> mon: 5 daemons, quorum cn6,cn7,cn8,cn9,cn10
>> mgr: cn6(active), standbys: cn7, cn9, cn10, cn8
>> osd: 340 osds: 340 up, 340 in
>>
>>   data:
>> pools:   1 pools, 8192 pgs
>> objects: 270M objects, 426 TB
>> usage:   581 TB used, 655 TB / 1237 TB avail
>> pgs: 8160 active+clean
>>  32   active+clean+scrubbing
>>
>>   io:
>> client:   *1032 MB/s rd*, 168 MB/s wr, 1908 op/s rd, 1594 op/s wr
>>
>>
>> Write operation we don't see this issue. Client traffic and this matches.
>> Is this expected behavior in Luminous and ceph-volume lvm or a bug ?
>> Wrong calculation in ceph status read B/W ?
>>
>
> You mentioned `ceph-volume simple` but here you say lvm. With LVM
> ceph-volume will create the OSDs from scratch, while "simple" will keep
> whatever OSD was created before.
>
> Have you created the OSDs from scratch with ceph-volume? or is it just
> using "simple" , managing a previously deployed OSD?
>
>>
>> Please provide your feedback.
>>
>> Thanks,
>> Muthu
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw bucket inaccessible - appears to be using incorrect index pool?

[Robin H. Johnson]

On Mon, Feb 19, 2018 at 07:57:18PM -0600, Graham Allan wrote:
> Sorry to send another long followup, but actually... I'm not sure how to 
> change the placement_rule for a bucket - or at least what I tried does 
> not seem to work. Using a different (more disposable) bucket, my attempt 
> went like this::
[snip]

> first created a new placement rule "old-placement" in both the zonegroup 
> and zone commit new period - this looks ok.
...
> I edit "placement_rule": to change "" -> "old-placement" and write it 
> back using:
> 
> > radosgw-admin metadata put bucket.instance:boto-demo-100:default.2170793.10 
> > < boto-demo-100.json
> 
> Now when I run "radosgw-admin bucket list --bucket=boto-demo-100" I am 
> getting an empty list, though I'm pretty sure the bucket contains some 
> objects.
> 
> When I read the bucket instance metadata back again, it still reads 
> "placement_rule": "" so I wonder if the bucket_info change is really 
> taking effect.
So it never showed the new placement_rule if you did a get after the
put?

> A quick debug session seems to show it still querying the wrong pool 
> (100) for the index, so it seems that my attempt to update the 
> bucket_info is either failing or incorrect!
Did you run a local build w/ the linked patch? I think that would have
more effect than 

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mgr[influx] Cannot transmit statistics: influxdb python module not found.

[knawnd]

Marc Roos wrote on 13/02/18 00:50:

why not use collectd? centos7 rpms  should do fine.
Marc, sorry I somehow missed your question. One of the reason could be that collectd is a additional 
daemon whereas influx plugin for ceph is just an additional part of the already running system (ceph).



On Feb 12, 2018 9:50 PM, Benjeman Meekhof  wrote:


In our case I think we grabbed the SRPM from Fedora and rebuilt it on
Scientific Linux (another RHEL derivative).  Presumably the binary
didn't work or I would have installed it directly.  I'm not quite sure
why it hasn't migrated to EPEL yet.

I haven't tried the SRPM for latest releases, we're actually quite far
behind the current python-influx version since I built it a while back
but if I were you I'd grab whatever SRPM gets you the latest
python-influxdb release and give it a try.

http://rpmfind.net/linux/rpm2html/search.php?query=python-influxdb

thanks,
Ben

On Mon, Feb 12, 2018 at 11:03 AM,   wrote:

Dear all,

I'd like to store ceph luminous metrics into influxdb. It seems like influx
plugin has been already backported for lumious:
rpm -ql ceph-mgr-12.2.2-0.el7.x86_64|grep -i influx
/usr/lib64/ceph/mgr/influx
/usr/lib64/ceph/mgr/influx/__init__.py
/usr/lib64/ceph/mgr/influx/__init__.pyc
/usr/lib64/ceph/mgr/influx/__init__.pyo
/usr/lib64/ceph/mgr/influx/module.py
/usr/lib64/ceph/mgr/influx/module.pyc
/usr/lib64/ceph/mgr/influx/module.pyo

So following http://docs.ceph.com/docs/master/mgr/influx/ doc I enabled
influx plugin by executing the following command on mgr node:
ceph mgr module enable influx

but in ceph log I see the following error:
2018-02-12 15:51:31.241854 7f95e7942600  0 ceph version 12.2.2
(cf0baba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process
(unknown), pid 96425
[]
2018-02-12 15:51:31.422414 7f95dec29700  1 mgr init Loading python module
'influx'
[]
2018-02-12 15:51:32.227206 7f95c36ec700  1 mgr load Constructed class from
module: influx
[]
2018-02-12 15:51:32.228163 7f95c0ee7700  0 mgr[influx] Cannot transmit
statistics: influxdb python module not found.  Did you install it?

Indeed there is no python-influxdb module install on my mgr node (CentOS 7
x64) but yum search can't find it with the following repos enabled:
repo id
repo name   status
Ceph/x86_64
Ceph packages for x86_64
Ceph-noarch
Ceph noarch packages
base/7/x86_64 CentOS-7 - Base
ceph-source
Ceph source packages
epel/x86_64
Extra Packages for Enterprise Linux 7 - x86_64
extras/7/x86_64 CentOS-7 - Extras
updates/7/x86_64 CentOS-7 - Updates

Python version is 2.7.5.

Is 'pip install' the only way to go or there is still some option to have
required python module via rpm? I wonder how other people deals with that
issue?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph df: Raw used vs. used vs. actual bytes in cephfs

[Flemming Frandsen]

I didn't know about ceph df detail, that's quite useful, thanks.

I was thinking that the problem had to do with some sort of internal 
fragmentation, because the filesystem in question does have millions 
(2.9 M or threabouts) of files, however, even if 4k is lost for each 
file, that only amounts to about 23 GB of raw space lost and I have 3276 
GB of raw space unaccounted for.


I've researched the min alloc option a bit and even though no 
documentation seems to exist, I've found that the default is 64k for 
hdd, but even if the lost space per file is 64k and that's mirrored, I 
can only account for 371 GB, so that doesn't really help a great deal.


I have set up an experimental cluster with "bluestore min alloc size = 
4096" and so far I've been unable to make it lose space like the first 
cluster.



I'm very worried that ceph is unusable because of this issue.



On 19/02/18 19:38, Pavel Shub wrote:

Could you be running into block size (minimum allocation unit)
overhead? Default bluestore block size is 4k for hdd and 64k for ssd.
This is exacerbated if you have tons of small files. I tend to see
this when "ceph df detail" sum of raw used in pools is less than the
global raw bytes used.

On Mon, Feb 19, 2018 at 2:09 AM, Flemming Frandsen
 wrote:

Each OSD lives on a separate HDD in bluestore with the journals on 2GB
partitions on a shared SSD.


On 16/02/18 21:08, Gregory Farnum wrote:

What does the cluster deployment look like? Usually this happens when you’re
sharing disks with the OS, or have co-located file journals or something.
On Fri, Feb 16, 2018 at 4:02 AM Flemming Frandsen
 wrote:

I'm trying out cephfs and I'm in the process of copying over some
real-world data to see what happens.

I have created a number of cephfs file systems, the only one I've
started working on is the one called jenkins specifically the one named
jenkins which lives in fs_jenkins_data and fs_jenkins_metadata.

According to ceph df I have about 1387 GB of data in all of the pools,
while the raw used space is 5918 GB, which gives a ratio of about 4.3, I
would have expected a ratio around 2 as the pool size has been set to 2.


Can anyone explain where half my space has been squandered?

  > ceph df
GLOBAL:
  SIZE  AVAIL RAW USED %RAW USED
  8382G 2463G5918G 70.61
POOLS:
  NAME ID USED   %USED MAX
AVAIL OBJECTS
  .rgw.root11113 0 258G
4
  default.rgw.control  2   0 0 258G
8
  default.rgw.meta 3   0 0 258G
0
  default.rgw.log  4   0 0 258G
207
  fs_docker-nexus_data 5  66120M 11.09 258G
22655
  fs_docker-nexus_metadata 6  39463k 0 258G
2376
  fs_meta_data 7 330 0 258G
4
  fs_meta_metadata 8567k 0 258G
22
  fs_jenkins_data  9   1321G 71.84 258G
28576278
  fs_jenkins_metadata  10 52178k 0 258G
2285493
  fs_nexus_data11  0 0 258G
0
  fs_nexus_metadata12   4181 0 258G
21

--
   Regards Flemming Frandsen - Stibo Systems - DK - STEP Release Manager
   Please use rele...@stibo.com for all Release Management requests

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
  Regards Flemming Frandsen - Stibo Systems - DK - STEP Release Manager
  Please use rele...@stibo.com for all Release Management requests


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
 Regards Flemming Frandsen - Stibo Systems - DK - STEP Release Manager
 Please use rele...@stibo.com for all Release Management requests

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com