Re: [ceph-users] RGW multisite replication failures

[Ben Morrice]

Hello Orit,

Yes, this bug looks to correlate. Was this included in 10.2.3?

I guess not as I have since updated to 10.2.3 but getting the same errors

This bug talks about not retrying after a failure, however do you know
why the sync fails in the first place? It seems that basically any
object over 500k in size fails :(

Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL ENT CBS BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 23/09/16 16:52, Orit Wasserman wrote:
> Hi Ben,
> It seems to be http://tracker.ceph.com/issues/16742.
> It is being backported to jewel http://tracker.ceph.com/issues/16794,
> you can try apply it and see if it helps you.
>
> Regards,
> Orit
>
> On Fri, Sep 23, 2016 at 9:21 AM, Ben Morrice  wrote:
>> Hello all,
>>
>> I have two separate ceph (10.2.2) clusters and have configured multisite
>> replication between the two. I can see some buckets get synced, however
>> others do not.
>>
>> Both clusters are RHEL7, and I have upgraded libcurl from 7.29 to 7.50
>> (to avoid http://tracker.ceph.com/issues/15915).
>>
>> Below is some debug output on the 'secondary' zone (bbp-gva-secondary)
>> after uploading a file to the bucket 'bentest1' from onto the master
>> zone (bbp-gva-master).
>>
>> This appears to to be happening very frequently. The size of my bucket
>> pool in the master is ~120GB, however on the secondary site it's only
>> 5GB so things are not very happy at the moment.
>>
>> What steps can I take to work out why RGW cannot create a lock in the
>> log pool?
>>
>> Is there a way to force a full sync, starting fresh (the secondary site
>> is not advertised to users, thus it's okay to even clean pools to start
>> again)?
>>
>>
>> 2016-09-23 09:03:28.498292 7f992e664700 20 execute(): read data:
>> [{"key":6,"val":["bentest1:bbp-gva-master.85732351.16:-1"]}]
>> 2016-09-23 09:03:28.498453 7f992e664700 20 execute(): modified
>> key=bentest1:bbp-gva-master.85732351.16:-1
>> 2016-09-23 09:03:28.498456 7f992e664700 20 wakeup_data_sync_shards:
>> source_zone=bbp-gva-master,
>> shard_ids={6=bentest1:bbp-gva-master.85732351.16:-1}
>> 2016-09-23 09:03:28.498547 7f9a72ffd700 20 incremental_sync(): async
>> update notification: bentest1:bbp-gva-master.85732351.16:-1
>> 2016-09-23 09:03:28.499137 7f9a7dffb700 20 get_system_obj_state:
>> rctx=0x7f9a3c5f8e08
>> obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16
>> state=0x7f9a0c069848 s->prefetch_data=0
>> 2016-09-23 09:03:28.501379 7f9a72ffd700 20 operate(): sync status for
>> bucket bentest1:bbp-gva-master.85732351.16:-1: 2
>> 2016-09-23 09:03:28.501433 7f9a877fe700 20 reading from
>> .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
>> 2016-09-23 09:03:28.501447 7f9a877fe700 20 get_system_obj_state:
>> rctx=0x7f9a877fc6d0
>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
>> state=0x7f9a340cfbe8 s->prefetch_data=0
>> 2016-09-23 09:03:28.503269 7f9a877fe700 20 get_system_obj_state:
>> rctx=0x7f9a877fc6d0
>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16
>> state=0x7f9a340cfbe8 s->prefetch_data=0
>> 2016-09-23 09:03:28.510428 7f9a72ffd700 20 sending request to
>> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=034.4578.3&type=bucket-index&rgwx-zonegroup=bbp-gva
>> 2016-09-23 09:03:28.625755 7f9a72ffd700 20 [inc sync] skipping object:
>> bentest1:bbp-gva-master.85732351.16:-1/1m: non-complete operation
>> 2016-09-23 09:03:28.625759 7f9a72ffd700 20 [inc sync] syncing object:
>> bentest1:bbp-gva-master.85732351.16:-1/1m
>> 2016-09-23 09:03:28.625831 7f9a72ffd700 20 bucket sync single entry
>> (source_zone=bbp-gva-master)
>> b=bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16]):-1/1m[0]
>> log_entry=036.4586.3 op=0 op_state=1
>> 2016-09-23 09:03:28.625857 7f9a72ffd700  5 bucket sync: sync obj:
>> bbp-gva-master/bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16])/1m[0]
>> 2016-09-23 09:03:28.626092 7f9a85ffb700 20 get_obj_state:
>> rctx=0x7f9a85ff96a0 obj=bentest1:1m state=0x7f9a30051cf8 s->prefetch_data=0
>> 2016-09-23 09:03:28.626119 7f9a72ffd700 20 sending request to
>> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=036.4586.3&type=bucket-index&rgwx-zonegroup=bbp-gva
>> 2016-09-23 09:03:28.627560 7f9a85ffb700 10 get_canon_resource():
>> dest=/bentest1/1m
>> /bentest1/1m
>> 2016-09-23 09:03:28.627612 7f9a85ffb700 20 sending request to
>> https://bbpobjectstorage.epfl.ch:443/bentest1/1m?rgwx-zonegroup=bbp-gva&rgwx-prepend-metada

Re: [ceph-users] How to maintain cluster properly (Part2)

[Eugen Block]

Thanks for your suggestions, but I managed it without removing OSDs.

Coming back to the office today I found ceph still in error state, but  
the number of inconsistent PGs seemed to be stable at 22. So I started  
all over with the manual repair (grepped log files for PG, searched  
the respective data in the healthy OSDs, removed faulty PGs or touched  
missing data files, then ran ceph pg repair again) and one after  
another came back to active + clean state.


Maybe I should note that we're running only with a replication size of  
2, I believe this caused one of the problems as ceph didn't know from  
which OSD it should recover missing data. Please correct me if I'm  
wrong. For production use we should probably increase to a rep size of  
3, I guess.


Regards
Eugen


Zitat von lyt_yudi :


在 2016年9月26日，下午10:44，Eugen Block  写道：

What I have tried is to manually repair single PGs as described in  
[1]. But some of the broken PGs have no entries in the log file so  
I don't have anything to look at.
In case there is one object in one OSD but is missing in the other.  
how do I get that copied back there? Everything I've tried so far  
didn't accomplish anything except the decreasing number of scrub  
errors, but they are increasing again, so no success at all.


met before in my case, no error in the log too,

like this back to normal:

1. offline the OSD(with inconsistent pgid in the osd) and marked as  
out, waiting for the completion of data synchronization, and then  
delete the OSD


2. manual repair again, waiting for a period of time, the cluster  
can be returned to normal


3. in the end, then re join the deletion of the OSD.

Good Luck!




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multi-site replication issues

[Orit Wasserman]

see comment below

On Mon, Sep 26, 2016 at 10:00 PM, John Rowe  wrote:
> Hi Orit,
>
> Sure thing, please see below.
>
> Thanks!
>
>
> DFW (Primary)
> radosgw-admin zonegroupmap get
> {
> "zonegroups": [
> {
> "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "val": {
> "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "name": "us",
> "api_name": "us",
> "is_master": "true",
> "endpoints": [
> "http:\/\/ELB_FQDN:80"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "zones": [
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "endpoints": [
> "http:\/\/cephrgw1-1:80"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "name": "us-dfw",
> "endpoints": [
> "http:\/\/cephrgw1-1-dfw:80"
> ],
> "log_meta": "true",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
> }
> ],
> "master_zonegroup": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
>  "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
>
> PHX (Secondary)
> {
> "zonegroups": [
> {
> "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "val": {
> "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "name": "us",
> "api_name": "us",
> "is_master": "true",
> "endpoints": [
> "http:\/\/ELB_FQDN:80"
> ],
> "hostnames": [],
> "hostnames_s3website": [],
> "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "zones": [
> {
> "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> "name": "us-phx",
> "endpoints": [
> "http:\/\/cephrgw1-1-phx:80"
> ],
> "log_meta": "false",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> },
> {
> "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> "name": "us-dfw",
> "endpoints": [
> "http:\/\/cephrgw1-1-dfw:80"
> ],
> "log_meta": "true",
> "log_data": "true",
> "bucket_index_max_shards": 0,
> "read_only": "false"
> }
> ],
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> }
> ],
> "default_placement": "default-placement",
> "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> }
> }
> ],
> "master_zonegroup": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> "bucket_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "max_size_kb": -1,
> "max_objects": -1
> }
> }
>
>
> Realms and Periods:
> DFW (Primary)
> rgw1-1-dfw
> radosgw-admin realm get
> {
> "id": "3af93a86-916a-490f-b38f-17922b472b19",
> "name": "my_realm",
> "current_period": "75e5f8df-5d53-4ac9-b87d-625eee5455c1",
> "epoch": 2
> }
> radosgw-admin period get
> {
> "id": "75e5f8df-5d53-4ac9-b87d-6

[ceph-users] Does the journal of a single OSD roll itself automatically?

[xxhdx1985126]

Hi, everyone.


After the file system synchronization, does OSD delete those journals that 
corresponds to operations before the synchronization point?___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Is it possible to recover the data of which all replicas are lost?

[xxhdx1985126]

Hi, everyone.


I've got a problem, here. Due to some miss operations, I deleted all three 
replicas of my data, is there any way to recover it?
This is a very urgent problem.


Please help me, Thanks.___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] how to trigger ms_Handle_reset on monitor

[agung Laksono]

Hi Ceph user,

I am learning bugs in ceph and am trying to reproduce
http://tracker.ceph.com/issues/15113 on my local machine.

On the description, I may see the scenario:

   1. ___*_*__ SafeTimer::timer_thread(), with mon_lock:
   2. ___*_*__ elector: in win_election(), it
   resend_routed_requests(), and collects the routed requests
   3. msgr: in ms_handle_reset(), it reset the session
   4. msgr: it waits for the lock
   5. ___*_*__ elector: in win_election(), it
   handle_command(), but the session is reset, hence it panics.
   6. msgr: remove session, and erase related requests from
   Monitor::routed_requests.


I could understand the whole flow of the bug except ms_handle_reset().
What's a thing that triggers this ms_handle_reset()?
I could not see this function is called in any part of the monitor class.
I've tried to reproduce this on a cluster with 3 MONs and 4 OSDs
and put the printf whenever this function's called. However, It seems
arbitrary. Perhaps anyone can help me to explain this?

thanks

-- 
Cheers,

Agung Laksono
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multi-site replication issues

[John Rowe]

Hi Orit,

That was my failed attempt at sanitizing :)

They are actually all identical:

Periods:
MD5 (cephrgw1-1-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
MD5 (cephrgw1-1-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
MD5 (cephrgw1-2-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
MD5 (cephrgw1-2-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
MD5 (cephrgw1-3-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
MD5 (cephrgw1-3-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d

Realms:
MD5 (cephrgw1-1-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
MD5 (cephrgw1-1-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
MD5 (cephrgw1-2-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
MD5 (cephrgw1-2-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
MD5 (cephrgw1-3-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
MD5 (cephrgw1-3-phx-realm.json) = 39a4e63bab64ed756961117d3629b109


On Tue, Sep 27, 2016 at 5:32 AM Orit Wasserman  wrote:

> see comment below
>
> On Mon, Sep 26, 2016 at 10:00 PM, John Rowe 
> wrote:
> > Hi Orit,
> >
> > Sure thing, please see below.
> >
> > Thanks!
> >
> >
> > DFW (Primary)
> > radosgw-admin zonegroupmap get
> > {
> > "zonegroups": [
> > {
> > "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> > "val": {
> > "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> > "name": "us",
> > "api_name": "us",
> > "is_master": "true",
> > "endpoints": [
> > "http:\/\/ELB_FQDN:80"
> > ],
> > "hostnames": [],
> > "hostnames_s3website": [],
> > "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> > "zones": [
> > {
> > "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> > "name": "us-phx",
> > "endpoints": [
> > "http:\/\/cephrgw1-1:80"
> > ],
> > "log_meta": "false",
> > "log_data": "true",
> > "bucket_index_max_shards": 0,
> > "read_only": "false"
> > },
> > {
> > "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> > "name": "us-dfw",
> > "endpoints": [
> > "http:\/\/cephrgw1-1-dfw:80"
> > ],
> > "log_meta": "true",
> > "log_data": "true",
> > "bucket_index_max_shards": 0,
> > "read_only": "false"
> > }
> > ],
> > "placement_targets": [
> > {
> > "name": "default-placement",
> > "tags": []
> > }
> > ],
> > "default_placement": "default-placement",
> > "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
> > }
> > }
> > ],
> > "master_zonegroup": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> > "bucket_quota": {
> > "enabled": false,
> > "max_size_kb": -1,
> > "max_objects": -1
> > },
> >  "user_quota": {
> > "enabled": false,
> > "max_size_kb": -1,
> > "max_objects": -1
> > }
> > }
> >
> > PHX (Secondary)
> > {
> > "zonegroups": [
> > {
> > "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> > "val": {
> > "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
> > "name": "us",
> > "api_name": "us",
> > "is_master": "true",
> > "endpoints": [
> > "http:\/\/ELB_FQDN:80"
> > ],
> > "hostnames": [],
> > "hostnames_s3website": [],
> > "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> > "zones": [
> > {
> > "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
> > "name": "us-phx",
> > "endpoints": [
> > "http:\/\/cephrgw1-1-phx:80"
> > ],
> > "log_meta": "false",
> > "log_data": "true",
> > "bucket_index_max_shards": 0,
> > "read_only": "false"
> > },
> > {
> > "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
> > "name": "us-dfw",
> > "endpoints": [
> > "http:\/\/cephrgw1-1-dfw:80"
> > ],
> > "log_m

[ceph-users] Mount Cephfs subtree

[mayqui . quintana]

How to mount a cephfs using subtree ??   mount.ceph mon0:/web/data 
/data/ -o name=admin,secretfile=/etc/ceph/admin.secret Receive mount 
error 2 = No such file or directory.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] fixing zones

[Michael Parson]

(I tried to start this discussion on irc, but I wound up with the wrong
paste buffer and wound up getting kicked off for a paste flood, sorry,
that was on me :(  )

We were having some weirdness with our Ceph and did an upgrade up to
10.2.3, which fixed some, but not all of our problems.

It looked like our users pool might have been corrupt, so we moved it
aside and created a new set:

$ sudo ceph osd pool rename .users old.users
$ sudo ceph osd pool rename .users.email old.users.email
$ sudo ceph osd pool rename .users.swift old.users.swift
$ sudo ceph osd pool rename .users.uid old.users.uid


$ sudo ceph osd pool create .users 16 16
$ sudo ceph osd pool create .users.email 16 16
$ sudo ceph osd pool create .users.swift 16 16
$ sudo ceph osd pool create .users.uid 16 16

This allowed me to create new users and swift subusers under them, but
only the first one is allowing auth, all others are getting 403s when
attempting to auth.

We googled around a bit and found the fix-zone script:

https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone

Which ran fine until the last command, which errors out with:

+ radosgw-admin zone default --rgw-zone=default
WARNING: failed to initialize zonegroup

the 'default' rgw-zone seems OK:

$ sudo radosgw-admin zone get --zone-id=default
{
"id": "default",
"name": "default",
"domain_root": ".rgw_",
"control_pool": ".rgw.control_",
"gc_pool": ".rgw.gc_",
"log_pool": ".log_",
"intent_log_pool": ".intent-log_",
"usage_log_pool": ".usage_",
"user_keys_pool": ".users_",
"user_email_pool": ".users.email_",
"user_swift_pool": ".users.swift_",
"user_uid_pool": ".users.uid_",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": ".rgw.buckets.index_",
"data_pool": ".rgw.buckets_",
"data_extra_pool": ".rgw.buckets.extra_",
"index_type": 0
}
}
],
"metadata_heap": ".rgw.meta",
"realm_id": "a113de3d-c506-4112-b419-0d5c94ded7af"
}

Not really sure where to go from here, any help would be appreciated.

--
Michael Parson
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rgw multi-site replication issues

[John Rowe]

Hi Orit,

It appears it must have been one of the known bugs in 10.2.2.  I just
upgraded to 10.2.3 and bi-directional syncing now works.

I am still seeing some errors when I run synch-related commands but they
don't seem to be affecting operations as of yet:

radosgw-admin sync status
2016-09-27 16:17:15.270331 7fe5e83ad9c0  0 error in read_id for id  : (2)
No such file or directory
2016-09-27 16:17:15.270883 7fe5e83ad9c0  0 error in read_id for id  : (2)
No such file or directory
  realm 3af93a86-916a-490f-b38f-17922b472b19 (my_realm)
  zonegroup 235b010c-22e2-4b43-8fcc-8ae01939273e (us)
   zone 58aa3eef-fc1f-492c-a08e-9c6019e7c266 (us-phx)
  metadata sync preparing for full sync
full sync: 0/64 shards
metadata is caught up with master
incremental sync: 64/64 shards
  data sync source: 6c830b44-4e39-4e19-9bd8-03c37c2021f2 (us-dfw)
preparing for full sync
full sync: 18/128 shards
full sync: 0 buckets to sync
incremental sync: 110/128 shards
data is behind on 20 shards
oldest incremental change not applied: 2016-09-27
16:17:08.0.922757s


I've also verified all of the existing data within each bucket has sync'd
over as well


On Tue, Sep 27, 2016 at 9:21 AM John Rowe  wrote:

> Hi Orit,
>
> That was my failed attempt at sanitizing :)
>
> They are actually all identical:
>
> Periods:
> MD5 (cephrgw1-1-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
> MD5 (cephrgw1-1-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
> MD5 (cephrgw1-2-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
> MD5 (cephrgw1-2-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
> MD5 (cephrgw1-3-dfw-period.json) = 12ed481381c1f2937a27b57db0473d6d
> MD5 (cephrgw1-3-phx-period.json) = 12ed481381c1f2937a27b57db0473d6d
>
> Realms:
> MD5 (cephrgw1-1-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
> MD5 (cephrgw1-1-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
> MD5 (cephrgw1-2-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
> MD5 (cephrgw1-2-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
> MD5 (cephrgw1-3-dfw-realm.json) = 39a4e63bab64ed756961117d3629b109
> MD5 (cephrgw1-3-phx-realm.json) = 39a4e63bab64ed756961117d3629b109
>
>
> On Tue, Sep 27, 2016 at 5:32 AM Orit Wasserman 
> wrote:
>
>> see comment below
>>
>> On Mon, Sep 26, 2016 at 10:00 PM, John Rowe 
>> wrote:
>> > Hi Orit,
>> >
>> > Sure thing, please see below.
>> >
>> > Thanks!
>> >
>> >
>> > DFW (Primary)
>> > radosgw-admin zonegroupmap get
>> > {
>> > "zonegroups": [
>> > {
>> > "key": "235b010c-22e2-4b43-8fcc-8ae01939273e",
>> > "val": {
>> > "id": "235b010c-22e2-4b43-8fcc-8ae01939273e",
>> > "name": "us",
>> > "api_name": "us",
>> > "is_master": "true",
>> > "endpoints": [
>> > "http:\/\/ELB_FQDN:80"
>> > ],
>> > "hostnames": [],
>> > "hostnames_s3website": [],
>> > "master_zone": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
>> > "zones": [
>> > {
>> > "id": "58aa3eef-fc1f-492c-a08e-9c6019e7c266",
>> > "name": "us-phx",
>> > "endpoints": [
>> > "http:\/\/cephrgw1-1:80"
>> > ],
>> > "log_meta": "false",
>> > "log_data": "true",
>> > "bucket_index_max_shards": 0,
>> > "read_only": "false"
>> > },
>> > {
>> > "id": "6c830b44-4e39-4e19-9bd8-03c37c2021f2",
>> > "name": "us-dfw",
>> > "endpoints": [
>> > "http:\/\/cephrgw1-1-dfw:80"
>> > ],
>> > "log_meta": "true",
>> > "log_data": "true",
>> > "bucket_index_max_shards": 0,
>> > "read_only": "false"
>> > }
>> > ],
>> > "placement_targets": [
>> > {
>> > "name": "default-placement",
>> > "tags": []
>> > }
>> > ],
>> > "default_placement": "default-placement",
>> > "realm_id": "3af93a86-916a-490f-b38f-17922b472b19"
>> > }
>> > }
>> > ],
>> > "master_zonegroup": "235b010c-22e2-4b43-8fcc-8ae01939273e",
>> > "bucket_quota": {
>> > "enabled": false,
>> > "max_size_kb": -1,
>> > "max_objects": -1
>> > },
>> >  "user_quota": {
>> > "enabled": false,
>> >   

[ceph-users] Adding new monitors to production cluster

[Nick @ Deltaband]

Hi Cephers,

We need to add two new monitors to a production cluster (0.94.9) which has
3 existing monitors. It looks like it's as easy as ceph-deploy mon add .

What's the best practice in terms of when to update the existing monitor
and osd ceph.conf file to include the new monitors in mon_initial_members
and mon_hosts? Before adding the new monitor, afterwards, or doesn't it
make any difference?

Will it cause any disruption on the cluster, or is it 100% safe to do with
no disruption? Any steps we can take to minimise risk?

Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mount Cephfs subtree

[Goncalo Borges]

Hi...

Here are a couple of examples based on what we do.

We have not tried all of them so some are just extrapolations based on 
the underlying logic. Maybe someone else can correct me if I am wrong. I 
the following examples, the root of the filesystem is /cephfs


- Mounting your cephfs filesystem as /cephfs:

   # mount -t ceph :/  /cephfs -o name=admin,secretfile=
   # df -h
   (...)
   192.231.127.8:6789:/   87T  1.4G   87T   1% /cephfs

- Mounting your /cephfs filesystem as /mydata/cephfs (which must exit) then

   # mount -t ceph :/cephfs  /mydata/cephfs -o
   name=admin,secretfile=

- Mounting the subtree /cephf/mydir under, for example, 
/mydata/cephfs/mydir/   (which must exit)  then


   # mount -t ceph :/cephfs/mydir /mydata/cephfs/mydir/ -o
   name=admin,secretfile=

- Mounting the subtree /cephf/mydir under, for example, /mydata/mydir/   
(which must exit)  then


   # mount -t ceph :/cephfs/mydir  /mydata/mydir/ -o
   name=admin,secretfile=

I would also would advise you to create a special user / client to mount 
and not use the admin key. Permissions for that user should be something 
like



client.mount_user
key: <...>
caps: [mds] allow
caps: [mon] allow r
caps: [osd] allow rw pool=

Cheers
Goncalo



On 09/28/2016 01:36 AM, mayqui.quint...@ciqa.info wrote:
How to mount a cephfs using subtree ??   mount.ceph mon0:/web/data 
/data/ -o name=admin,secretfile=/etc/ceph/admin.secret Receive mount 
error 2 = No such file or directory.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: Ceph user manangerment question

[卢 迪]

Hi Daleep,



Thank you for reply.

I have read the document for a moment. Let me try to clarify this.



In my case, I only assgin “mon ‘allow r” permission to account appuser. But, I 
still can mount cephfs and see the directory created before(the folder name is 
“test”).


And, I can create a folder under this folder too. (the folder is “test2”)

However, when I created and edited an text file(“test.txt”) with a read only 
error.When I quit with "q!", I still see the file with 0 bytes.

 [cid:b065595b-09a9-49d0-8e3a-c50bc25cccd5]

I'm wondering I must misunderstand something. I thought I shouldn't see this 
folder "test" because the user didn't have the read/write permission against 
any pool in this cluster. I shouldn't create the "test.txt" in this folder too 
because of premission.(But, I CREATED it with nothing)



Let's say assigning an OS user permission(for example, Linux). I have to give 
read permission if a user want to read a file; If it has to execute a script, I 
have to grant the exeucte permission. I want to understand when and why I 
should assign which permssion to an user by meeting a special task. Can I find 
this kind of document?



Thanks,

Dillon


发件人: Daleep Singh Bais 
发送时间: 2016年9月27日 6:55:10
收件人: 卢 迪; ceph-users@lists.ceph.com
主题: Re: [ceph-users] Ceph user manangerment question

Hi Dillon,

Ceph uses CephX authentication, which gives permission to users on selected 
Pools  to read / write.  We give mon 'allow r'
 to get cluster/Crush map for client.

You can refer to below URL for more information on CephX and creating user 
keyrings for access to selected / specific pools.

http://docs.ceph.com/docs/jewel/rados/configuration/auth-config-ref/
Cephx Config Reference ― Ceph 
Documentation
docs.ceph.com
Deployment Scenarios? There are two main scenarios for deploying a Ceph 
cluster, which impact how you initially configure Cephx. Most first time Ceph 
users use ceph ...




The below URL will give you information on various permissions which can be 
applied while creating a CephX authentication key.

http://docs.ceph.com/docs/firefly/rados/operations/auth-intro/
Ceph Authentication & Authorization ― Ceph 
Documentation
docs.ceph.com
Ceph Authentication & Authorization? Ceph is a distributed storage system where 
a typical deployment involves a relatively small quorum of monitors, scores of 
...




Hope this will give some insight and way forward to proceed.

Thanks,

Daleep Singh Bais

On 09/27/2016 12:02 PM, 卢 迪 wrote:

Hello all,


I'm a newbie of Ceph. I read the document and created a ceph cluster against 
VM. I have a question about how to apply user managerment to the cluster. I'm 
not asking how to create or modify users or user privileges. I have found this 
in the Ceph document.


I want to know:


1. Is there a way to know the usage of all privileges? For example, I created 
an user client.appuser with mon "allow r", this user can accsess the Ceph; If I 
removed the mon "allow r", it will be time out. (in this case, I mount the 
cluster with cephfs). If someone has these information, could you please share 
with me?


2. What kind of situation would you create differnet users for cluster? In 
currently, I user admin user to access the all cluster, such as start cluster, 
mount file system and etc. It looks like the appuser( I created above) can 
mount file system too. Is it possible to create an user liking the OS user or 
database user? So, one user upload some data, the others can't see them or can 
only read them.


ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com