Re: [ceph-users] iSCSI over RBD

2018-01-06 Thread Joshua Chen 
That is Awesome! and wonderful,
Thanks for making this acl option available.


Cheers
Joshua

On Sat, Jan 6, 2018 at 7:17 AM, Mike Christie  wrote:

> On 01/04/2018 09:36 PM, Joshua Chen wrote:
> > Hello Michael,
> >   Thanks for the reply.
> >   I did check this ceph doc at
> > http://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
> >   And yes, I need acl instead of chap usr/passwd, but I will negotiate
> > with my colleagues for changing the management style.
> >   Really appreciated for pointing the doc's bug and current status of
> > chap/acl limitation. looking forwarding to this ACL function adding to
> > gwcli.
>
> I made a patch for that here:
>
> https://github.com/ceph/ceph-iscsi-config/pull/44
>
> It is enabled by default when you first create a initiator/client. If
> you have chap enabled but want to switch then when you do "auth nochap"
> it will switch to the initiator ACL.
>
>
> >
> >
> > Cheers
> > Joshua
> >
> > On Fri, Jan 5, 2018 at 12:47 AM, Michael Christie  > > wrote:
> >
> > On 01/04/2018 03:50 AM, Joshua Chen wrote:
> > > Dear all,
> > >   Although I managed to run gwcli and created some iqns, or luns,
> > > but I do need some working config example so that my initiator
> could
> > > connect and get the lun.
> > >
> > >   I am familiar with targetcli and I used to do the following ACL
> > style
> > > connection rather than password,
> > > the targetcli setting tree is here:
> >
> > What docs have you been using? Did you check out the gwcli man page
> and
> > upstream ceph doc:
> >
> > http://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
> > 
> >
> > Let me know what is not clear in there.
> >
> > There is a bug in the upstream doc and instead of doing
> > > cd /iscsi-target/iqn.2003-01.com
> > .redhat.iscsi-gw:/disks/
> >
> > you do
> >
> > > cd /disks
> >
> > in step 3. Is that the issue you are hitting?
> >
> >
> > For gwcli, a client is the initiator. It only supports one way chap,
> so
> > there is just the 3 commands in those docs above.
> >
> > 1. create client/initiator-name. This is the same as creating the
> ACL in
> > targetcli.
> >
> > > create  iqn.1994-05.com.redhat:15dbed23be9e
> >
> > 2. set CHAP username and password for that initiator. You have to do
> > this with gwcli right now due to a bug, or maybe feature :), in the
> > code. This is simiar to doing the set auth command in targetcli.
> >
> > auth chap=/
> >
> > 3. export a image as a lun. This is equivalent to creating the lun in
> > targetcli.
> >
> > disk add rbd.some-image
> >
> >
> > >
> > > (or see this page
> >  > >)
> > >
> > > #targetcli ls
> > > o- /
> > >
> > 
> .
> > > [...]
> > >   o- backstores
> > >
> > 
> ..
> > > [...]
> > >   | o- block
> > >
> > 
> ..
> > > [Storage Objects: 1]
> > >   | | o- vmware_5t
> > > ..
> > > [/dev/rbd/rbd/vmware_5t (5.0TiB) write-thru activated]
> > >   | |   o- alua
> > >
> > 
> ...
> > > [ALUA Groups: 1]
> > >   | | o- default_tg_pt_gp
> > >
> > 
> ...
> > > [ALUA state: Active/optimized]
> > >   | o- fileio
> > >
> > 
> .
> > > [Storage Objects: 0]
> > >   | o- pscsi
> > >
> > 
> ..
> > > [Storage Objects: 0]
> > >   | o- ramdisk
> > >
> > 
> 
> > > [Storage Objects: 0]
> > >   | o- user:rbd
> > >
> > 
> ...
> > > [Storage Objects: 0]
> > >   o- iscsi
> > >
> > 
> 
> > > [Targets: 1]
> > >   | o- iqn.2017-12.asiaa.cephosd1:vmware5t
> > >
> >

[ceph-users] [luminous 12.2.2]bluestore cache uses much more memory than setting value

2018-01-06 Thread shadow_lin 
Hi all,
I have already know that luminous would use more memory for bluestore cache 
than the config setting, but I was expecting 1.5x not 7-8x.
below is my bluestore cache setting

[osd]

osd max backfills = 4

bluestore_cache_size = 134217728

bluestore_cache_kv_max = 134217728

osd client message size cap = 67108864

  My osd nodes have only 2G memory and I ran 2 osds per node.So I  set the 
cache value very low.

  I was running a read throughput test and then found some of my osds were 
killed by oom killer and restarted.I found the oom killed osd used much more 
memory for bluestore_cache_data than the normal ones.
 The oom killed osd  used 795MB ram in mempool and 722MB in 
bluestore_cache_data
 The normal osd used about 120MB ram in mempool and 17MB in 
bluestore_cache_data
 
 graph of memory useage of the oom killed osd: 
https://pasteboard.co/H1GzihS.png


 graph of memory useage of the nomral osd: https://pasteboard.co/H1GzaeF.png

Is this a bug of bluestore cache or I misunderstood the meaning of 
bluestore cache setting in config?



2018-01-06



lin.yunfan___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Adding Monitor ceph freeze, monitor 100% cpu usage

2018-01-06 Thread Nico Schottelius 

Hello,

our problems with ceph monitors continue in version 12.2.2:

Adding a specific monitor causes all monitors to hang and not respond to
ceph -s or similar anymore.

Interestingly when this monitor is on (mon.server2), the other two
monitors (mon.server3, mon.server5) randomly begin to consume 100% cpu
time, until we restart them, when the procedure repeats.

The monitor mon.server2 interestingly has a different view on the
cluster: when the other two are electing, it is in state synchronising.

We recently noticed that the MTUs of the bond0 device that we use was
setup to be 9200 and the vlan tagged device bond0.2, that we use for
ceph, also had an 9200 mtu. We raised the underlying devices and bond0
to 9204, restarted the monitors, but the problem persists.

Does anyone have a hint on how to further debug this problem?

I have added the logs from the time when we tried to restart the monitor
on server2.

Best,

Nico



ceph-mon.server2.log.bz2
Description: BZip2 compressed data


ceph-mon.server5.log.bz2
Description: BZip2 compressed data



--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] permission denied, unable to bind socket

2018-01-06 Thread Nathan Dehnel 
So I'm following the guide at
http://docs.ceph.com/docs/master/install/manual-deployment/

ceph-mon@gentooserver.service fails.

Jan 06 19:12:40 gentooserver systemd[1]: Started Ceph cluster monitor
daemon.
Jan 06 19:12:41 gentooserver ceph-mon[2674]: warning: unable to create
/var/run/ceph: (13) Permission denied
Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.000507
7f3163008f80 -1 asok(0x563d0f97d2c0) AdminSocketConfigObs::init: failed:
AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to
'/var/run/ceph/ceph-mon.gentooserver.asok': (2) No such file or directory
Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152781
7f3163008f80 -1  Processor -- bind unable to bind to
[2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested
address
Jan 06 19:12:41 gentooserver ceph-mon[2674]: 2018-01-06 19:12:41.152789
7f3163008f80 -1  Processor -- bind was unable to bind. Trying again in 5
seconds
Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152982
7f3163008f80 -1  Processor -- bind unable to bind to
[2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested
address
Jan 06 19:12:46 gentooserver ceph-mon[2674]: 2018-01-06 19:12:46.152996
7f3163008f80 -1  Processor -- bind was unable to bind. Trying again in 5
seconds
Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153218
7f3163008f80 -1  Processor -- bind unable to bind to
[2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0: (99) Cannot assign requested
address
Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153244
7f3163008f80 -1  Processor -- bind was unable to bind after 3 attempts:
(99) Cannot assign requested address
Jan 06 19:12:51 gentooserver ceph-mon[2674]: 2018-01-06 19:12:51.153250
7f3163008f80 -1 unable to bind monitor to
[2001:1c:d64b:91c5:3a84:dfce:8546:998]:6789/0
Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Main process exited, code=exited, status=1/FAILURE
Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Unit entered failed state.
Jan 06 19:12:51 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Failed with result 'exit-code'.
Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Service hold-off time over, scheduling restart.
Jan 06 19:13:01 gentooserver systemd[1]: Stopped Ceph cluster monitor
daemon.
Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Start request repeated too quickly.
Jan 06 19:13:01 gentooserver systemd[1]: Failed to start Ceph cluster
monitor daemon.
Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Unit entered failed state.
Jan 06 19:13:01 gentooserver systemd[1]: ceph-mon@gentooserver.service:
Failed with result 'exit-code'.

cat /etc/ceph/ceph.conf
[global]
fsid = a736559a-92d1-483e-9289-d2c7feed510f
ms bind ipv6 = true
mon initial members = gentooserver

[mon.mona]
host = gentooserver
mon addr = [2001:1c:d64b:91c5:3a84:dfce:8546:9982]

[osd]
osd journal size = 1

I'm not sure if the problem is the permissions error, or the IP address
appearing to get truncated in the output, or both.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com