Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade

[Bjoern Laessig]

On Do, 2016-10-27 at 15:47 +0200, mj wrote:
> Hi Jelle,
> 
> On 10/27/2016 03:04 PM, Jelle de Jong wrote:
> > Hello everybody,
> >
> > I want to upgrade my small ceph cluster to 10Gbit networking and would
> > like some recommendation from your experience.
> >
> > What is your recommend budget 10Gbit switch suitable for Ceph?
> 
> We are running a 3-node cluster, with _direct_ 10G cable connections 
> (quasi crosslink) between the three hosts. This is very low-budget, as 
> it gives you 10G speed, without a (relatively) expensive 10G switch.
> 
> Working fine here, with each host having a double 10G intel nic, plus a 
> regular 1G interface.

Could you please send me details how you configured your network?
I am planning this configuration too, but didn't test it yet.
My special interest is how you prevented ethernet loops und what did you
changed on the default ceph settings?

Thanks a lot,
Björn Lässig

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[David Disseldorp]

Hi Prabu,

On Thu, 15 Dec 2016 13:11:50 +0530, gjprabu wrote:

>   We are using ceph version 10.2.4 (Jewel) and data's are mounted 
> with cephfs file system in linux. We are trying to set quota for directory 
> and files but its don't worked with below document. I have set 100mb for 
> directory quota but after reaching keep me allowing to put the data in that 
> location. Highly appreciate any one help on this issue.

Are you using the Linux kernel CephFS client (mount.ceph), or the
userspace ceph-fuse back end? Quota enforcement is performed by the
client, and is currently only supported by ceph-fuse.

Cheers, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade

[ulembke]

Hi Björn,
i think he use something like this: 
http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server


Udo

Am 2016-12-15 11:10, schrieb Bjoern Laessig:

On Do, 2016-10-27 at 15:47 +0200, mj wrote:

Hi Jelle,

On 10/27/2016 03:04 PM, Jelle de Jong wrote:
> Hello everybody,
>
> I want to upgrade my small ceph cluster to 10Gbit networking and would
> like some recommendation from your experience.
>
> What is your recommend budget 10Gbit switch suitable for Ceph?

We are running a 3-node cluster, with _direct_ 10G cable connections
(quasi crosslink) between the three hosts. This is very low-budget, as
it gives you 10G speed, without a (relatively) expensive 10G switch.

Working fine here, with each host having a double 10G intel nic, plus 
a

regular 1G interface.


Could you please send me details how you configured your network?
I am planning this configuration too, but didn't test it yet.
My special interest is how you prevented ethernet loops und what did 
you

changed on the default ceph settings?

Thanks a lot,
Björn Lässig

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how recover the data in image

[Patrick McGarry]

Moving this to Ceph user where it can get some eyeballs.


On Dec 15, 2016 1:46 AM, "杨维云"  wrote:

>
> hi，
>
>
> we know ,the image can't share the data on the different client host , so
> if the client host is down or crash,how can i recover the data from the
> image which map to this crash host.
>
> For the  data safe  avoid  one-point disaster ,how should i do?
>
> THX
>
> Allen Yang
>
>
>
> --
> 杨维云
>
> 炫彩互动网络科技有限公司/南京总部
>
> 建邺区嘉陵江东街18号国家广告产业园4-17
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph pg active+clean+inconsistent

[Andras Pataki]

Hi everyone,

Yesterday scrubbing turned up an inconsistency in one of our placement 
groups.  We are running ceph 10.2.3, using CephFS and RBD for some VM 
images.


[root@hyperv017 ~]# ceph -s
cluster d7b33135-0940-4e48-8aa6-1d2026597c2f
 health HEALTH_ERR
1 pgs inconsistent
1 scrub errors
noout flag(s) set
 monmap e15: 3 mons at 
{hyperv029=10.4.36.179:6789/0,hyperv030=10.4.36.180:6789/0,hyperv031=10.4.36.181:6789/0}
election epoch 27192, quorum 0,1,2 
hyperv029,hyperv030,hyperv031

  fsmap e17181: 1/1/1 up {0=hyperv029=up:active}, 2 up:standby
 osdmap e342930: 385 osds: 385 up, 385 in
flags noout
  pgmap v37580512: 34816 pgs, 5 pools, 673 TB data, 198 Mobjects
1583 TB used, 840 TB / 2423 TB avail
   34809 active+clean
   4 active+clean+scrubbing+deep
   2 active+clean+scrubbing
   1 active+clean+inconsistent
  client io 87543 kB/s rd, 671 MB/s wr, 23 op/s rd, 2846 op/s wr

# ceph pg dump | grep inconsistent
6.13f1  46920   0   0   0 16057314767 3087
3087active+clean+inconsistent 2016-12-14 16:49:48.391572  
342929'41011342929:43966 [158,215,364]   158 [158,215,364]   
158 342928'40540 2016-12-14 16:49:48.391511  342928'40540
2016-12-14 16:49:48.391511


I tried a couple of other deep scrubs on pg 6.13f1 but got repeated 
errors.  In the OSD logs:


2016-12-14 16:48:07.733291 7f3b56e3a700 -1 log_channel(cluster) log 
[ERR] : deep-scrub 6.13f1 6:8fc91b77:::1000187bb70.0009:head on disk 
size (0) does not match object info size (1835008) adjusted for ondisk 
to (1835008)
I looked at the objects on the 3 OSD's on their respective hosts and 
they are the same, zero length files:


# cd ~ceph/osd/ceph-158/current/6.13f1_head
# find . -name *1000187bb70* -ls
6697380 -rw-r--r--   1 ceph ceph0 Dec 13 17:00 
./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.0009__head_EED893F1__6


# cd ~ceph/osd/ceph-215/current/6.13f1_head
# find . -name *1000187bb70* -ls
5398156470 -rw-r--r--   1 ceph ceph0 Dec 13 17:00 
./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.0009__head_EED893F1__6


# cd ~ceph/osd/ceph-364/current/6.13f1_head
# find . -name *1000187bb70* -ls
18814322150 -rw-r--r--   1 ceph ceph0 Dec 13 17:00 
./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.0009__head_EED893F1__6


At the time of the write, there wasn't anything unusual going on as far 
as I can tell (no hardware/network issues, all processes were up, etc).


This pool is a CephFS data pool, and the corresponding file (inode hex 
1000187bb70, decimal 1099537300336) looks like this:


# ls -li chr4.tags.tsv
1099537300336 -rw-r--r-- 1 xichen xichen 14469915 Dec 13 17:01 chr4.tags.tsv

Reading the file is also ok (no errors, right number of bytes):
# cat chr4.tags.tsv > /dev/null
# wc chr4.tags.tsv
  592251  2961255 14469915 chr4.tags.tsv

We are using the standard 4MB block size for CephFS, and if I interpret 
this right, this is the 9th chunk, so there shouldn't be any data (or 
even a 9th chunk), since the file is only 14MB.  Should I run pg repair 
on this?  Any ideas on how this could come about? Any other recommendations?


Thanks,

Andras
apat...@apataki.net

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

[Bjoern Laessig]

On Mi, 2016-12-14 at 18:01 +0100, Ilya Dryomov wrote:
> On Wed, Dec 14, 2016 at 5:10 PM, Bjoern Laessig  
> wrote:
> > i triggered a Kernel bug in the ceph-krbd code
> >  * http://www.spinics.net/lists/ceph-devel/msg33802.html
> 
> The fix is ready and is set to be merged into 4.10-rc1.
> 
> How often can you hit it?

I cannot hit it. My kernel guy says it is more probable, that the server
room burns down before that happens again. But if you send me the patch
at least i can test it on top of 4.9.

The whole disabling-cephx-sign-message thing is a test, whether i *can*
disable it, if it happens again.

> > Actually i do not have to delete the logfiles every 12 hours, so my pain
> > has gone but its a workaround for a workaround. That is painful. What
> > could i do to disable cephx-message-signing only for the krbd clients?
> 
> I don't think you can enable/disable message signing on a per
> connection basis - once the feature bit is negotiated, messengers on
> both sides expect everything to be signed.  Feature bits are static and
> the MSG_AUTH feature bit is enabled since bobtail and kernel 3.19.
> 
> It has to be disabled both on the server side (via ceph.conf, all
> daemons need to be restarted) and on the client side (via rbd map -o
> nocephx_sign_messages).

I will not disable security Foo. It is simple to disable, but often it
is huge mess to enable it again.

> Suppressing logging is obviously the wrong thing to do here ;)

Definitively you're right, but i sleep well in silence.

Thanks,

Björn


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cannot commit period: period does not have a master zone of a master zonegroup

[Wido den Hollander]

Hi,

On a Ceph cluster running Jewel 10.2.5 I'm running into a problem.

I want to change the amount of shards:

# radosgw-admin zonegroup-map get > zonegroup.json
# nano zonegroup.json
# radosgw-admin zonegroup-map set --infile zonegroup.json
# radosgw-admin period update --commit

Now, the error arrises:

cannot commit period: period does not have a master zone of a master zonegroup
failed to commit period: (22) Invalid argument

Looking at the output:

# radosgw-admin period update

{
...
"master_zonegroup": "",
"master_zone": "",
...
}

# radosgw-admin zone list

{
"default_info": "zm1",
"zones": [
"default",
"zm1"
]
}

To me it seems like there is something wrong with the period since there is no 
UUID present in master_zone/zonegroup.

Any idea on how to fix this?

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 10Gbit switch advice for small ceph cluster upgrade

[Bjoern Laessig]

On Do, 2016-12-15 at 14:31 +0100, ulem...@polarzone.de wrote:
> Hi Björn,
> i think he use something like this: 
> http://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

i looked into it. This was the first idea for a config i throw away.

Imagine you have 3 Systems. A,B,C

Your cable between B and C is faulty.
->
 for A: 
   everything is working.
 for B:
   Server A is up, Server C is down
 for C:
   Server A is up, Server B is down

What does your Cluster do? Where is your data. What happens now?

At the moment i think of a way ebtables preventing broadcast loops.
Testing it soon.

kind regards,
Björn Lässig

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Monitors stores not trimming after upgrade from Dumpling to Hammer

[Wido den Hollander]

> Op 7 november 2016 om 13:17 schreef Wido den Hollander :
> 
> 
> 
> > Op 4 november 2016 om 2:05 schreef Joao Eduardo Luis :
> > 
> > 
> > On 11/03/2016 06:18 PM, w...@42on.com wrote:
> > >
> > >> Personally, I don't like this solution one bit, but I can't see any 
> > >> other way without a patched monitor, or maybe ceph_monstore_tool.
> > >>
> > >> If you are willing to wait till tomorrow, I'll be happy to kludge a 
> > >> sanitation feature onto ceph_monstore_tool that will clean those 
> > >> versions for you (latency being due to coding + testing + building).
> > >>
> > >
> > > Yes, no rush. I will wait. Thanks!
> > 
> > Hey Wido,
> > 
> > I just pushed 'wip-monstore-tool-sanitize-paxos.hammer' to upstream's 
> > ceph.git. It should be built by morning.
> > 
> > This is a half-assed feature to ceph-monstore-tool, so you should test 
> > it first on a copy of your monitors, just to make sure I didn't forget 
> > any corner case.
> > 
> > The command you want is
> > 
> >ceph-monstore-tool  sanitize-paxos -- --dry-run
> > 
> > and, if you're happy with whatever it spits out, then drop --dry-run.
> > 
> > Should this work for you, next step will be to merge this onto master 
> > and, hopefully, backport to previous versions. We'll see.
> > 
> 
> I tried, but it doesn't seem to work.
> 

To wrap this up, after having a off the list discussion with Joao this was 
fixed.

The fix is in this branch: 
https://github.com/jecluis/ceph/tree/wip-monstore-tool-copy.hammer

Using 'store-copy-conservative' from the old to new path the store got trimmed 
and went down from 36GB to ~100M.

Ran this on all three Monitors one by one and it fixed it.

Joao will work on getting this back in upstream.

Wido

> # time ./ceph-monstore-tool /var/lib/ceph/mon/ceph-mon1 sanitize-paxos --
> 
> It ran for an hour (without dry-run) and outputted:
> 
> paxos [175000202 .. 175000831]
> found surplus paxos versions that ought to be removed:
>  - [ 225964 .. 226320]
>  - [ 260032 .. 260401]
>  - [ 891292 .. 891642]
> ...
> ...
> ...
>  - [ 35210293 .. 35210571]
>  - [ 35213765 .. 35213766]
> removing paxos versions [225964 .. 226320]
> removing paxos versions [260032 .. 260401]
> ...
> ...
> ...
> removing paxos versions [35188867 .. 35189115]
> removing paxos versions [35210293 .. 35210571]
> removing paxos versions [35213765 .. 35213766]
> all done!
> 
> real  57m18.904s
> user  56m26.928s
> sys   0m9.884s
> 
> 
> I ran it on the Monitor which will become primary again after it syncs. After 
> running the tool the store is still 36GB in size, grows to 50GB during the 
> sync and then trims down again to 36GB.
> 
> Wido
> 
> > HTH,
> > 
> >-Joao
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cannot commit period: period does not have a master zone of a master zonegroup

[Orit Wasserman]

Hi Wido,

This looks like you are hitting http://tracker.ceph.com/issues/17364
The fix is being backported to jewel: https://github.com/ceph/ceph/pull/12315

A workaround:
save the realm, zonegroup and zones json file
make a copy of .rgw.root (the pool contain the multisite config)
remove .rgw.root
stop the gateway
radosgw-admin realm set < json
radosgw-admin zonegroup set < json
raodsgw-admin zone set < json
radosgw-admin period update --commit
start the gateway

If the realm set will give you problems you can create a new realm
and will need to update the realm id in the zonegroup and zones json
files before using them

Orit


On Thu, Dec 15, 2016 at 4:47 PM, Wido den Hollander  wrote:
> Hi,
>
> On a Ceph cluster running Jewel 10.2.5 I'm running into a problem.
>
> I want to change the amount of shards:
>
> # radosgw-admin zonegroup-map get > zonegroup.json
> # nano zonegroup.json
> # radosgw-admin zonegroup-map set --infile zonegroup.json
> # radosgw-admin period update --commit
>
> Now, the error arrises:
>
> cannot commit period: period does not have a master zone of a master zonegroup
> failed to commit period: (22) Invalid argument
>
> Looking at the output:
>
> # radosgw-admin period update
>
> {
> ...
> "master_zonegroup": "",
> "master_zone": "",
> ...
> }
>
> # radosgw-admin zone list
>
> {
> "default_info": "zm1",
> "zones": [
> "default",
> "zm1"
> ]
> }
>
> To me it seems like there is something wrong with the period since there is 
> no UUID present in master_zone/zonegroup.
>
> Any idea on how to fix this?
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 10.2.3: Howto disable cephx_sign_messages and preventing a LogFlood

[Ilya Dryomov]

On Thu, Dec 15, 2016 at 4:31 PM, Bjoern Laessig
 wrote:
> On Mi, 2016-12-14 at 18:01 +0100, Ilya Dryomov wrote:
>> On Wed, Dec 14, 2016 at 5:10 PM, Bjoern Laessig  
>> wrote:
>> > i triggered a Kernel bug in the ceph-krbd code
>> >  * http://www.spinics.net/lists/ceph-devel/msg33802.html
>>
>> The fix is ready and is set to be merged into 4.10-rc1.
>>
>> How often can you hit it?
>
> I cannot hit it. My kernel guy says it is more probable, that the server
> room burns down before that happens again. But if you send me the patch
> at least i can test it on top of 4.9.

I concur with your kernel guy ;)  If you relieve the memory pressure,
it won't happen again.

The patch [1] is based on a few other auth-related changes.  If you
checkout [2], you'll get all of them - v4.9..b3bbd3f2ab19.

[1] 
https://github.com/ceph/ceph-client/commit/7af3ea189a9a13f090de51c97f676215dabc1205
[2] 
https://github.com/ceph/ceph-client/commit/b3bbd3f2ab19c8ca319003b4b51ce4c4ca74da06

>
> The whole disabling-cephx-sign-message thing is a test, whether i *can*
> disable it, if it happens again.
>
>> > Actually i do not have to delete the logfiles every 12 hours, so my pain
>> > has gone but its a workaround for a workaround. That is painful. What
>> > could i do to disable cephx-message-signing only for the krbd clients?
>>
>> I don't think you can enable/disable message signing on a per
>> connection basis - once the feature bit is negotiated, messengers on
>> both sides expect everything to be signed.  Feature bits are static and
>> the MSG_AUTH feature bit is enabled since bobtail and kernel 3.19.
>>
>> It has to be disabled both on the server side (via ceph.conf, all
>> daemons need to be restarted) and on the client side (via rbd map -o
>> nocephx_sign_messages).
>
> I will not disable security Foo. It is simple to disable, but often it
> is huge mess to enable it again.

Enabling it is the reverse: cephx_sign_messages = true and rbd map as
usual.  Nothing to it, but I hear you.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues on Jewel 10.2.2

[Vincent Godin]

Hello,

I didn't look at your video but i already can tell you some tracks :

1 - there is a bug in 10.2.2 which make the client cache not working. The
client cache works as it never recieved a flush so it will stay in
writethrough mode. This bug is clear in 10.2.3

2 - 2 SSDs in JBOD and 12 x 4TB NL SAS in RAID0 are not very well optimized
if your workload is based on write. You will perform in write at the max
speed of your two SSD only. I don't know the real speed of your SSD nor
your SAS disks but let's say:

your SSD can reach a 400 MB/s in write throughput
your SAS can reach a 130 MB/s in write throughput

i suppose that you use 1 SSD to host the journals of 6 SAS
Your max throughput in write will be 2 x 400 MB/s so 800 MB/s compare to
the 12 x 130 MB/s = 1560 MB/s of your SAS

if you had 4 SSD for the journal, 1 SSD for 3 SAS
Your max throughput would be 4 x 400 MB/s so 1600 MB/s very near of the
1560 MB/s of your SAS

Of course, you need to adjust that with the real throughput of your SSD ans
SAS disks

Vincent
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Revisiting: Many clients (X) failing to respond to cache pressure

[Goncalo Borges]

Hi John...

Regarding logs, we still do not have them available. We just realized 
that ceph-fuse tries to log to /var/log/ceph, which in our case didn't 
exist in the clients. So, we had to create that directory everywhere, 
and we are in the process of remounting every client so that they starts 
logging. Since by umounting we are forcing the client to free the 
inodes, we have to wait for the situation to reappear.


However. I have a bit further information. Maybe it can shed a further 
light on the topic.


- If I currently loop through all my clients, I now get a total of 29604 
inodes.


 $ cat clients_inodes_20161216-0938.txt | grep inode_count | awk
   '{print $2}' | sed 's/,//g' | awk '{s+=$1} END {print s}'
   29604

- However, the mds reports '"inodes": 1779521' and '"inodes_with_caps": 
32823,'. Is there a need for the MDS to keep in memory such a large 
amount of inodes without associated caps? I also expect that these are 
the first ones to be trimmed once inodes > inode_max.


  "mds": {
(...)
"inode_max": 200,
"inodes": 1779521,
"inodes_top": 18119,
"inodes_bottom": 1594129,
"inodes_pin_tail": 167273,
"inodes_pinned": 182643,
"inodes_expired": 53391877,
"inodes_with_caps": 32823,
"caps": 35133,
   (...)
},

- I am also seeing some false positives (I think). As I explained 
before, we have currently umounted all clients except 2 (they are 
interactive machines where our users run tmux sessions and so on... So, 
it is hard to kick them out :-) ). One of those two is still reported as 
problematic by MDS although inodes < inodes_max. Looking to the number 
of inodes of that machine, I get "inode_count": 13862. So, it seems that 
the client is still tagged as problematic although it has an inode_count 
bellow 16384 and inodes < inodes_max. Maybe a consequence of 
https://github.com/ceph/ceph/pull/11373 ? And this fix seems to only go 
on Kraken?


Cheers

Goncalo





Cheers

Goncalo



On 12/14/2016 10:16 AM, Goncalo Borges wrote:

Hi John.

Comments in line.



Hi Ceph(FS)ers...

I am currently running in production the following environment:

- ceph/cephfs in 10.2.2.
- All infrastructure is in the same version (rados cluster, mons, 
mds and

cephfs clients).
- We mount cephfs using ceph-fuse.

Since yesterday that we have our cluster in warning state with the 
message
"mds0: Many clients (X) failing to respond to cache pressure". X has 
been
changing with time, from ~130 to ~70. I am able to correlate the 
appearance

of this message with burst of jobs in our cluster.

This subject has been discussed in the mailing list a lot of times, and
normally, the recipe is to look for something wrong in the clients. 
So, I

have tried to look to clients first:

1) I've started to loop through all my clients, and run 'ceph 
--admin-daemon
/var/run/ceph/ceph-client.mount_user.asok status' to get the 
inodes_count

reported in each client.

$ cat all.txt | grep inode_count | awk '{print $2}' | sed 's/,//g' | 
awk

'{s+=$1} END {print s}'
2407659

2) I've then compared with the number of inodes the mds had in its 
cache

(obtained by a perf dump)
  inode_max": 200 and "inodes": 2413826

3) I've tried to understand how many clients had a number of inodes 
higher

than 16384 (the default) and got

$ for i in `cat all.txt | grep inode_count | awk '{print $2}' | sed 
's/,//g'

`; do if [ $i -ge 16384 ]; then echo $i; fi; done | wc -l
27

4) My conclusion is that the core of inodes is held by a couple of 
machines.

However, while the majority is running user jobs, others are not doing
anything at all. For example, an idle machine (which had no users 
logged in,
no jobs running, updatedb does not search for cephfs filesystem) 
reported

more than > 30 inodes). To regain those inodes, I had to umount and
remount cephfs in that machine.

5) Based on my previous observations I suspect that there are still 
some
problems in the ceph-fuse client regarding recovering these inodes 
(or it

happens at a very slow rate).

Seems that way.  Can you come up with a reproducer for us, and/or
gather some client+mds debug logs where a client is failing to respond
to cache pressure?


I think I've nailed this down to a specific user workload. Everytime 
this user runs, it lefts the client with a huge number of inodes, 
normally more than 10. The workload consists in the generations of 
a big number of analysis files spread over multiple directories. I am 
going to try to inject some debug parameters and see what do we come 
up with. Will reply on this thread later on.




Also, what kernel is in use on the clients?  It's possible that the
issue is in FUSE itself (or the way that it responses to ceph-fuse's
attempts to ask for some inodes to be released).


All our clusters run SL6 because CERN experiments software is only 
certified to that OS flavour. Because of the SL6 restriction, to 
enable pos infernalis c

Re: [ceph-users] Suggestion:-- Disable warning in ceph -s output

[Jayaram Radhakrishnan]

Hello Team,

Can I get any info for this query please ?

Thanks

On Thu, Dec 15, 2016 at 7:15 PM, Jayaram Radhakrishnan <
jayaram161...@gmail.com> wrote:

> Hello Team,
>
> Is there any way to disable warning messages prompting from
> ceph -s" output
>
> ~~~
>
> WARNING: the following dangerous and experimental features are enabled:
> bluestore,rocksdb
>
> ~~~
>
>
> Here I'm testing with kracken and bluestore, I beleive there is one
> directive which is reposible for  hiding these warnings from ceph -s output.
>
>
> Awaiting for the suggestions.
>
>
> --
> Thanks & Regards
>
> jayaram
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[gjprabu]

Hi David,



Thanks for your mail, We are currently using Linux kernel CephFS, 
Is it possible to use ceph-fuse without disturbing current setup.



Regards

Prabu GJ




 On Thu, 15 Dec 2016 15:55:12 +0530 David Disseldorp  
wrote 




Hi Prabu, 

 

On Thu, 15 Dec 2016 13:11:50 +0530, gjprabu wrote: 

 

> We are using ceph version 10.2.4 (Jewel) and data's are mounted with 
cephfs file system in linux. We are trying to set quota for directory and files 
but its don't worked with below document. I have set 100mb for directory quota 
but after reaching keep me allowing to put the data in that location. Highly 
appreciate any one help on this issue. 

 

Are you using the Linux kernel CephFS client (mount.ceph), or the 

userspace ceph-fuse back end? Quota enforcement is performed by the 

client, and is currently only supported by ceph-fuse. 

 

Cheers, David 






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

[gjprabu]

Hi David,



Now we are mounted client using ceph-fuse and still allowing me to 
put a data above the limit(100MB). Below is quota details.



getfattr -n ceph.quota.max_bytes test

# file: test

ceph.quota.max_bytes="1"   



ceph-fuse  fuse.ceph-fuse  5.3T  485G  4.8T  10% /mnt/test



du -sh test

1.9G test





Regards

Prabu GJ


 On Fri, 16 Dec 2016 11:18:46 +0530 gjprabu  
wrote 




Hi David,



Thanks for your mail, We are currently using Linux kernel CephFS, 
Is it possible to use ceph-fuse without disturbing current setup.



Regards

Prabu GJ





 On Thu, 15 Dec 2016 15:55:12 +0530 David Disseldorp  
wrote 










___ 

ceph-users mailing list 

ceph-users@lists.ceph.com 

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 


Hi Prabu, 



On Thu, 15 Dec 2016 13:11:50 +0530, gjprabu wrote: 



> We are using ceph version 10.2.4 (Jewel) and data's are mounted with 
cephfs file system in linux. We are trying to set quota for directory and files 
but its don't worked with below document. I have set 100mb for directory quota 
but after reaching keep me allowing to put the data in that location. Highly 
appreciate any one help on this issue. 



Are you using the Linux kernel CephFS client (mount.ceph), or the 

userspace ceph-fuse back end? Quota enforcement is performed by the 

client, and is currently only supported by ceph-fuse. 



Cheers, David 






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com