dear ALL:
I used PCIE-SSD to OSD disk . But I found it very bottom performance.
I have two hosts, each host 1 PCIE-SSD,so i create two osd by PCIE-SSD.
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.35999 root default
-2 0.17999 host tds_node03
0 0.1
Hello,
from all the pertinent points by Somnath, the one about pre-conditioning
would be pretty high on my list, especially if this slowness persists and
nothing else (scrub) is going on.
This might be "fixed" by doing a fstrim.
Additionally the levelDB's per OSD are of course sync'ing heavily
Hello,
On Thu, 20 Aug 2015 11:55:55 +1000 Jiri Kanicky wrote:
> Hi all,
>
> We are experimenting with an idea to run OSD nodes in XenServer VMs. We
> believe this could provide better flexibility, backups for the nodes etc.
>
> For example:
> Xenserver with 4 HDDs dedicated for Ceph.
> We wou
Just to clarify - you unmounted the filesystem with "umount -l"? That almost
never a good idea, and it puts the OSD in a very unusual situation where IO
will actually work on the open files, but it can't open any new ones. I think
this would be enough to confuse just about any piece of software.
Hey all,
We are currently testing CephFS on a small (3 node) cluster.
The setup is currently:
Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:
The servers are running: 0.94.2-0.el7
The clients are running: Ceph: 0.80.10-1.fc21, Kernel: 4.0.6-200.fc21.x86_64
ceph -s
cluster 4ed5ec
Hi Samuel, we try to fix it in trick way.
we check all rbd_data chunks from logs (OSD) which are affected, then query
rbd info to compare which rbd consist bad rbd_data, after that we mount
this rbd as rbd0, create empty rbd, and DD all info from bad volume to new
one.
But after that - scrub erro
The code is at https://github.com/ceph/samba.git wip-acl. So far the
code does not handle default ACL (files created by samba do not
inherit parent directory's default ACL)
Regards
Yan, Zheng
On Tue, Aug 18, 2015 at 6:57 PM, Gregory Farnum wrote:
> On Mon, Aug 17, 2015 at 4:12 AM, Yan, Zheng w
Hi,
Just to update the mailing list, we ended up going back to default
ceph.conf without any additional settings than what is mandatory. We are
now reaching speeds we never reached before, both in recovery and in
regular usage. There was definitely something we set in the ceph.conf
bogging everyth
>
> Just to update the mailing list, we ended up going back to default
> ceph.conf without any additional settings than what is mandatory. We are
> now reaching speeds we never reached before, both in recovery and in
> regular usage. There was definitely something we set in the ceph.conf
> bogging
Are you sure it was because of configuration changes?
Maybe it was restarting the OSDs that fixed it?
We often hit an issue with backfill_toofull where the recovery/backfill
processes get stuck until we restart the daemons (sometimes setting
recovery_max_active helps as well). It still shows reco
Ok, you appear to be using a replicated cache tier in front of a
replicated base tier. Please scrub both inconsistent pgs and post the
ceph.log from before when you started the scrub until after. Also,
what command are you using to take snapshots?
-Sam
On Thu, Aug 20, 2015 at 3:59 AM, Voloshanen
Also, was there at any point a power failure/power cycle event,
perhaps on osd 56?
-Sam
On Thu, Aug 20, 2015 at 9:23 AM, Samuel Just wrote:
> Ok, you appear to be using a replicated cache tier in front of a
> replicated base tier. Please scrub both inconsistent pgs and post the
> ceph.log from b
Samuel, we turned off cache layer few hours ago...
I will post ceph.log in few minutes
For snap - we found issue, was connected with cache tier..
2015-08-20 19:23 GMT+03:00 Samuel Just :
> Ok, you appear to be using a replicated cache tier in front of a
> replicated base tier. Please scrub both
What was the issue?
-Sam
On Thu, Aug 20, 2015 at 9:41 AM, Voloshanenko Igor
wrote:
> Samuel, we turned off cache layer few hours ago...
> I will post ceph.log in few minutes
>
> For snap - we found issue, was connected with cache tier..
>
> 2015-08-20 19:23 GMT+03:00 Samuel Just :
>>
>> Ok, you a
Issue, that in forward mode, fstrim doesn't work proper, and when we take
snapshot - data not proper update in cache layer, and client (ceph) see
damaged snap.. As headers requested from cache layer.
2015-08-20 19:53 GMT+03:00 Samuel Just :
> What was the issue?
> -Sam
>
> On Thu, Aug 20, 2015 at
Is there a bug for this in the tracker?
-Sam
On Thu, Aug 20, 2015 at 9:54 AM, Voloshanenko Igor
wrote:
> Issue, that in forward mode, fstrim doesn't work proper, and when we take
> snapshot - data not proper update in cache layer, and client (ceph) see
> damaged snap.. As headers requested from c
Not yet. I will create.
But according to mail lists and Inktank docs - it's expected behaviour when
cache enable
2015-08-20 19:56 GMT+03:00 Samuel Just :
> Is there a bug for this in the tracker?
> -Sam
>
> On Thu, Aug 20, 2015 at 9:54 AM, Voloshanenko Igor
> wrote:
> > Issue, that in forward mo
Which docs?
-Sam
On Thu, Aug 20, 2015 at 9:57 AM, Voloshanenko Igor
wrote:
> Not yet. I will create.
> But according to mail lists and Inktank docs - it's expected behaviour when
> cache enable
>
> 2015-08-20 19:56 GMT+03:00 Samuel Just :
>>
>> Is there a bug for this in the tracker?
>> -Sam
>>
>
Inktank:
https://download.inktank.com/docs/ICE%201.2%20-%20Cache%20and%20Erasure%20Coding%20FAQ.pdf
Mail-list:
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg18338.html
2015-08-20 20:06 GMT+03:00 Samuel Just :
> Which docs?
> -Sam
>
> On Thu, Aug 20, 2015 at 9:57 AM, Voloshanenko Igor
Guys,
I'm Igor's colleague, working a bit on CEPH, together with Igor.
This is production cluster, and we are becoming more desperate as the time
goes by.
Im not sure if this is appropriate place to seek commercial support, but
anyhow, I do it...
If anyone feels like and have some experience i
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Jacek Jarosiewicz
> Sent: 20 August 2015 07:31
> To: Nick Fisk ; ceph-us...@ceph.com
> Subject: Re: [ceph-users] requests are blocked - problem
>
> On 08/19/2015 03:41 PM, Nick Fisk wrote:
Someone using the email address
lgx...@nxtzas.com
is trying to subscribe to the Ceph Redmine tracker, but neither redmine nor I
can use that email address; it bounces with
: Host or domain name not found. Name service error for
name=nxtzas.com type=: Host not found
If this is you, ple
Ah, this is kind of silly. I think you don't have 37 errors, but 2
errors. pg 2.490 object
3fac9490/rbd_data.eb5f22eb141f2.04ba/snapdir//2 is missing
snap 141. If you look at the objects after that in the log:
2015-08-20 20:15:44.865670 osd.19 10.12.2.6:6838/1861727 298 : cluster
[E
The feature bug for the tool is http://tracker.ceph.com/issues/12740.
-Sam
On Thu, Aug 20, 2015 at 2:52 PM, Samuel Just wrote:
> Ah, this is kind of silly. I think you don't have 37 errors, but 2
> errors. pg 2.490 object
> 3fac9490/rbd_data.eb5f22eb141f2.04ba/snapdir//2 is missing
thank you Sam!
I also noticed this linked errors during scrub...
Now all lools like reasonable!
So we will wait for bug to be closed.
do you need any help on it?
I mean i can help with coding/testing/etc...
2015-08-21 0:52 GMT+03:00 Samuel Just :
> Ah, this is kind of silly. I think you don'
Actually, now that I think about it, you probably didn't remove the
images for 3fac9490/rbd_data.eb5f22eb141f2.04ba/snapdir//2
and 22ca30c4/rbd_data.e846e25a70bf7.0307/snapdir//2, but
other images (that's why the scrub errors went down briefly, those
objects -- which were fi
This was related to the caching layer, which doesnt support snapshooting
per docs...for sake of closing the thread.
On 17 August 2015 at 21:15, Voloshanenko Igor
wrote:
> Hi all, can you please help me with unexplained situation...
>
> All snapshot inside ceph broken...
>
> So, as example, we ha
Sam, i try to understand which rbd contain this chunks.. but no luck. No
rbd images block names started with this...
Actually, now that I think about it, you probably didn't remove the
> images for 3fac9490/rbd_data.eb5f22eb141f2.04ba/snapdir//2
> and 22ca30c4/rbd_data.e846e25a70bf7.00
Interesting. How often do you delete an image? I'm wondering if
whatever this is happened when you deleted these two images.
-Sam
On Thu, Aug 20, 2015 at 3:42 PM, Voloshanenko Igor
wrote:
> Sam, i try to understand which rbd contain this chunks.. but no luck. No rbd
> images block names started
Image? One?
We start deleting images only to fix thsi (export/import)m before - 1-4
times per day (when VM destroyed)...
2015-08-21 1:44 GMT+03:00 Samuel Just :
> Interesting. How often do you delete an image? I'm wondering if
> whatever this is happened when you deleted these two images.
>
Ok, so images are regularly removed. In that case, these two objects
probably are left over from previously removed images. Once
ceph-objectstore-tool can dump the SnapSet from those two objects, you
will probably find that those two snapdir objects each have only one
bogus clone, in which case y
Snapshotting with cache/tiering *is* supposed to work. Can you open a bug?
-Sam
On Thu, Aug 20, 2015 at 3:36 PM, Andrija Panic wrote:
> This was related to the caching layer, which doesnt support snapshooting per
> docs...for sake of closing the thread.
>
> On 17 August 2015 at 21:15, Voloshanen
Also, can you include the kernel version?
-Sam
On Thu, Aug 20, 2015 at 3:51 PM, Samuel Just wrote:
> Snapshotting with cache/tiering *is* supposed to work. Can you open a bug?
> -Sam
>
> On Thu, Aug 20, 2015 at 3:36 PM, Andrija Panic
> wrote:
>> This was related to the caching layer, which doe
Yes, will do.
What we see. When cache tier in forward mod, if i did
rbd snap create - it's use rbd_header not from cold tier, but from
hot-tier, butm this 2 headers not synced
And can;t be evicted from hot-storage, as it;s locked by KVM (Qemu). If i
kill lock, evict header - all start to work..
Bu
root@test:~# uname -a
Linux ix-s5 4.0.4-040004-generic #201505171336 SMP Sun May 17 17:37:22 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
2015-08-21 1:54 GMT+03:00 Samuel Just :
> Also, can you include the kernel version?
> -Sam
>
> On Thu, Aug 20, 2015 at 3:51 PM, Samuel Just wrote:
> > Snapshotting
Hmm, that might actually be client side. Can you attempt to reproduce
with rbd-fuse (different client side implementation from the kernel)?
-Sam
On Thu, Aug 20, 2015 at 3:56 PM, Voloshanenko Igor
wrote:
> root@test:~# uname -a
> Linux ix-s5 4.0.4-040004-generic #201505171336 SMP Sun May 17 17:37
We used 4.x branch, as we have "very good" Samsung 850 pro in production,
and they don;t support ncq_trim...
And 4,x first branch which include exceptions for this in libsata.c.
sure we can backport this 1 line to 3.x branch, but we prefer no to go
deeper if packege for new kernel exist.
2015-08
I already kill cache layer, but will try to reproduce on lab
2015-08-21 1:58 GMT+03:00 Samuel Just :
> Hmm, that might actually be client side. Can you attempt to reproduce
> with rbd-fuse (different client side implementation from the kernel)?
> -Sam
>
> On Thu, Aug 20, 2015 at 3:56 PM, Volosha
What's supposed to happen is that the client transparently directs all
requests to the cache pool rather than the cold pool when there is a
cache pool. If the kernel is sending requests to the cold pool,
that's probably where the bug is. Odd. It could also be a bug
specific 'forward' mode either
Certainly, don't reproduce this with a cluster you care about :).
-Sam
On Thu, Aug 20, 2015 at 4:02 PM, Samuel Just wrote:
> What's supposed to happen is that the client transparently directs all
> requests to the cache pool rather than the cold pool when there is a
> cache pool. If the kernel i
We switch to forward mode as step to switch cache layer off.
Right now we have "samsung 850 pro" in cache layer (10 ssd, 2 per nodes)
and they show 2MB for 4K blocks... 250 IOPS... intead of 18-20K for intel
S3500 240G which we choose as replacement..
So with such good disks - cache layer - very
Good joke )
2015-08-21 2:06 GMT+03:00 Samuel Just :
> Certainly, don't reproduce this with a cluster you care about :).
> -Sam
>
> On Thu, Aug 20, 2015 at 4:02 PM, Samuel Just wrote:
> > What's supposed to happen is that the client transparently directs all
> > requests to the cache pool
So you started draining the cache pool before you saw either the
inconsistent pgs or the anomalous snap behavior? (That is, writeback
mode was working correctly?)
-Sam
On Thu, Aug 20, 2015 at 4:07 PM, Voloshanenko Igor
wrote:
> Good joke )
>
> 2015-08-21 2:06 GMT+03:00 Samuel Just :
>>
>
Created a ticket to improve our testing here -- this appears to be a hole.
http://tracker.ceph.com/issues/12742
-Sam
On Thu, Aug 20, 2015 at 4:09 PM, Samuel Just wrote:
> So you started draining the cache pool before you saw either the
> inconsistent pgs or the anomalous snap behavior? (That is
No, when we start draining cache - bad pgs was in place...
We have big rebalance (disk by disk - to change journal side on both
hot/cold layers).. All was Ok, but after 2 days - arrived scrub errors and
2 pgs inconsistent...
In writeback - yes, looks like snapshot works good. but it's stop to work
Not sure what you mean by:
but it's stop to work in same moment, when cache layer fulfilled with
data and evict/flush started...
-Sam
On Thu, Aug 20, 2015 at 4:11 PM, Voloshanenko Igor
wrote:
> No, when we start draining cache - bad pgs was in place...
> We have big rebalance (disk by disk - to
Also, what do you mean by "change journal side"?
-Sam
On Thu, Aug 20, 2015 at 4:15 PM, Samuel Just wrote:
> Not sure what you mean by:
>
> but it's stop to work in same moment, when cache layer fulfilled with
> data and evict/flush started...
> -Sam
>
> On Thu, Aug 20, 2015 at 4:11 PM, Voloshanen
WE haven't set values for max_bytes / max_objects.. and all data initially
writes only to cache layer and not flushed at all to cold layer.
Then we received notification from monitoring that we collect about 750GB
in hot pool ) So i changed values for max_object_bytes to be 0,9 of disk
size... And
But that was still in writeback mode, right?
-Sam
On Thu, Aug 20, 2015 at 4:18 PM, Voloshanenko Igor
wrote:
> WE haven't set values for max_bytes / max_objects.. and all data initially
> writes only to cache layer and not flushed at all to cold layer.
>
> Then we received notification from monito
Our initial values for journal sizes was enough, but flush time was 5 secs,
so we increase journal side to fit flush timeframe min|max for 29/30
seconds.
I mean
filestore max sync interval = 30
filestore min sync interval = 29
when said flush time
2015-08-21 2:16 GMT+03:00 Samuel Just :
> Al
Right. But issues started...
2015-08-21 2:20 GMT+03:00 Samuel Just :
> But that was still in writeback mode, right?
> -Sam
>
> On Thu, Aug 20, 2015 at 4:18 PM, Voloshanenko Igor
> wrote:
> > WE haven't set values for max_bytes / max_objects.. and all data
> initially
> > writes only to cache lay
Yeah, I'm trying to confirm that the issues did happen in writeback mode.
-Sam
On Thu, Aug 20, 2015 at 4:21 PM, Voloshanenko Igor
wrote:
> Right. But issues started...
>
> 2015-08-21 2:20 GMT+03:00 Samuel Just :
>>
>> But that was still in writeback mode, right?
>> -Sam
>>
>> On Thu, Aug 20, 2015
Specifically, the snap behavior (we already know that the pgs went
inconsistent while the pool was in writeback mode, right?).
-Sam
On Thu, Aug 20, 2015 at 4:22 PM, Samuel Just wrote:
> Yeah, I'm trying to confirm that the issues did happen in writeback mode.
> -Sam
>
> On Thu, Aug 20, 2015 at 4:
I mean in forward mode - it;s permanent problem - snapshots not working.
And for writeback mode after we change max_bytes/object values, it;s around
30 by 70... 70% of time it;s works... 30% - not. Looks like for old images
- snapshots works fine (images which already exists before we change
values
Right ( but also was rebalancing cycle 2 day before pgs corrupted)
2015-08-21 2:23 GMT+03:00 Samuel Just :
> Specifically, the snap behavior (we already know that the pgs went
> inconsistent while the pool was in writeback mode, right?).
> -Sam
>
> On Thu, Aug 20, 2015 at 4:22 PM, Samuel Just wr
Exactly
пятница, 21 августа 2015 г. пользователь Samuel Just написал:
> And you adjusted the journals by removing the osd, recreating it with
> a larger journal, and reinserting it?
> -Sam
>
> On Thu, Aug 20, 2015 at 4:24 PM, Voloshanenko Igor
> > wrote:
> > Right ( but also was rebalancing cycle
And you adjusted the journals by removing the osd, recreating it with
a larger journal, and reinserting it?
-Sam
On Thu, Aug 20, 2015 at 4:24 PM, Voloshanenko Igor
wrote:
> Right ( but also was rebalancing cycle 2 day before pgs corrupted)
>
> 2015-08-21 2:23 GMT+03:00 Samuel Just :
>>
>> Specifi
Ok, create a ticket with a timeline and all of this information, I'll
try to look into it more tomorrow.
-Sam
On Thu, Aug 20, 2015 at 4:25 PM, Voloshanenko Igor
wrote:
> Exactly
>
> пятница, 21 августа 2015 г. пользователь Samuel Just написал:
>
>> And you adjusted the journals by removing the os
As i we use journal collocation for journal now (because we want to utilize
cache layer ((( ) i use ceph-disk to create new OSD (changed journal size
on ceph.conf). I don;t prefer manual work))
So create very simple script to update journal size
2015-08-21 2:25 GMT+03:00 Voloshanenko Igor :
> Ex
Will do, Sam!
thank in advance for you help!
2015-08-21 2:28 GMT+03:00 Samuel Just :
> Ok, create a ticket with a timeline and all of this information, I'll
> try to look into it more tomorrow.
> -Sam
>
> On Thu, Aug 20, 2015 at 4:25 PM, Voloshanenko Igor
> wrote:
> > Exactly
> >
> > пятница, 2
Attachment blocked, so post as text...
root@zzz:~# cat update_osd.sh
#!/bin/bash
ID=$1
echo "Process OSD# ${ID}"
DEV=`mount | grep "ceph-${ID} " | cut -d " " -f 1`
echo "OSD# ${ID} hosted on ${DEV::-1}"
TYPE_RAW=`smartctl -a ${DEV} | grep Rota | cut -d " " -f 6`
if [ "${TYPE_RAW}" == "Solid" ]
It would help greatly if, on a disposable cluster, you could reproduce
the snapshot problem with
debug osd = 20
debug filestore = 20
debug ms = 1
on all of the osds and attach the logs to the bug report. That should
make it easier to work out what is going on.
-Sam
On Thu, Aug 20, 2015 at 4:40
Hi Jiri,
On Thu, 20 Aug 2015 11:55:55 +1000
Jiri Kanicky wrote:
> We are experimenting with an idea to run OSD nodes in XenServer VMs.
> We believe this could provide better flexibility, backups for the
> nodes etc.
Could you expand on this? As written, it seems like a bad idea to me,
just beca
dear Loic:
I'm sorry to bother you.But I have a question about ceph.
I used PCIE-SSD to OSD disk . But I found it very bottom performance.
I have two hosts, each host 1 PCIE-SSD,so i create two osd by PCIE-SSD.
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.35999 ro
Hello,
On Thu, 20 Aug 2015 15:47:46 +0800 scott_tan...@yahoo.com wrote:
The reason that you're not getting any replies is because we're not
psychic/telepathic/clairvoyant.
Meaning that you're not giving us enough information by far.
> dear ALL:
> I used PCIE-SSD to OSD disk . But I found
my ceph.conf
[global]
auth_service_required = cephx
osd_pool_default_size = 2
filestore_xattr_use_omap = true
auth_client_required = cephx
auth_cluster_required = cephx
mon_host = 172.168.2.171
mon_initial_members = tds_node01
fsid = fef619c
On Fri, Aug 21, 2015 at 2:02 AM, Samuel Just wrote:
> What's supposed to happen is that the client transparently directs all
> requests to the cache pool rather than the cold pool when there is a
> cache pool. If the kernel is sending requests to the cold pool,
> that's probably where the bug is.
Hello,
I cloned the master branch of Ceph and after setting up the cluster, when I
tried to use the rados commands, I got this error:
rados: symbol lookup error: rados: undefined symbol:
_ZN5MutexC1ERKSsbbbP11CephContext
I saw a similar post here: http://tracker.ceph.com/issues/12563 but I am
Hi ,
I've do that before and when I try to write file into rbd. It's get
freeze.
Beside resource, is there any other reason not recommend to combined mon
and osd?
Best wishes,
Mika
2015-08-18 15:52 GMT+08:00 Межов Игорь Александрович :
> Hi!
>
> You can run mons on the same hosts, tho
Exact as in our case.
Ilya, same for images from our side. Headers opened from hot tier
пятница, 21 августа 2015 г. пользователь Ilya Dryomov написал:
> On Fri, Aug 21, 2015 at 2:02 AM, Samuel Just > wrote:
> > What's supposed to happen is that the client transparently directs all
> > requests
70 matches
Mail list logo
